Why does accessing columns of a pandas dataframe with .loc[] produce duplicate rows?
Why is .loc producing duplicate rows in my DataFrame? I'm trying to select a few columns from m3, a DataFrame with 47 columns,to create a new DataFrame called output.
The problem: after accessing m3's columns with .loc, output has way more duplicates than m3 started with. Where could these duplicates have come from? I haven't found anything online about .loc duplicating rows. The output DataFrame is declared on the line that reads output = m3.loc[...], by the way.
The Code:
print("ARE THERE DUPLICATES in m3? ")
print(m3.duplicated().loc[lambda x: x==True])
output = m3.loc[:,["PLC_name", "line", "track", "notes", "final_source",
"s_name", "s_line", "s_track", "loc", "alt_loc", "suffix", "alt_match_name"]]
print("ARE THERE DUPLICATES in output? ")
print(output.duplicated().loc[lambda x: x==True].size, "duplicates")
The Terminal Output:
ARE THERE DUPLICATES in m3?
5241 True
5242 True
5243 True
5355 True
5356 True
5357 True
dtype: bool
ARE THERE DUPLICATES in output?
1838 duplicates
Of course, I could easily fix the problem by calling .drop_duplicates(keep="first"), but I'm more interesting in learning why .loc displays this behavior.
python pandas csv dataframe duplicates
asked Nov 16 '18 at 22:57
DavidDavid
316
add a comment |
Why is .loc producing duplicate rows in my DataFrame? I'm trying to select a few columns from m3, a DataFrame with 47 columns,to create a new DataFrame called output.
The problem: after accessing m3's columns with .loc, output has way more duplicates than m3 started with. Where could these duplicates have come from? I haven't found anything online about .loc duplicating rows. The output DataFrame is declared on the line that reads output = m3.loc[...], by the way.
The Code:
print("ARE THERE DUPLICATES in m3? ")
print(m3.duplicated().loc[lambda x: x==True])
output = m3.loc[:,["PLC_name", "line", "track", "notes", "final_source",
"s_name", "s_line", "s_track", "loc", "alt_loc", "suffix", "alt_match_name"]]
print("ARE THERE DUPLICATES in output? ")
print(output.duplicated().loc[lambda x: x==True].size, "duplicates")
The Terminal Output:
ARE THERE DUPLICATES in m3?
5241 True
5242 True
5243 True
5355 True
5356 True
5357 True
dtype: bool
ARE THERE DUPLICATES in output?
1838 duplicates
Of course, I could easily fix the problem by calling .drop_duplicates(keep="first"), but I'm more interesting in learning why .loc displays this behavior.
python pandas csv dataframe duplicates
asked Nov 16 '18 at 22:57
DavidDavid
316
add a comment |
Why is .loc producing duplicate rows in my DataFrame? I'm trying to select a few columns from m3, a DataFrame with 47 columns,to create a new DataFrame called output.
The problem: after accessing m3's columns with .loc, output has way more duplicates than m3 started with. Where could these duplicates have come from? I haven't found anything online about .loc duplicating rows. The output DataFrame is declared on the line that reads output = m3.loc[...], by the way.
The Code:
print("ARE THERE DUPLICATES in m3? ")
print(m3.duplicated().loc[lambda x: x==True])
output = m3.loc[:,["PLC_name", "line", "track", "notes", "final_source",
"s_name", "s_line", "s_track", "loc", "alt_loc", "suffix", "alt_match_name"]]
print("ARE THERE DUPLICATES in output? ")
print(output.duplicated().loc[lambda x: x==True].size, "duplicates")
The Terminal Output:
ARE THERE DUPLICATES in m3?
5241 True
5242 True
5243 True
5355 True
5356 True
5357 True
dtype: bool
ARE THERE DUPLICATES in output?
1838 duplicates
Of course, I could easily fix the problem by calling .drop_duplicates(keep="first"), but I'm more interesting in learning why .loc displays this behavior.
python pandas csv dataframe duplicates
asked Nov 16 '18 at 22:57
DavidDavid
316
Why is .loc producing duplicate rows in my DataFrame? I'm trying to select a few columns from m3, a DataFrame with 47 columns,to create a new DataFrame called output.
The problem: after accessing m3's columns with .loc, output has way more duplicates than m3 started with. Where could these duplicates have come from? I haven't found anything online about .loc duplicating rows. The output DataFrame is declared on the line that reads output = m3.loc[...], by the way.
The Code:
print("ARE THERE DUPLICATES in m3? ")
print(m3.duplicated().loc[lambda x: x==True])
output = m3.loc[:,["PLC_name", "line", "track", "notes", "final_source",
"s_name", "s_line", "s_track", "loc", "alt_loc", "suffix", "alt_match_name"]]
print("ARE THERE DUPLICATES in output? ")
print(output.duplicated().loc[lambda x: x==True].size, "duplicates")
The Terminal Output:
ARE THERE DUPLICATES in m3?
5241 True
5242 True
5243 True
5355 True
5356 True
5357 True
dtype: bool
ARE THERE DUPLICATES in output?
1838 duplicates
Of course, I could easily fix the problem by calling .drop_duplicates(keep="first"), but I'm more interesting in learning why .loc displays this behavior.
python pandas csv dataframe duplicates
python pandas csv dataframe duplicates
asked Nov 16 '18 at 22:57
DavidDavid
316
asked Nov 16 '18 at 22:57
DavidDavid
316
asked Nov 16 '18 at 22:57
DavidDavid
316
asked Nov 16 '18 at 22:57
DavidDavid
316
asked Nov 16 '18 at 22:57
DavidDavid
316
316
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
output filters for selected columns from m3. When you call duplicated on m3, all columns from the original dataframe are considered. When you call duplicated on output, only a subset of those columns is considered.
Therefore, you can have duplicates in output even when there are no duplicates in m3.
Here's a minimal and reproducible example of what you're seeing:
df = pd.DataFrame([[3, 8, 9], [4, 8, 9]])
print(df.duplicated().sum(), 'duplicates')
# 0 duplicates
df_filtered = df.loc[:, [1, 2]]
print(df_filtered.duplicated().sum(), 'duplicates')
# 1 duplicates
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
1
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346473%2fwhy-does-accessing-columns-of-a-pandas-dataframe-with-loc-produce-duplicate-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
output filters for selected columns from m3. When you call duplicated on m3, all columns from the original dataframe are considered. When you call duplicated on output, only a subset of those columns is considered.
Therefore, you can have duplicates in output even when there are no duplicates in m3.
Here's a minimal and reproducible example of what you're seeing:
df = pd.DataFrame([[3, 8, 9], [4, 8, 9]])
print(df.duplicated().sum(), 'duplicates')
# 0 duplicates
df_filtered = df.loc[:, [1, 2]]
print(df_filtered.duplicated().sum(), 'duplicates')
# 1 duplicates
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
1
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
add a comment |
output filters for selected columns from m3. When you call duplicated on m3, all columns from the original dataframe are considered. When you call duplicated on output, only a subset of those columns is considered.
Therefore, you can have duplicates in output even when there are no duplicates in m3.
Here's a minimal and reproducible example of what you're seeing:
df = pd.DataFrame([[3, 8, 9], [4, 8, 9]])
print(df.duplicated().sum(), 'duplicates')
# 0 duplicates
df_filtered = df.loc[:, [1, 2]]
print(df_filtered.duplicated().sum(), 'duplicates')
# 1 duplicates
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
1
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
add a comment |
output filters for selected columns from m3. When you call duplicated on m3, all columns from the original dataframe are considered. When you call duplicated on output, only a subset of those columns is considered.
Therefore, you can have duplicates in output even when there are no duplicates in m3.
Here's a minimal and reproducible example of what you're seeing:
df = pd.DataFrame([[3, 8, 9], [4, 8, 9]])
print(df.duplicated().sum(), 'duplicates')
# 0 duplicates
df_filtered = df.loc[:, [1, 2]]
print(df_filtered.duplicated().sum(), 'duplicates')
# 1 duplicates
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
output filters for selected columns from m3. When you call duplicated on m3, all columns from the original dataframe are considered. When you call duplicated on output, only a subset of those columns is considered.
Therefore, you can have duplicates in output even when there are no duplicates in m3.
Here's a minimal and reproducible example of what you're seeing:
df = pd.DataFrame([[3, 8, 9], [4, 8, 9]])
print(df.duplicated().sum(), 'duplicates')
# 0 duplicates
df_filtered = df.loc[:, [1, 2]]
print(df_filtered.duplicated().sum(), 'duplicates')
# 1 duplicates
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
answered Nov 16 '18 at 23:01
jppjpp
98.9k2160110
98.9k2160110
1
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
add a comment |
1
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
1
1
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
Thanks @jpp! I was looking at this for a solid hour and now I'm having a real "duh" moment. Like why didn't I see it sooner! Anyhow, I upvoted your answer too, I suppose it'll show when I have more reputation.
– David
Nov 16 '18 at 23:23
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346473%2fwhy-does-accessing-columns-of-a-pandas-dataframe-with-loc-produce-duplicate-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
