pandas dataframe O(1) index by column
0
I have a large dataframe and want to support O(1) index by one column. I tried set_index but that seems to be not that fast when using with iloc. Any thoughts on how to speed this up?
python pandas
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
add a comment |
0
I have a large dataframe and want to support O(1) index by one column. I tried set_index but that seems to be not that fast when using with iloc. Any thoughts on how to speed this up?
python pandas
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
4
Not sure what you mean bysupport O(1) index by one column
– RafaelC
Nov 20 '18 at 1:25
Pandas indices are effectively hash tables.pd.Indexcan only contain hashable items. One way to get a bit more speedup is by making sure the index doesn't contain duplicates. I believe the getitem lookup is O(1) when both of those conditions are satisfied
– Brad Solomon
Nov 20 '18 at 2:07
add a comment |
0
0
0
I have a large dataframe and want to support O(1) index by one column. I tried set_index but that seems to be not that fast when using with iloc. Any thoughts on how to speed this up?
python pandas
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
I have a large dataframe and want to support O(1) index by one column. I tried set_index but that seems to be not that fast when using with iloc. Any thoughts on how to speed this up?
python pandas
python pandas
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
asked Nov 20 '18 at 1:20
Mr.cyslMr.cysl
312416
312416
4
Not sure what you mean bysupport O(1) index by one column
– RafaelC
Nov 20 '18 at 1:25
Pandas indices are effectively hash tables.pd.Indexcan only contain hashable items. One way to get a bit more speedup is by making sure the index doesn't contain duplicates. I believe the getitem lookup is O(1) when both of those conditions are satisfied
– Brad Solomon
Nov 20 '18 at 2:07
add a comment |
4
Not sure what you mean bysupport O(1) index by one column
– RafaelC
Nov 20 '18 at 1:25
Pandas indices are effectively hash tables.pd.Indexcan only contain hashable items. One way to get a bit more speedup is by making sure the index doesn't contain duplicates. I believe the getitem lookup is O(1) when both of those conditions are satisfied
– Brad Solomon
Nov 20 '18 at 2:07
4
4
Not sure what you mean by
support O(1) index by one column– RafaelC
Nov 20 '18 at 1:25
Not sure what you mean by
support O(1) index by one column– RafaelC
Nov 20 '18 at 1:25
Pandas indices are effectively hash tables.
pd.Index can only contain hashable items. One way to get a bit more speedup is by making sure the index doesn't contain duplicates. I believe the getitem lookup is O(1) when both of those conditions are satisfied– Brad Solomon
Nov 20 '18 at 2:07
Pandas indices are effectively hash tables.
pd.Index can only contain hashable items. One way to get a bit more speedup is by making sure the index doesn't contain duplicates. I believe the getitem lookup is O(1) when both of those conditions are satisfied– Brad Solomon
Nov 20 '18 at 2:07
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53384911%2fpandas-dataframe-o1-index-by-column%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53384911%2fpandas-dataframe-o1-index-by-column%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
4
Not sure what you mean by
support O(1) index by one column– RafaelC
Nov 20 '18 at 1:25
Pandas indices are effectively hash tables.
pd.Indexcan only contain hashable items. One way to get a bit more speedup is by making sure the index doesn't contain duplicates. I believe the getitem lookup is O(1) when both of those conditions are satisfied– Brad Solomon
Nov 20 '18 at 2:07