How many shards per node are appropriate for elasticsearch?
0
If there are three nodes, is it reasonable to run ten indexes with three shards?
Or is it reasonable to have 10 indexes with two shards?
(Replication 1)
Because it is a date-based index, i plan to create a snapshot when the capacity is exceeded.
elasticsearch
asked Nov 13 '18 at 14:44
김태우김태우
869
add a comment |
0
If there are three nodes, is it reasonable to run ten indexes with three shards?
Or is it reasonable to have 10 indexes with two shards?
(Replication 1)
Because it is a date-based index, i plan to create a snapshot when the capacity is exceeded.
elasticsearch
asked Nov 13 '18 at 14:44
김태우김태우
869
1
That all depends on the load and uptime you need to achieve. Less shards you have - less redundancy you have. so if you have 3 nodes, I would say 2 shards is reasonable number. But you should set up 1 node to be read only for example. Number of indexes - is not enough to make any decision. How big are indexes? 1Gb? 10Gb? 10Gb? 1Tb? that is the question. How powerful are your nodes? How many clients knock to elastic to get data? etc...
– Alex
Nov 13 '18 at 14:54
1
Alex is right about the number of indices, But there is probaly a misunderstanding of sharding. Please be aware that sharding has nothing to do with realibility. Shrading is just about distributing your data into several lucene indices (shards). If done right, it can improve your performance. But loosing a node means loosing your shards on this node if there a no replicas available. So if you want realibility, you should have at least one replica. In this way, if a primary shard is lost, the cluster will be able to route the requests to the replica shard and still serve further requests.
– ibexit
Nov 13 '18 at 16:41
1
please have also a look on this question I answered few days ago. This should help you to choose a good number of shards per index: stackoverflow.com/questions/53214628/…
– ibexit
Nov 13 '18 at 16:45
Thank you. @Alex Each shard is for logging for 40 Gb in a week. Data is input from a number of logstash similar to elasticsearch.
– 김태우
Nov 14 '18 at 0:55
If you use a high-performance server computer, i will have fewer nodes, and i can use many low-performance servers. I think a high performance server is more efficient.
– 김태우
Nov 14 '18 at 2:19
add a comment |
0
0
0
If there are three nodes, is it reasonable to run ten indexes with three shards?
Or is it reasonable to have 10 indexes with two shards?
(Replication 1)
Because it is a date-based index, i plan to create a snapshot when the capacity is exceeded.
elasticsearch
asked Nov 13 '18 at 14:44
김태우김태우
869
If there are three nodes, is it reasonable to run ten indexes with three shards?
Or is it reasonable to have 10 indexes with two shards?
(Replication 1)
Because it is a date-based index, i plan to create a snapshot when the capacity is exceeded.
elasticsearch
elasticsearch
asked Nov 13 '18 at 14:44
김태우김태우
869
asked Nov 13 '18 at 14:44
김태우김태우
869
asked Nov 13 '18 at 14:44
김태우김태우
869
asked Nov 13 '18 at 14:44
김태우김태우
869
asked Nov 13 '18 at 14:44
김태우김태우
869
869
1
That all depends on the load and uptime you need to achieve. Less shards you have - less redundancy you have. so if you have 3 nodes, I would say 2 shards is reasonable number. But you should set up 1 node to be read only for example. Number of indexes - is not enough to make any decision. How big are indexes? 1Gb? 10Gb? 10Gb? 1Tb? that is the question. How powerful are your nodes? How many clients knock to elastic to get data? etc...
– Alex
Nov 13 '18 at 14:54
1
Alex is right about the number of indices, But there is probaly a misunderstanding of sharding. Please be aware that sharding has nothing to do with realibility. Shrading is just about distributing your data into several lucene indices (shards). If done right, it can improve your performance. But loosing a node means loosing your shards on this node if there a no replicas available. So if you want realibility, you should have at least one replica. In this way, if a primary shard is lost, the cluster will be able to route the requests to the replica shard and still serve further requests.
– ibexit
Nov 13 '18 at 16:41
1
please have also a look on this question I answered few days ago. This should help you to choose a good number of shards per index: stackoverflow.com/questions/53214628/…
– ibexit
Nov 13 '18 at 16:45
Thank you. @Alex Each shard is for logging for 40 Gb in a week. Data is input from a number of logstash similar to elasticsearch.
– 김태우
Nov 14 '18 at 0:55
If you use a high-performance server computer, i will have fewer nodes, and i can use many low-performance servers. I think a high performance server is more efficient.
– 김태우
Nov 14 '18 at 2:19
add a comment |
1
That all depends on the load and uptime you need to achieve. Less shards you have - less redundancy you have. so if you have 3 nodes, I would say 2 shards is reasonable number. But you should set up 1 node to be read only for example. Number of indexes - is not enough to make any decision. How big are indexes? 1Gb? 10Gb? 10Gb? 1Tb? that is the question. How powerful are your nodes? How many clients knock to elastic to get data? etc...
– Alex
Nov 13 '18 at 14:54
1
Alex is right about the number of indices, But there is probaly a misunderstanding of sharding. Please be aware that sharding has nothing to do with realibility. Shrading is just about distributing your data into several lucene indices (shards). If done right, it can improve your performance. But loosing a node means loosing your shards on this node if there a no replicas available. So if you want realibility, you should have at least one replica. In this way, if a primary shard is lost, the cluster will be able to route the requests to the replica shard and still serve further requests.
– ibexit
Nov 13 '18 at 16:41
1
please have also a look on this question I answered few days ago. This should help you to choose a good number of shards per index: stackoverflow.com/questions/53214628/…
– ibexit
Nov 13 '18 at 16:45
Thank you. @Alex Each shard is for logging for 40 Gb in a week. Data is input from a number of logstash similar to elasticsearch.
– 김태우
Nov 14 '18 at 0:55
If you use a high-performance server computer, i will have fewer nodes, and i can use many low-performance servers. I think a high performance server is more efficient.
– 김태우
Nov 14 '18 at 2:19
1
1
That all depends on the load and uptime you need to achieve. Less shards you have - less redundancy you have. so if you have 3 nodes, I would say 2 shards is reasonable number. But you should set up 1 node to be read only for example. Number of indexes - is not enough to make any decision. How big are indexes? 1Gb? 10Gb? 10Gb? 1Tb? that is the question. How powerful are your nodes? How many clients knock to elastic to get data? etc...
– Alex
Nov 13 '18 at 14:54
That all depends on the load and uptime you need to achieve. Less shards you have - less redundancy you have. so if you have 3 nodes, I would say 2 shards is reasonable number. But you should set up 1 node to be read only for example. Number of indexes - is not enough to make any decision. How big are indexes? 1Gb? 10Gb? 10Gb? 1Tb? that is the question. How powerful are your nodes? How many clients knock to elastic to get data? etc...
– Alex
Nov 13 '18 at 14:54
1
1
Alex is right about the number of indices, But there is probaly a misunderstanding of sharding. Please be aware that sharding has nothing to do with realibility. Shrading is just about distributing your data into several lucene indices (shards). If done right, it can improve your performance. But loosing a node means loosing your shards on this node if there a no replicas available. So if you want realibility, you should have at least one replica. In this way, if a primary shard is lost, the cluster will be able to route the requests to the replica shard and still serve further requests.
– ibexit
Nov 13 '18 at 16:41
Alex is right about the number of indices, But there is probaly a misunderstanding of sharding. Please be aware that sharding has nothing to do with realibility. Shrading is just about distributing your data into several lucene indices (shards). If done right, it can improve your performance. But loosing a node means loosing your shards on this node if there a no replicas available. So if you want realibility, you should have at least one replica. In this way, if a primary shard is lost, the cluster will be able to route the requests to the replica shard and still serve further requests.
– ibexit
Nov 13 '18 at 16:41
1
1
please have also a look on this question I answered few days ago. This should help you to choose a good number of shards per index: stackoverflow.com/questions/53214628/…
– ibexit
Nov 13 '18 at 16:45
please have also a look on this question I answered few days ago. This should help you to choose a good number of shards per index: stackoverflow.com/questions/53214628/…
– ibexit
Nov 13 '18 at 16:45
Thank you. @Alex Each shard is for logging for 40 Gb in a week. Data is input from a number of logstash similar to elasticsearch.
– 김태우
Nov 14 '18 at 0:55
Thank you. @Alex Each shard is for logging for 40 Gb in a week. Data is input from a number of logstash similar to elasticsearch.
– 김태우
Nov 14 '18 at 0:55
If you use a high-performance server computer, i will have fewer nodes, and i can use many low-performance servers. I think a high performance server is more efficient.
– 김태우
Nov 14 '18 at 2:19
If you use a high-performance server computer, i will have fewer nodes, and i can use many low-performance servers. I think a high performance server is more efficient.
– 김태우
Nov 14 '18 at 2:19
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53283529%2fhow-many-shards-per-node-are-appropriate-for-elasticsearch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53283529%2fhow-many-shards-per-node-are-appropriate-for-elasticsearch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
That all depends on the load and uptime you need to achieve. Less shards you have - less redundancy you have. so if you have 3 nodes, I would say 2 shards is reasonable number. But you should set up 1 node to be read only for example. Number of indexes - is not enough to make any decision. How big are indexes? 1Gb? 10Gb? 10Gb? 1Tb? that is the question. How powerful are your nodes? How many clients knock to elastic to get data? etc...
– Alex
Nov 13 '18 at 14:54
1
Alex is right about the number of indices, But there is probaly a misunderstanding of sharding. Please be aware that sharding has nothing to do with realibility. Shrading is just about distributing your data into several lucene indices (shards). If done right, it can improve your performance. But loosing a node means loosing your shards on this node if there a no replicas available. So if you want realibility, you should have at least one replica. In this way, if a primary shard is lost, the cluster will be able to route the requests to the replica shard and still serve further requests.
– ibexit
Nov 13 '18 at 16:41
1
please have also a look on this question I answered few days ago. This should help you to choose a good number of shards per index: stackoverflow.com/questions/53214628/…
– ibexit
Nov 13 '18 at 16:45
Thank you. @Alex Each shard is for logging for 40 Gb in a week. Data is input from a number of logstash similar to elasticsearch.
– 김태우
Nov 14 '18 at 0:55
If you use a high-performance server computer, i will have fewer nodes, and i can use many low-performance servers. I think a high performance server is more efficient.
– 김태우
Nov 14 '18 at 2:19