Couchbase Get operation slows down when the number of incoming threads increases
Summary:
We have a major performance issue with Spring-Boot 2.0.4
and Couchbase server
5.5.1
We are experiencing a rapid decline in DB response time performance when the number of threads is increasing. Here is another report about the issue.
In Details:
Spring Boot is running with 500 threads:
server:
tomcat:
max-threads: 500
max-connections: 500
We are using the following dependency:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-couchbase</artifactId>
<version>3.0.9.RELEASE</version>
</dependency>
Our "select" from DB is performed with Spring-Data repository:
Cat findFirstByOwnerIdAndNameAndColor(String ownerId, String name, String color);
We have an index that is especially for this query:
CREATE INDEX `cat_by_ownerId_name_and_color_idx` ON `pets`(`ownerId`,`name`,`color`) WHERE (`_class` = "com.example.Cat")
As the number of requests increase, we can see a quick degradation in the time it takes the DB to answer the query.
For example, when running 300 requests per second, the 99's percentile of response time is about 10 Seconds!! and the 50's percentile is about 5 seconds.
The average size of the returned document is about 300 Bytes. Meaning that we are trying to extract about 90 Kilobytes per second. A relatively low amount.
Edit:
I'm adding here the result of running the same query in the UI of Couchbase:
(In the UI, the query takes 1.75ms to complete).
{
"plan": {
"#operator": "Sequence",
"~children": [
{
"#operator": "IndexScan3",
"index": "cats_by_ownerId_name_and_color_idx",
"index_id": "c061141c2d373067",
"index_projection": {
"primary_key": true
},
"keyspace": "pets",
"namespace": "default",
"spans": [
{
"exact": true,
"range": [
{
"high": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5"",
"inclusion": 3,
"low": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5""
},
{
"high": ""Oscar"",
"inclusion": 3,
"low": ""Oscar""
},
{
"high": ""red"",
"inclusion": 3,
"low": ""red""
}
]
}
],
"using": "gsi"
},
{
"#operator": "Fetch",
"keyspace": "pets",
"namespace": "default"
},
{
"#operator": "Parallel",
"~child": {
"#operator": "Sequence",
"~children": [
{
"#operator": "Filter",
"condition": "(((((`pets`.`_class`) = "com.example.Cat") and ((`pets`.`ownerId`) = "bf23fa4c-22c3-42ac-b141-39cdc76bb2x5")) and ((`pets`.`name`) = "Oscar")) and ((`pets`.`color`) = "red"))"
},
{
"#operator": "InitialProject",
"result_terms": [
{
"expr": "self",
"star": true
}
]
},
{
"#operator": "FinalProject"
}
]
}
}
]
},
"text": "select * from pets where _class="com.example.Cat" and projectId="bf23fa4c-22c3-42ac-b141-39cdc76bb2x5" and name="Oscar" and color="red""
}
EDIT 2
We also tried to implicitly write the N1ql query, but the outcome is the same. As before, we get many TimeOutExceptions:
Servlet.service() for servlet [dispatcherServlet] in context with path threw exception [Request processing failed; nested exception is org.springframework.dao.QueryTimeoutException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}; nested exception is java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}] with root cause
java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131) ~[java-client-2.7.0.jar:na]
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127) ~[java-client-2.7.0.jar:na]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.3.8.jar:1.3.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Is there a way to fix this, or we need a different DB?
spring-boot spring-data couchbase
add a comment |
Summary:
We have a major performance issue with Spring-Boot 2.0.4
and Couchbase server
5.5.1
We are experiencing a rapid decline in DB response time performance when the number of threads is increasing. Here is another report about the issue.
In Details:
Spring Boot is running with 500 threads:
server:
tomcat:
max-threads: 500
max-connections: 500
We are using the following dependency:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-couchbase</artifactId>
<version>3.0.9.RELEASE</version>
</dependency>
Our "select" from DB is performed with Spring-Data repository:
Cat findFirstByOwnerIdAndNameAndColor(String ownerId, String name, String color);
We have an index that is especially for this query:
CREATE INDEX `cat_by_ownerId_name_and_color_idx` ON `pets`(`ownerId`,`name`,`color`) WHERE (`_class` = "com.example.Cat")
As the number of requests increase, we can see a quick degradation in the time it takes the DB to answer the query.
For example, when running 300 requests per second, the 99's percentile of response time is about 10 Seconds!! and the 50's percentile is about 5 seconds.
The average size of the returned document is about 300 Bytes. Meaning that we are trying to extract about 90 Kilobytes per second. A relatively low amount.
Edit:
I'm adding here the result of running the same query in the UI of Couchbase:
(In the UI, the query takes 1.75ms to complete).
{
"plan": {
"#operator": "Sequence",
"~children": [
{
"#operator": "IndexScan3",
"index": "cats_by_ownerId_name_and_color_idx",
"index_id": "c061141c2d373067",
"index_projection": {
"primary_key": true
},
"keyspace": "pets",
"namespace": "default",
"spans": [
{
"exact": true,
"range": [
{
"high": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5"",
"inclusion": 3,
"low": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5""
},
{
"high": ""Oscar"",
"inclusion": 3,
"low": ""Oscar""
},
{
"high": ""red"",
"inclusion": 3,
"low": ""red""
}
]
}
],
"using": "gsi"
},
{
"#operator": "Fetch",
"keyspace": "pets",
"namespace": "default"
},
{
"#operator": "Parallel",
"~child": {
"#operator": "Sequence",
"~children": [
{
"#operator": "Filter",
"condition": "(((((`pets`.`_class`) = "com.example.Cat") and ((`pets`.`ownerId`) = "bf23fa4c-22c3-42ac-b141-39cdc76bb2x5")) and ((`pets`.`name`) = "Oscar")) and ((`pets`.`color`) = "red"))"
},
{
"#operator": "InitialProject",
"result_terms": [
{
"expr": "self",
"star": true
}
]
},
{
"#operator": "FinalProject"
}
]
}
}
]
},
"text": "select * from pets where _class="com.example.Cat" and projectId="bf23fa4c-22c3-42ac-b141-39cdc76bb2x5" and name="Oscar" and color="red""
}
EDIT 2
We also tried to implicitly write the N1ql query, but the outcome is the same. As before, we get many TimeOutExceptions:
Servlet.service() for servlet [dispatcherServlet] in context with path threw exception [Request processing failed; nested exception is org.springframework.dao.QueryTimeoutException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}; nested exception is java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}] with root cause
java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131) ~[java-client-2.7.0.jar:na]
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127) ~[java-client-2.7.0.jar:na]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.3.8.jar:1.3.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Is there a way to fix this, or we need a different DB?
spring-boot spring-data couchbase
1
Couchbase can handle much more than that easily. I run in my notebook a load higher than that. What is the size of the machine you are running CB? Can you show some stats of the database during the peek?
– deniswsrosa
Nov 20 '18 at 11:04
@deniswsrosa According to the statistics page of Couchbase, the "Ops Per Second" range between 40 and 100 during the load test. The test runs on Dell Laptop with 16GB or RAM and the Couchbase DB in in a container. It is important to say that we see same behavior when running the code in other envs where the DB is in regular mode and not in container. Is is possible that additional Spring-Data configuration is needed?
– riorio
Nov 20 '18 at 11:18
1
Can you execute the same query on CB web console and update your post with the query plan?
– deniswsrosa
Nov 20 '18 at 12:17
@deniswsrosa added as you suggested
– riorio
Nov 20 '18 at 12:37
1
Everything looks fine, my guess is that your high concurrency is too much for a single query service. Add a second CB instance, and configure it to run in a cluster. (The query service must be running in both of them)
– deniswsrosa
Nov 20 '18 at 13:11
add a comment |
Summary:
We have a major performance issue with Spring-Boot 2.0.4
and Couchbase server
5.5.1
We are experiencing a rapid decline in DB response time performance when the number of threads is increasing. Here is another report about the issue.
In Details:
Spring Boot is running with 500 threads:
server:
tomcat:
max-threads: 500
max-connections: 500
We are using the following dependency:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-couchbase</artifactId>
<version>3.0.9.RELEASE</version>
</dependency>
Our "select" from DB is performed with Spring-Data repository:
Cat findFirstByOwnerIdAndNameAndColor(String ownerId, String name, String color);
We have an index that is especially for this query:
CREATE INDEX `cat_by_ownerId_name_and_color_idx` ON `pets`(`ownerId`,`name`,`color`) WHERE (`_class` = "com.example.Cat")
As the number of requests increase, we can see a quick degradation in the time it takes the DB to answer the query.
For example, when running 300 requests per second, the 99's percentile of response time is about 10 Seconds!! and the 50's percentile is about 5 seconds.
The average size of the returned document is about 300 Bytes. Meaning that we are trying to extract about 90 Kilobytes per second. A relatively low amount.
Edit:
I'm adding here the result of running the same query in the UI of Couchbase:
(In the UI, the query takes 1.75ms to complete).
{
"plan": {
"#operator": "Sequence",
"~children": [
{
"#operator": "IndexScan3",
"index": "cats_by_ownerId_name_and_color_idx",
"index_id": "c061141c2d373067",
"index_projection": {
"primary_key": true
},
"keyspace": "pets",
"namespace": "default",
"spans": [
{
"exact": true,
"range": [
{
"high": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5"",
"inclusion": 3,
"low": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5""
},
{
"high": ""Oscar"",
"inclusion": 3,
"low": ""Oscar""
},
{
"high": ""red"",
"inclusion": 3,
"low": ""red""
}
]
}
],
"using": "gsi"
},
{
"#operator": "Fetch",
"keyspace": "pets",
"namespace": "default"
},
{
"#operator": "Parallel",
"~child": {
"#operator": "Sequence",
"~children": [
{
"#operator": "Filter",
"condition": "(((((`pets`.`_class`) = "com.example.Cat") and ((`pets`.`ownerId`) = "bf23fa4c-22c3-42ac-b141-39cdc76bb2x5")) and ((`pets`.`name`) = "Oscar")) and ((`pets`.`color`) = "red"))"
},
{
"#operator": "InitialProject",
"result_terms": [
{
"expr": "self",
"star": true
}
]
},
{
"#operator": "FinalProject"
}
]
}
}
]
},
"text": "select * from pets where _class="com.example.Cat" and projectId="bf23fa4c-22c3-42ac-b141-39cdc76bb2x5" and name="Oscar" and color="red""
}
EDIT 2
We also tried to implicitly write the N1ql query, but the outcome is the same. As before, we get many TimeOutExceptions:
Servlet.service() for servlet [dispatcherServlet] in context with path threw exception [Request processing failed; nested exception is org.springframework.dao.QueryTimeoutException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}; nested exception is java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}] with root cause
java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131) ~[java-client-2.7.0.jar:na]
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127) ~[java-client-2.7.0.jar:na]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.3.8.jar:1.3.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Is there a way to fix this, or we need a different DB?
spring-boot spring-data couchbase
Summary:
We have a major performance issue with Spring-Boot 2.0.4
and Couchbase server
5.5.1
We are experiencing a rapid decline in DB response time performance when the number of threads is increasing. Here is another report about the issue.
In Details:
Spring Boot is running with 500 threads:
server:
tomcat:
max-threads: 500
max-connections: 500
We are using the following dependency:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-couchbase</artifactId>
<version>3.0.9.RELEASE</version>
</dependency>
Our "select" from DB is performed with Spring-Data repository:
Cat findFirstByOwnerIdAndNameAndColor(String ownerId, String name, String color);
We have an index that is especially for this query:
CREATE INDEX `cat_by_ownerId_name_and_color_idx` ON `pets`(`ownerId`,`name`,`color`) WHERE (`_class` = "com.example.Cat")
As the number of requests increase, we can see a quick degradation in the time it takes the DB to answer the query.
For example, when running 300 requests per second, the 99's percentile of response time is about 10 Seconds!! and the 50's percentile is about 5 seconds.
The average size of the returned document is about 300 Bytes. Meaning that we are trying to extract about 90 Kilobytes per second. A relatively low amount.
Edit:
I'm adding here the result of running the same query in the UI of Couchbase:
(In the UI, the query takes 1.75ms to complete).
{
"plan": {
"#operator": "Sequence",
"~children": [
{
"#operator": "IndexScan3",
"index": "cats_by_ownerId_name_and_color_idx",
"index_id": "c061141c2d373067",
"index_projection": {
"primary_key": true
},
"keyspace": "pets",
"namespace": "default",
"spans": [
{
"exact": true,
"range": [
{
"high": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5"",
"inclusion": 3,
"low": ""bf23fa4c-22c3-42ac-b141-39cdc76bb2x5""
},
{
"high": ""Oscar"",
"inclusion": 3,
"low": ""Oscar""
},
{
"high": ""red"",
"inclusion": 3,
"low": ""red""
}
]
}
],
"using": "gsi"
},
{
"#operator": "Fetch",
"keyspace": "pets",
"namespace": "default"
},
{
"#operator": "Parallel",
"~child": {
"#operator": "Sequence",
"~children": [
{
"#operator": "Filter",
"condition": "(((((`pets`.`_class`) = "com.example.Cat") and ((`pets`.`ownerId`) = "bf23fa4c-22c3-42ac-b141-39cdc76bb2x5")) and ((`pets`.`name`) = "Oscar")) and ((`pets`.`color`) = "red"))"
},
{
"#operator": "InitialProject",
"result_terms": [
{
"expr": "self",
"star": true
}
]
},
{
"#operator": "FinalProject"
}
]
}
}
]
},
"text": "select * from pets where _class="com.example.Cat" and projectId="bf23fa4c-22c3-42ac-b141-39cdc76bb2x5" and name="Oscar" and color="red""
}
EDIT 2
We also tried to implicitly write the N1ql query, but the outcome is the same. As before, we get many TimeOutExceptions:
Servlet.service() for servlet [dispatcherServlet] in context with path threw exception [Request processing failed; nested exception is org.springframework.dao.QueryTimeoutException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}; nested exception is java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}] with root cause
java.util.concurrent.TimeoutException: {"b":"pets","s":"n1ql","t":7500000,"i":"f8cdf670-d32a-4d74-858c-f9dd9789d264"}
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131) ~[java-client-2.7.0.jar:na]
at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127) ~[java-client-2.7.0.jar:na]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.3.8.jar:1.3.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Is there a way to fix this, or we need a different DB?
spring-boot spring-data couchbase
spring-boot spring-data couchbase
edited Nov 21 '18 at 7:01
riorio
asked Nov 20 '18 at 9:02
riorioriorio
1,89831034
1,89831034
1
Couchbase can handle much more than that easily. I run in my notebook a load higher than that. What is the size of the machine you are running CB? Can you show some stats of the database during the peek?
– deniswsrosa
Nov 20 '18 at 11:04
@deniswsrosa According to the statistics page of Couchbase, the "Ops Per Second" range between 40 and 100 during the load test. The test runs on Dell Laptop with 16GB or RAM and the Couchbase DB in in a container. It is important to say that we see same behavior when running the code in other envs where the DB is in regular mode and not in container. Is is possible that additional Spring-Data configuration is needed?
– riorio
Nov 20 '18 at 11:18
1
Can you execute the same query on CB web console and update your post with the query plan?
– deniswsrosa
Nov 20 '18 at 12:17
@deniswsrosa added as you suggested
– riorio
Nov 20 '18 at 12:37
1
Everything looks fine, my guess is that your high concurrency is too much for a single query service. Add a second CB instance, and configure it to run in a cluster. (The query service must be running in both of them)
– deniswsrosa
Nov 20 '18 at 13:11
add a comment |
1
Couchbase can handle much more than that easily. I run in my notebook a load higher than that. What is the size of the machine you are running CB? Can you show some stats of the database during the peek?
– deniswsrosa
Nov 20 '18 at 11:04
@deniswsrosa According to the statistics page of Couchbase, the "Ops Per Second" range between 40 and 100 during the load test. The test runs on Dell Laptop with 16GB or RAM and the Couchbase DB in in a container. It is important to say that we see same behavior when running the code in other envs where the DB is in regular mode and not in container. Is is possible that additional Spring-Data configuration is needed?
– riorio
Nov 20 '18 at 11:18
1
Can you execute the same query on CB web console and update your post with the query plan?
– deniswsrosa
Nov 20 '18 at 12:17
@deniswsrosa added as you suggested
– riorio
Nov 20 '18 at 12:37
1
Everything looks fine, my guess is that your high concurrency is too much for a single query service. Add a second CB instance, and configure it to run in a cluster. (The query service must be running in both of them)
– deniswsrosa
Nov 20 '18 at 13:11
1
1
Couchbase can handle much more than that easily. I run in my notebook a load higher than that. What is the size of the machine you are running CB? Can you show some stats of the database during the peek?
– deniswsrosa
Nov 20 '18 at 11:04
Couchbase can handle much more than that easily. I run in my notebook a load higher than that. What is the size of the machine you are running CB? Can you show some stats of the database during the peek?
– deniswsrosa
Nov 20 '18 at 11:04
@deniswsrosa According to the statistics page of Couchbase, the "Ops Per Second" range between 40 and 100 during the load test. The test runs on Dell Laptop with 16GB or RAM and the Couchbase DB in in a container. It is important to say that we see same behavior when running the code in other envs where the DB is in regular mode and not in container. Is is possible that additional Spring-Data configuration is needed?
– riorio
Nov 20 '18 at 11:18
@deniswsrosa According to the statistics page of Couchbase, the "Ops Per Second" range between 40 and 100 during the load test. The test runs on Dell Laptop with 16GB or RAM and the Couchbase DB in in a container. It is important to say that we see same behavior when running the code in other envs where the DB is in regular mode and not in container. Is is possible that additional Spring-Data configuration is needed?
– riorio
Nov 20 '18 at 11:18
1
1
Can you execute the same query on CB web console and update your post with the query plan?
– deniswsrosa
Nov 20 '18 at 12:17
Can you execute the same query on CB web console and update your post with the query plan?
– deniswsrosa
Nov 20 '18 at 12:17
@deniswsrosa added as you suggested
– riorio
Nov 20 '18 at 12:37
@deniswsrosa added as you suggested
– riorio
Nov 20 '18 at 12:37
1
1
Everything looks fine, my guess is that your high concurrency is too much for a single query service. Add a second CB instance, and configure it to run in a cluster. (The query service must be running in both of them)
– deniswsrosa
Nov 20 '18 at 13:11
Everything looks fine, my guess is that your high concurrency is too much for a single query service. Add a second CB instance, and configure it to run in a cluster. (The query service must be running in both of them)
– deniswsrosa
Nov 20 '18 at 13:11
add a comment |
1 Answer
1
active
oldest
votes
So after further investigation, the problem was found in the Spring-Data
component.
To over come it, we had to move to non-blocking mechanism.
We did 2 things:
- All the calls from controller layer down to service & repository layers, were changed to
CompleteableFuture<Cat>
To bypass Spring-Data connection to the couchbase, we created a repository class of our own with implementation code that looks something like that:
Statement statement = select("*")
.from(i(bucket.name()))
.where(x("name").eq(s(name))
.and(x("ownerId").eq(s(ownerId)))
.and(x("color").eq(s(color)))
.and(x("_class").eq(s("com.example.Cat"))));
CompletableFuture<Cat> completableFuture = new CompletableFuture();
bucket.async().query(statement)
...
After we did that, the latency problem disappeared and the performance are about 2 Milliseconds for query, even during about few hundreds concurrent requests.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53389476%2fcouchbase-get-operation-slows-down-when-the-number-of-incoming-threads-increases%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
So after further investigation, the problem was found in the Spring-Data
component.
To over come it, we had to move to non-blocking mechanism.
We did 2 things:
- All the calls from controller layer down to service & repository layers, were changed to
CompleteableFuture<Cat>
To bypass Spring-Data connection to the couchbase, we created a repository class of our own with implementation code that looks something like that:
Statement statement = select("*")
.from(i(bucket.name()))
.where(x("name").eq(s(name))
.and(x("ownerId").eq(s(ownerId)))
.and(x("color").eq(s(color)))
.and(x("_class").eq(s("com.example.Cat"))));
CompletableFuture<Cat> completableFuture = new CompletableFuture();
bucket.async().query(statement)
...
After we did that, the latency problem disappeared and the performance are about 2 Milliseconds for query, even during about few hundreds concurrent requests.
add a comment |
So after further investigation, the problem was found in the Spring-Data
component.
To over come it, we had to move to non-blocking mechanism.
We did 2 things:
- All the calls from controller layer down to service & repository layers, were changed to
CompleteableFuture<Cat>
To bypass Spring-Data connection to the couchbase, we created a repository class of our own with implementation code that looks something like that:
Statement statement = select("*")
.from(i(bucket.name()))
.where(x("name").eq(s(name))
.and(x("ownerId").eq(s(ownerId)))
.and(x("color").eq(s(color)))
.and(x("_class").eq(s("com.example.Cat"))));
CompletableFuture<Cat> completableFuture = new CompletableFuture();
bucket.async().query(statement)
...
After we did that, the latency problem disappeared and the performance are about 2 Milliseconds for query, even during about few hundreds concurrent requests.
add a comment |
So after further investigation, the problem was found in the Spring-Data
component.
To over come it, we had to move to non-blocking mechanism.
We did 2 things:
- All the calls from controller layer down to service & repository layers, were changed to
CompleteableFuture<Cat>
To bypass Spring-Data connection to the couchbase, we created a repository class of our own with implementation code that looks something like that:
Statement statement = select("*")
.from(i(bucket.name()))
.where(x("name").eq(s(name))
.and(x("ownerId").eq(s(ownerId)))
.and(x("color").eq(s(color)))
.and(x("_class").eq(s("com.example.Cat"))));
CompletableFuture<Cat> completableFuture = new CompletableFuture();
bucket.async().query(statement)
...
After we did that, the latency problem disappeared and the performance are about 2 Milliseconds for query, even during about few hundreds concurrent requests.
So after further investigation, the problem was found in the Spring-Data
component.
To over come it, we had to move to non-blocking mechanism.
We did 2 things:
- All the calls from controller layer down to service & repository layers, were changed to
CompleteableFuture<Cat>
To bypass Spring-Data connection to the couchbase, we created a repository class of our own with implementation code that looks something like that:
Statement statement = select("*")
.from(i(bucket.name()))
.where(x("name").eq(s(name))
.and(x("ownerId").eq(s(ownerId)))
.and(x("color").eq(s(color)))
.and(x("_class").eq(s("com.example.Cat"))));
CompletableFuture<Cat> completableFuture = new CompletableFuture();
bucket.async().query(statement)
...
After we did that, the latency problem disappeared and the performance are about 2 Milliseconds for query, even during about few hundreds concurrent requests.
edited Nov 26 '18 at 7:03
answered Nov 25 '18 at 11:48
riorioriorio
1,89831034
1,89831034
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53389476%2fcouchbase-get-operation-slows-down-when-the-number-of-incoming-threads-increases%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Couchbase can handle much more than that easily. I run in my notebook a load higher than that. What is the size of the machine you are running CB? Can you show some stats of the database during the peek?
– deniswsrosa
Nov 20 '18 at 11:04
@deniswsrosa According to the statistics page of Couchbase, the "Ops Per Second" range between 40 and 100 during the load test. The test runs on Dell Laptop with 16GB or RAM and the Couchbase DB in in a container. It is important to say that we see same behavior when running the code in other envs where the DB is in regular mode and not in container. Is is possible that additional Spring-Data configuration is needed?
– riorio
Nov 20 '18 at 11:18
1
Can you execute the same query on CB web console and update your post with the query plan?
– deniswsrosa
Nov 20 '18 at 12:17
@deniswsrosa added as you suggested
– riorio
Nov 20 '18 at 12:37
1
Everything looks fine, my guess is that your high concurrency is too much for a single query service. Add a second CB instance, and configure it to run in a cluster. (The query service must be running in both of them)
– deniswsrosa
Nov 20 '18 at 13:11