Performance Issue with Spring BOOT 1.5.6

We are using a spring boot java based REST API application where we have the below spring MVC async parameters. Under heavy load when the endpoint is tested the endpoint is returning API response avg of 30-50 seconds. This is happening when we have a sudden burst of 10 minutes. Our ideal time for the API response 75% percentile is between 1- 2 seconds. Below is the configuration, we are using 6 C5x large instances having 4 cores each/per instance.

spring.mvc.async.properties.web.executor.minPoolSize=50

spring.mvc.async.properties.web.executor.maxPoolSize=100

spring.mvc.async.properties.web.executor.maxQueueSize=50

#Hikari Data source properties.

spring.datasource.hikari.minimumIdle=25

spring.datasource.hikari.maximumPoolSize=90

spring.datasource.hikari.idleTimeout=600000

Appreciate for any scalability suggestions.

Also we also identified in few calls that dB calls are taking time and we are trying to find out if anything need to be fined tuned in the query but I think the threads are waiting on the dB response.Also with async thread executor with policy as discard policy is there any chance of rejecting any task submitted ? Iam expecting the tasks to be queued instead of rejecting under load .we moved away from callerRuns policy to discard policy.Any thoughts on that or anything else required from spring boot side or from thread pool size execution side ? Thanks

I think as a first resort you should try to identify the bottleneck of this flow

The key tool for this is metrics.

I see that you use Hikari here and it exposes metrics automatically by its own.
Maybe the Database works hard and it becomes a bottleneck, in this case, it will take a relatively long time to acquire a DB connection from the pool.

Another possible issue can be if the actual requests to the service carry a lot of content with it (maybe its a "big file upload" operation, I don't know whether its a case, but still worth checking).

So I suggest using metrics (built-in or custom). Spring boot has an excellent integration with Metering systems (Micrometer for spring boot 2 and dropwizard metrics for spring boot 1.x)

Thnsks Mark for your reply. That’s correct Iam looking at the metrics exposed by micrometer with /metrics. Also we just modified to see how it works from bounded queue to unbounded queue by only specifying the core pool size to 100 and removing all the above mentioned custom tuning of the min pool size and max pool size. The DB connections we have right now are 90*6 instances. Under load it’s using only till 300+ connections. The service returns a payload between 1.8kb to max of 2MB.