Third party api monitoring on AWS
up vote
0
down vote
favorite
My application consumes a large amount of APIs and also have webhooks it calls. Failure to detect outage of one API could go unnoticed and leads to bad user experience.
I would like to log the requests and responses as well as the exception that occured. My first instinct was to push this log to elasticsearch. But I wanted to utilize a more dynamic tooling on AWS.
I need two things from this monitoring infrastructure.
- I need the graphs to show me the performance of the service.
I should be able to see the count of transaction that are happening
against the total failure count. - I need to alert by taking previous data into consideration. It could
be Monday compared to the past 4 Mondays per service. - I need to be able to drill down and see the raw logs and search
through them
I have a couple of things in mind, and I would like to get suggestions.
Push everything to elasticsearch
I can push everything to elasticsearch and create graphs on Kibana or Grafana (Grafana seems more fit for the job).
I figured out how to graph it on Kibana but comparing against the past 4 Mondays was a challenge. Which made me think there must be better tools
Push everything to Cloud watch
I didn't try this one yet, I'm not sure if this has a different capability than elasticsearch.
Kinesis Analytics
This seems to be a good fit the analytics part but persisting and drill down reports seem to be hard if this is in the path. I could push the Kinesis stream to both Analytics and ES but I'm not sure if the cost is worth it.
I appreciate any help
amazon-web-services
elasticsearch amazon-kinesis asked Nov 8 at 6:32
Amanuel Nega
9211229
add a comment |
up vote
0
down vote
favorite
My application consumes a large amount of APIs and also have webhooks it calls. Failure to detect outage of one API could go unnoticed and leads to bad user experience.
I would like to log the requests and responses as well as the exception that occured. My first instinct was to push this log to elasticsearch. But I wanted to utilize a more dynamic tooling on AWS.
I need two things from this monitoring infrastructure.
- I need the graphs to show me the performance of the service.
I should be able to see the count of transaction that are happening
against the total failure count. - I need to alert by taking previous data into consideration. It could
be Monday compared to the past 4 Mondays per service. - I need to be able to drill down and see the raw logs and search
through them
I have a couple of things in mind, and I would like to get suggestions.
Push everything to elasticsearch
I can push everything to elasticsearch and create graphs on Kibana or Grafana (Grafana seems more fit for the job).
I figured out how to graph it on Kibana but comparing against the past 4 Mondays was a challenge. Which made me think there must be better tools
Push everything to Cloud watch
I didn't try this one yet, I'm not sure if this has a different capability than elasticsearch.
Kinesis Analytics
This seems to be a good fit the analytics part but persisting and drill down reports seem to be hard if this is in the path. I could push the Kinesis stream to both Analytics and ES but I'm not sure if the cost is worth it.
I appreciate any help
amazon-web-services
elasticsearch amazon-kinesis asked Nov 8 at 6:32
Amanuel Nega
9211229
@ who ever flaged this for closing, It's not opinion based, I'm asking for suggestions on ways to do it. I'm reaching out to experienced people to enlighten me
– Amanuel Nega
Nov 9 at 8:58
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
My application consumes a large amount of APIs and also have webhooks it calls. Failure to detect outage of one API could go unnoticed and leads to bad user experience.
I would like to log the requests and responses as well as the exception that occured. My first instinct was to push this log to elasticsearch. But I wanted to utilize a more dynamic tooling on AWS.
I need two things from this monitoring infrastructure.
- I need the graphs to show me the performance of the service.
I should be able to see the count of transaction that are happening
against the total failure count. - I need to alert by taking previous data into consideration. It could
be Monday compared to the past 4 Mondays per service. - I need to be able to drill down and see the raw logs and search
through them
I have a couple of things in mind, and I would like to get suggestions.
Push everything to elasticsearch
I can push everything to elasticsearch and create graphs on Kibana or Grafana (Grafana seems more fit for the job).
I figured out how to graph it on Kibana but comparing against the past 4 Mondays was a challenge. Which made me think there must be better tools
Push everything to Cloud watch
I didn't try this one yet, I'm not sure if this has a different capability than elasticsearch.
Kinesis Analytics
This seems to be a good fit the analytics part but persisting and drill down reports seem to be hard if this is in the path. I could push the Kinesis stream to both Analytics and ES but I'm not sure if the cost is worth it.
I appreciate any help
amazon-web-services
elasticsearch amazon-kinesis asked Nov 8 at 6:32
Amanuel Nega
9211229
My application consumes a large amount of APIs and also have webhooks it calls. Failure to detect outage of one API could go unnoticed and leads to bad user experience.
I would like to log the requests and responses as well as the exception that occured. My first instinct was to push this log to elasticsearch. But I wanted to utilize a more dynamic tooling on AWS.
I need two things from this monitoring infrastructure.
- I need the graphs to show me the performance of the service.
I should be able to see the count of transaction that are happening
against the total failure count. - I need to alert by taking previous data into consideration. It could
be Monday compared to the past 4 Mondays per service. - I need to be able to drill down and see the raw logs and search
through them
I have a couple of things in mind, and I would like to get suggestions.
Push everything to elasticsearch
I can push everything to elasticsearch and create graphs on Kibana or Grafana (Grafana seems more fit for the job).
I figured out how to graph it on Kibana but comparing against the past 4 Mondays was a challenge. Which made me think there must be better tools
Push everything to Cloud watch
I didn't try this one yet, I'm not sure if this has a different capability than elasticsearch.
Kinesis Analytics
This seems to be a good fit the analytics part but persisting and drill down reports seem to be hard if this is in the path. I could push the Kinesis stream to both Analytics and ES but I'm not sure if the cost is worth it.
I appreciate any help
amazon-web-services
elasticsearch amazon-kinesis amazon-web-services
elasticsearch amazon-kinesis asked Nov 8 at 6:32
Amanuel Nega
9211229
asked Nov 8 at 6:32
Amanuel Nega
9211229
asked Nov 8 at 6:32
Amanuel Nega
9211229
asked Nov 8 at 6:32
Amanuel Nega
9211229
asked Nov 8 at 6:32
Amanuel Nega
9211229
9211229
@ who ever flaged this for closing, It's not opinion based, I'm asking for suggestions on ways to do it. I'm reaching out to experienced people to enlighten me
– Amanuel Nega
Nov 9 at 8:58
add a comment |
@ who ever flaged this for closing, It's not opinion based, I'm asking for suggestions on ways to do it. I'm reaching out to experienced people to enlighten me
– Amanuel Nega
Nov 9 at 8:58
@ who ever flaged this for closing, It's not opinion based, I'm asking for suggestions on ways to do it. I'm reaching out to experienced people to enlighten me
– Amanuel Nega
Nov 9 at 8:58
@ who ever flaged this for closing, It's not opinion based, I'm asking for suggestions on ways to do it. I'm reaching out to experienced people to enlighten me
– Amanuel Nega
Nov 9 at 8:58
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53202549%2fthird-party-api-monitoring-on-aws%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
@ who ever flaged this for closing, It's not opinion based, I'm asking for suggestions on ways to do it. I'm reaching out to experienced people to enlighten me
– Amanuel Nega
Nov 9 at 8:58