Wsrtjtyk

I'm using elasticsearch + kibana + logstash + filebeat latest 6.4.1 to collect and analyze web logs. The columns of my log are like:

timestamp, http_method, request_uri, http_status, host, user_agent, client_ip, client_port


I have configured ELK to show my logs in Kibana. But now I want to see my logs in sessions. I hope the log lines can be grouped by session and shown in Kibana's Discover page. In my scenario, the log lines with the same (host, client_ip) belong to the same session.

I hope to have this:

1. 
   

   Session table

   
   
   

   name, client_ip, host

   
   
   

   session1, www.google.com, 1.2.3.4

   
   
   

   session2, www.bing.com, 5.6.7.8

   
   
   

   session3, www.google.com, 4.3.2.1

   
   

When I click one of the above session (e.g., session1), I can see all the records of that session in the following 2nd table:

2. 
   

   Log table

   
   
   

   timestamp, http_method, request_uri, http_status, host, user_agent, client_ip, client_port

   
   
   

   20181105, 21:33:17.773, POST, /index.html, 200, www.google.com, chrome 59, 1.2.3.4, 1234

   
   
   

   20181105, 21:33:18.773, POST, /abc.html, 200, www.google.com, chrome 59, 1.2.3.4, 1234

   
   
   

   20181105, 21:33:19.773, POST, /index.html, 404, www.google.com, chrome 59, 1.2.3.4, 5678

   
   

I know Elasticsearch does flat indexing, it's not easy to have hierarchy between documents. I'm OK to create separated indices for the above two tables. I know Dashboard can show two Discover tables at the same time. But my question is:

How to link these two tables? When I click one item in the Session table, the Log table will show corresponding contents?

Or is there any other way to fulfill my requirement (view session-based logs easily in Kibana)? Thanks.

UPDATE

The index for Log table contains the session field, which can be session1, session2, etc. Both indices are under my control. So I can add any field if needed.

What I would do is to add a session field in the log table containing the same session name as in the session table for each log line. That's going to be your "join key".

Then, you can create one sessions index and another logs index. Both indexes must have that session field. Then go in Kibana and can create one index-pattern for each index.

Next, go to the Discover tab and create one saved search per index pattern, you now should have a saved search named "Logs" and another one named "Sessions".

Finally, head to the Dashboard tab and add the two saved searches you created previously. On the screenshot below, you can see that when hovering over the session field you can filter by that field. If you click on (e.g.) session2, then both tables will be filtered accordingly, thus achieving what you need.