Observability Glossary

ELK Stack

The ELK Stack is a combination of tools for managing and visualizing log data. Let's break it down:

Elasticsearch: This is where the log data is stored and indexed, making it easy to search and analyze. It's like a supercharged database specifically designed for log data.
Logstash: Logstash is the tool used to collect, parse, and transform the log data before sending it to Elasticsearch. It's like a filter that cleans up the data before it's stored.
Kibana: Kibana is the visualization tool that enables developers to create dashboards and visualizations from the log data stored in Elasticsearch. It's like the window into your log data, making it easy to see trends and troubleshoot issues.

Let's say you're running a web application and you want to track errors and user activity. You can use the ELK Stack to collect all the log data from your application servers, parse it to extract useful information, store it in Elasticsearch, and then create visualizations in Kibana to monitor the health of your application.

The ELK stack is however not always suitable for applications with a logs with both high cardinality and high dimentionality as a result of the indexing mechanisms in Elasticsearch and its tendency to demand a lot of resources as the datasets grow.

Explore related concepts

Logging

Logging is the process of recording events, actions, or messages that occur within an application, typically for troubleshooting or analysis purposes.

Canonical logs

Canonical logs are logs you add at the end of every request or transaction. Each of these logs contains all the important parameters about the request, for instance, user ID, duration, response status code, etc. Adding all these details in a single log line enable you to debug faster and perform more complex aggregations without the need to join multiple log lines together.

Observability

Observability is the ability to understand how your application is working and behaving in production through telemetry data (logs, metrics, traces, wide events, etc.). It enables you detect, diagnose and resolve issues in your app before they impact your users and become problems.