Monitoring with Elastic Stack
Monitoring & managing complete pipelines is a difficult and complicated task. Especially in the world of big and distributed IT ecosystem with microservices that are horizontally scalable and can manage big amounts of data together with cloud solutions and different environments.
Consider the following situation. Let’s take a simple pipeline that involves 2 microservices. These microservices communicate with each other through message queues. Microservice A is the source of truth for the data and microservice B is getting changes on specific data from micorservice A. Now imagine that a record in microservice B starts diverging from the source of truth or certain changes happen much slower than expected.
Detecting and debugging these types of problem might require both investigating the internals of each microservice and communication of each miroservice with its internal datastore and the other microservice. The more components are involved, the more complex a problem will get. To make monitoring easier, all different logs and metrics from different services need to be aggregated and managed in a central view. Alerts can be set in place to report failures and other critical scenarios. In addition, reoccurring patterns in logs and metrics can be used for anomaly detection and load prediction.
One way to perform this comprehensive monitoring is with Elastic stack; a powerful set of tools to collect, manage and make sense of structured and unstructured data. At the core of this stack is Elastic search which is an open source, scalable, distributed search engine based on Lucene. Other common tools are logstash and Beats for data processing and ingestion and Kibana which is an easy to use user interface with a lot of features that bring all of the different Elastic tools together. These days Elastic.co has developed many tools that when used together can provide a full stack, complete and comprehensive monitoring solution.
In many companies, Elastic search has been already used for log and metrics management but most log and metric management systems do not keep cold historical data. This data can be used to detect certain patterns (holiday peaks for example). Elastic search offers you the freedom to manage historical data using rollovers and ways to reduce the size of cold data. Logstash or Beats can be used for gathering, transforming and transmitting logs and metrics from different servers to Elastic search. Kibana can be used for viewing, searching and dashboarding the aggregated logs and metrics together. In addition to existing logs and metrics, Elastic APM library can provide application level performance monitoring.
If you look at the picture below, you can see all of the elastic stack monitoring capabilities.
Next to these open source features, Elastic.co is providing some tools that require licensing. They are mostly inside the x-pack package – a commercial plugin for elastic which includes security, alerting, monitoring, graph exploration, reporting and machine learning. Here is the list of features:
- x-pack security:
- Authentication (Native, LDAP, Active Directory & PKI)
- roles & users for cluster, every index, every query & every field
- encryption (SSL support)
- x-pack reporting
- scheduled reports
- rule based report
- x-pack cluster monitoring for elastic search
- x-pack machine learning:
- anomaly detection and alerting
- forecasting the patterns for the future
- field categorisation of unstructured data
- x-pack graphs
X-pack machine learning and reporting are very useful for monitoring applications. Machine learning features can help detecting anomalies or failures based on historical data.
And this wraps up our review of monitoring with Elastic stack. We’ve planned to make it a two-parter blog series. In the next article we will explore the different ways to gather data in elastic search and use it for monitoring.