elasticon 2018

What is new at Elastic{ON} 2018?

Last year Anchormen visited Elastic{ON} (you can read about it here and here) and discussed the new capabilities of the elastic products. With all that has been happening lately, we just couldn’t miss the opportunity of going this year as well.

As I was listening to the presentations I found myself thinking about everything I heard. My focus up to now was mostly on the innovation of the Elastic stack itself (rather than on client cases) and I wanted to share the bits and pieces I found most interesting.

Elasticsearch: faster and better

Elastic is on a never ending journey to make Elasticsearch better. With every release there are innovations that bring it to the next level; sorting of indices, faster shard recovery and adaptive replica selection to avoid hitting nodes under stress are just some of the examples. Some innovations that stood out for me this year are data roll-up, cross cluster replication and SQL support.

Most people know Elasticsearch can aggregate data extremely fast and this fuels thousands of Kibana dashboards all around the world. As long as the data remains in Elasticsearch it is possible to look back in time. But keeping this ‘cold’ data around for occasional use is not very cost effective. This is where ‘data rollup’ comes into play. This upcoming feature, part of X-Pack Basic, will enable users to schedule predefined aggregations which are put in a new index. This is very flexible as multiple aggregation fields and metrics can be configured. Logically, this new index is a lot smaller and aggregations on it will run even faster than on the original index. The original index can be closed and/or moved to free up resources. Updates on the original data will not be put into the rolled up index if this was already done before. However, it is possible to set a delay on the roll up process to make sure the data will not be changed anymore. It should be released with 6.3 as a beta feature.

I am very happy with the announcement of the cross cluster replication feature as it is a bit overdue in my opinion. Quite often there is a requirement to be robust against failure of an entire data center. This means setting up ES clusters in multiple data centers and keeping them in sync which is very tedious work. Cross-cluster replication will make this a lot easier so it’s good it has finally arrived.

Last but not least, SQL support will be part of the 6.3 release. This was already announced at Elastic{ON} 2017 but turned out to be a tough challenge. We already knew this as we once build a JDBC driver for Elasticsearch in 2016 (also see this post). It’s good that Elastic now supports SQL as it will make it easier to work with Elasticsearch and enable other (BI) front-ends to leverage Elasticsearch’s speed and functions. The first release will be a bit limited as it will only support aggregations on one field and will not yet flatten nested tables. But this will be ‘fixed’ in future releases.

Kibana: making things easier

Elastic is also putting a lot of effort to make it easier to work with their stack. From the start Elastic has been using an ‘API first strategy’ (which is very good in my opinion). However, this also means Elasticsearch users had to write JSON documents and work with the REST API for almost everything they wanted to do. This year Kibana will catch up by providing graphical interfaces for common tasks like index management, rollups and lifecycle management. This will make it a lot easier to work with Elasticsearch.

Kibana will also get a complete overhaul in order to provide better and more consistent user experience and more visualizations. In addition, it will come with more pre-configured dashboards used for monitoring.

Monitoring: cover it all

A lot of the talks at this year’s Elastic{ON}, both from Elastic but also from clients, were on the topic of monitoring. Elastic has taken a large step by adding Application Performance Monitoring (APM) functionality to its stack. Applications can be instrumented with APM through a few lines of code after which Elasticsearch will receive information on response times, errors, stack traces and much more. This means that Kibana will be able to monitor everything: infrastructure (through Beats), logs and applications. This will enable engineers to solve issues and track down bottlenecks a lot easier than they can do now. Check this page from their own website for more information.

Machine learning: more than just time series

Lastly, there are innovations done on the machine learning component of the Elastic stack as well. A rather logical extension of the time-series anomaly detection is forecasting and now it’s finally possible. Elastic machine learning jobs can predict “near future” values. This can for example be used to predict stock and infrastructure capacity utilization (the “when do I have to scale up or down?” type of questions).

The Elastic stack will also get machine learning capabilities for non-time-series data. An example shown on the conference was outlier detection on webshop behavior. For example SKU’s that are bought surprisingly often or customers that make a lot of purchases.

The last thing I want to mention is the new Log Categorization function which will be able to group together raw log messages based on commonalities in the text. It will also show the behavior of each category of log events over time. This will provide insight if a certain type of event is growing over time or not. The best part is that it will be possible to generate a GROK pattern for each category… So, say goodbye to hours or days spent to build, test and optimize GROK patterns as Elastic will now be able to do it by itself!

These were the biggest changes that caught my attention this year. I hope you are excited about them as I was! If you want to learn more about what you can do with Elastic or what data can do for your business, make sure to contact us.

Like this article and want to stay updated of more news and events?
Then sign up for our newsletter!

Don't miss out!

Subscribe to our newsletter and stay up to date with our latest articles and events!

Subscribe now

Newsletter Subscription