Elasticsearch and the Elastic Stack (formerly known as ELK Stack) have been around for some years now, for almost half an eternity in the life of someone in IT. The Elastic Stack is offered by the American company Elastic and is based on Apache Lucene and Java. I’ve been monitoring Elasticsearch for some time, but so far I’ve only had some rough experience with Elasticsearch: using it as a search appliance for near-real-time searches on the web (results almost returned instantly). I have not dealt intensively with Elasticsearch until now. But that is changing.

Currently these free products belong to the Elastic Stack:

• Beats (transfer data from various sources)
• Logstash (pipeline data, enriches data, filters out data)
• Elasticsearch (saves data and returns it – the main product)
• Kibana (visualizes data and helps to handle Elasticsearch)

The Elastic Stack is Linux-based and you have to take care of the installation and the operation yourself: for example, directly in a data center like Interxion or a hoster like AWS. You can perform either as a direct installation or you shovel the products into a Docker container (which is the preferred way). But you can also easily connect Windows systems. The free Barebone products from the Elastic Stack are then available for a fee. After all, hundreds of employees of Elastic also want to be paid somehow:

• X-Pack as an Elastic Stack extension pack (for on-demand monitoring or paid-for extensions to secure the elastic stack and for data analysis)
• Elastic Cloud as a version directly hosted by Elastic (version upgrades go unnoticed by the end user and you get full support)
• Elastic Enterprise Cloud (like Elastic Cloud, only one can manage it in its own data center)

Elasticsearch is the main reason for the great success of the software, because everything started here. With Elasticsearch you can process, search and display mass information. As if you were organizing data like in a database: you dump data into Elasticsearch in a structured way and then retrieve it – but very fast compared to a database. And it comes with built-in clustering across different services.

Elasticsearch as Content Search Engine
Unlike a database, Elasticsearch knows neither primary keys nor relations. Therefore, Elasticsearch should not be your primary location of the data but it will remain, for example, in a database. But because databases are slow, Elasticsearch is often used as a search engine: the database contents and documents that should be searchable are indexed in Elasticsearch and are searchable via a web frontend, for example. The special thing about this: you can search for data and texts quickly and your results will be ranked according to importance. Almost like Google.

But Elasticsearch can do more than just search and rank texts.

Elasticsearch for Log Aggregation
Most users use Elastissearch today because of the added products from the Elastic Stack. This is because data from a wide range of sources converges centrally and across platforms. The Elasticsearch software eats what it gets to eat: login attempts to an operating system, CPU load of computers, the number of accesses to web pages, or the integrity of files on the hard disk. Elasticsearch is therefore ideal for large amounts of data: headword big data. If you collect all the data sources centrally and adjust to the correct queries, then you will recognize completely new relationships or problems are already visible in the data before they become recognizable to the end user.

Everything else about Elastic Stack is available on Elastic’s website.

A little tip for all those who are interested in Elasticsearch: A first step is to watch complementary introduction videos on Youtube. But who wants to go deeper, requires paid training and certifications. I’ve learned my basics in Elasticsearch at the Linux Academy.

At the Linux Academy, you do not just watch educational videos, but there are hands-on labs with test servers and learning quizzes. Only when you answer all questions in the training, you also reach the training goal.