Monitoring with ELK Stack
The traditional strategy of infrastructure management based on monitoring is not suitable for large scale infrastructure and/or when dynamic infrastructure creation and destruction strategies are used (much in vogue in CI/CD clouds pipelines). In addition to the fact that the correct functioning of a given service depends on many interdependent components, it is often virtually impossible to diagnose an incident because the problematic instance has already been destroyed and eventually replaced by another where the causes of the incident cannot be diagnosed.
In the meantime, control theory strategies have been applied to technology, namely observability, in which the adequacy of a system’s internal states is determined based on its output results; as such, we change the behavior from reactive to alarms to proactive based on metrics observation, often obtained based on the crossing of several data sources that are typically system, application and service records (logs).
It is possible to implement metric-based alerts, just as in traditional monitoring it is possible to trigger alerts based on state changes (available to unavailable, etc). Eurotux teams have used Stack ELK with excellent results for analysis and correlation of records from any source, in any format, and which allows to search, analyze and view data in real time, being an excellent alternative to some commercial solutions.
What is the ELK Stack?
“ELK” is the acronym for three open source projects: Elasticsearch, Logstash and Kibana. Elasticsearch is a search and analysis engine. Logstash is a server-side data processing pipeline that takes data from numerous sources simultaneously, transforms it, and sends it to a “hideout” like Elasticsearch. Kibana allows users to view data with diagrams and graphs on Elasticsearch.
Another more recent alternative is Open Distro, also Open-Source, and sponsored by AWS. Both Stack ELK and OpenDistro are highly customizable and have a number of advanced capabilities such as alarming.
