Kafka cluster and clients health check : offsets are "almost" enough

The aim of this presentation is to demonstrate some of the tips and tools for a better use of offsets metrics in the monitoring of the health of clusters and applications.

As is generally known, a "Normal" increase of end offsets means that the producers work properly. In the same way, a "Normal" increase of consumer groups offsets means that consumers work properly.
Most of the time, in the presence of an issue on the cluster, this often affects to some extent producers and consumers. This impact could be detected through offsets.

Our suggestion of health check monitoring is based on:

· Metrics: end offsets, current offsets, lags, lags in time, producing rate, and consuming rate
· Thresholds

As tools for the implementation of this monitoring, we will make use of:
· Offsets and thresholds exporter.
· Usual monitoring and alerting stacks : Elastic stack (Beats, Elasticsearch, Kibana, Machine Learning), Splunk, Prometheus Grafana

The session contains a demo for different kinds of use cases: high throughput, batched data, irregular traffic. We will also try to show how an optimised use of offsets could offer valuable information about the health of clusters and applications.