As OpenSearch clusters grow beyond 10–20 nodes, centralized monitoring becomes a bottleneck: single points of failure, API overload, and higher latency. Traditional approaches don’t scale.

This session presents a production-ready distributed monitoring architecture using Apache NiFi and MiNiFi. MiNiFi agents on each node collect local metrics via a custom NodeStatsProcessor (CPU, heap, JVM, I/O, thread pools), while central NiFi collectors aggregate and deduplicate cluster-wide metrics using a custom ClusterStatsProcessor and forward them to OpenSearch.

Results include linear scalability, sub-millisecond node metrics, HA, and minimal overhead. Key insights: separating local vs cluster-wide collection, deployment patterns (bare metal, VMs, containers), HA strategies with multiple NiFi collectors, and lessons from production clusters of 10–50+ nodes processing millions of metrics daily with 99.9% reliability.

Ideal for OpenSearch operators managing 10+ nodes, platform engineers building observability pipelines, and anyone hitting centralized monitoring limits. Walk away with a distributed architecture you can implement immediately.