At Serviceware we're moving more and more applications into our Nomad powered cloud.
A diverse tech stack where each log format is different, only very temporary metrics from the Nomad UI and no traces at all while more and more services are talking to each other made it hard for our developers to understand what's going on in production.

Luckily as OpenTelemetry is becoming the de-facto standard for observability and Grafana brings the necessary counter parts we set out to build our new telemetry infrastructure.

In this talk we look at the basic building blocks of OpenTelemetry and what to consider when setting up your OpenTelemtry Collector pipeline.
We go through a demo derived from our production setup on Nomad, where we use Grafana's Alloy as OpenTelemetry Collector as it blends perfectly into the HashiCorp ecosystem. With the infrastructure in place we learn the required parts for your Nomad Job file to instrument a service.
Lastly we introduce the Nomad admission controller (NACP) to take the complexity out of instrumented deployments by auto-injecting those required parts.
Equipped with this knowledge you can start building your own observability stack on Nomad and get the insights you need to keep your applications running smoothly.