At Delivery Hero we have a [Reliability Manifesto](https://tech.deliveryhero.com/our-reliability-manifesto/) and in section *R-9* It states that all services should be load tested, be able to process at least 4x load rump-up and should be run on regular bases.

During Load Testing we generate a huge amount of monitoring data in different systems: DataDog APM, Prometheus and AWS Cloudwatch.

All this data should be aggregated in one presentable report.

The problem we want to solve:
- How to abstract reports from different metrics sources and different data they provide.
- How to give a flexible and simple instrument to generate templates automatically.
- How to make the lives of our engineers easier.

For instance, in my tribe we have dozens of services, in Logistics we have hundreds of services, In Delivery Hero overall we have thousands of services.

For each of them we need to run load tests on regular bases.

And for each of them we need to generate reports.

The problem of running tests automatically is easy to solve. But how to solve the problem with reporting?
We have an answer, because we designed a tool which helps us to abstract reports from metrics data sources and at the same time keep flexibility of calling different sources with different queries.

You provide a template in a format of Golang templates

{ { ddResult := queryDD .StartTime .EndTime “query” } }

The maximum value of the metric is { { findMax ddResult } }

Or

{ { promResult := queryProm .StartTime .EndTime “query” } }

{ { drawProm promResult } }

And as the output you receive a generated PDF with either text “The maximum value of the metric is XX” or drawn graph from Prometheus.

All these `queryDD`, `findMax`, `queryProm` and `drawProm` are predefined `template.Funcs` and empower great flexibility of your report.

I will be talking about how to implement these functions, how to call different sources, process data and render outputs.