On March 8, 2023 Datadog experienced a massive global outage that took almost 24 hours to mitigate and a further ~24 hours to backfill data after restoring full app availability. We’ll share the trigger for the incident and why it was such a massive effort to recover from. We’ll review the technical details of the incident: why and how we lost more than 60% of our Kubernetes nodes in less than an hour, and the challenges we faced to recover the tens of thousands of impacted nodes across hundreds of clusters. This was a very tough day for us, and we will share those hard-won technical and community lessons.