The modern observability stack has transformed the way you troubleshoot issues in a microservice environment. Some situations however, ask for investigation of a single application pod that seems to be misbehaving. Ephemeral containers provide a way to attach to a seemingly problematic pod without restarting it.This allows developers to observe an issue in a live or staging environment running on top of Kubernetes.

We will discuss the practicality of launching a Go debugger (Delve) within an ephemeral container to remotely debug an application both on the CLI and in VS Code, highlighting the requirements and possible limitations one might encounter when trying to set up a similar troubleshooting routine. As part of that, we will explore the API for ephemeral containers and the current implementation in kubectl.

While the talk will use Go and Delve as an example, the considerations and steps presented are of universal importance to running a debugger for your language stack of choice.