Distributed tracing is crucial for troubleshooting in today's microservice architecture. At FUTU, over 100 million Trace Spans are generated every second, requiring 200TB of disk space per day to store them. Collecting the entire volume of Trace data can be expensive, and many Traces that appear identical may not be helpful.

OpenTelemetry offers various sampling policies. These policies help to reduce resource usage, but different policies require different amounts of network I/O, memory, and storage space to achieve varying sampling quality. Finding the best combo of different policies for a specific scenario is essential.

We will share FUTU's experience in evaluating different combo of sampling policies by analyzing their cost and sampling quality quantitatively. We'll also introduce a new sampling policy that aims to cover more edge cases. You will learn about the specific costs and benefits of different sampling policies and discover how to customize them for your own business.