Session
Your Benchmark Is Lying to You
I made a function forty percent faster. I had the numbers. I put it in production and nothing changed at all.
The function was not the problem. The benchmark was, and it had been lying to me in about four different ways at once.
This talk is about measuring things correctly, which is a separate skill from making them fast and one that almost nobody teaches. We will go through the ways a microbenchmark deceives you. Dead code elimination, where the compiler notices you never use the result and helpfully removes the thing you were timing. Warmup, and why your first thousand iterations are measuring the JIT rather than your code. Tiered compilation quietly changing what runs halfway through your run. CPU frequency scaling on a laptop, which means your afternoon numbers and your morning numbers disagree for reasons that have nothing to do with software.
Then the bigger error, which is measuring the wrong statistic. An average hides the tail, and the tail is what your users experience. I will show you a change that improved the mean and made the 99th percentile worse, and how long it took us to notice.
We will use BenchmarkDotNet on screen, but the failure modes are universal and so is the fix.
Chris Houdeshell
VP of Eng. and Ops | Bit Herder | ☕
State College, Pennsylvania, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top