Session

LLM Inference Benchmarks: Demystified and Explained

## Abstract

TTFT, TPOT, ITL, end-to-end latency, tokens per second: inference benchmarks are full of acronyms.

In this demo-driven talk, we’ll follow one streamed request from dispatch to final token. Along the way, we’ll show what each common metric measures and how it maps to user experience. Then we’ll tour popular benchmarking tools and run one live. You’ll leave knowing how to design a fair benchmark, interpret the results, and choose the metrics that match your application.

## Description

A benchmark number means little unless you know what was measured, on which workload, and under how much load. Two endpoints can both claim to be fast and still behave very differently for real users.

We start by tracing a single streamed request from the client to the GPU and back to see where the time goes. Along the way, we explain time to first token (TTFT), time per output token (TPOT), inter-token latency (ITL), end-to-end latency, and throughput.

Through side-by-side examples, we show how these metrics trade off. One endpoint might start fast but stream slowly. Another might be quick for a single user but degrade under concurrency. A third might post high aggregate throughput while each individual user waits. We also show why averages hide tail latency, and why p95 and p99 often matter more than the mean. Finally, we look at how prompt length, output length, concurrency, cache state, generation settings, errors, and the network path can each shift the numbers.

With those foundations in place, we compare three tools built for different needs:
- **vLLM Benchmark CLI** measures serving performance with configurable workloads and concurrency.
- **NVIDIA GenAI-Perf** generates load and reports latency and throughput for generative AI endpoints.
- **ZebraBench** runs in the browser and compares OpenAI-compatible endpoints with no local Python setup.

We’ll then do a couple of benchmarks, compare the results across repeated runs

## Key Takeaways

- Understand TTFT, TPOT, ITL, end-to-end latency, and throughput, and how each maps to user experience
- Distinguish single-request speed from system capacity, and use percentiles to expose tail latency
- Design fair benchmarks by controlling workload, settings, cache state, and concurrency
- Choose the right tool, from quick browser-based comparisons to controlled load tes


## Talk Level

Beginner to intermediate

## Audience

AI engineers and platform engineers who select, operate, or compare LLMs and inference endpoints

No prior experience with inference benchmarking is required.

## Duration

45 minutes

Sujee Maniyam

Developer Advocate @ Nebius

San Jose, California, United States

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top