AI wrote the change faster. The review took longer. Production became less stable. Was the team actually more productive?

Research gives conflicting answers because it measures different things. Controlled studies have found faster task completion and higher output in some settings. METR found experienced open-source maintainers were slower with early-2025 AI tools. DORA reports higher AI adoption alongside both greater throughput and greater instability.

These results are not interchangeable. A faster coding task does not automatically improve team flow, software delivery, or customer outcomes.

This session turns that measurement problem into a practical scorecard. We will connect four layers: developer effort and comprehension; team flow, review latency, and rework; delivery throughput and stability; and product outcomes. We will examine how to establish a baseline, segment results by task type, and detect when AI simply moves work downstream.

We will also challenge seductive metrics such as lines of code, pull-request count, and suggestion acceptance. They can prove that people are using AI without proving that the investment created value.

Attendees will leave able to evaluate AI productivity claims, choose a balanced set of measures, and run a useful experiment with their own team.

Measure the system, not the typing.

Audience: Engineering leaders, developer-productivity teams, and senior engineers measuring AI adoption.
Format: 20–45-minute evidence-driven leadership talk.
Example: Build a measurement scorecard that pairs delivery outcomes with review, rework, durability, and comprehension signals.
Research: METR randomized trial: https://arxiv.org/abs/2507.09089
Vendor scope: Vendor-neutral.