"Are we more productive with AI?" is being asked in almost every product and technology organisation right now. It is also, as usually stated, close to unanswerable — and the ways people make it answerable tend to quietly guarantee a positive result.
My background is in performance metrics and lean-agile measurement: designing how technology organisations measure themselves, teaching performance metrics and OKRs at postgraduate level for the last five years, and working through several generations of delivery measurement across different industries. This session is the method I'd give someone who has just been handed this question and a deadline.
What I'll cover:
1. Why does the question defeat people? "Productivity" collapses three different things — output, throughput and value delivered — that move independently and sometimes in opposite directions. I'll show how conflating them produces confident conclusions that don't survive contact with reality.
2. Decomposition. A practical way to break the question into observable signals: adoption, flow, quality, and the experience of the people doing the work. What each one can and can't tell you, and why the fourth is the one most organisations skip.
3. The four traps. Attribution (something else changed too), novelty (the first eight weeks lie), optimisation (people meet the metric you chose), and the output–value gap (shipping more of the wrong thing faster). For each: how it shows up, and one check that catches it. [FILL: for at least two of these, add a short anonymised story from your own career — "in one organisation…" — where you or a team you worked with fell into it]
4. What the published research actually says. A practitioner's read on the current studies of AI coding assistants and developer throughput — where the findings are consistent, where they conflict, and where the study design doesn't transfer to how most of us work. [FILL: name the two or three studies you've actually read so you can credit them properly — the guidelines require crediting sources, and it strengthens the talk]
5. Designing a defensible approach. How to assemble a small measurement set you could put in front of both a sceptical engineer and an executive who wants a number this quarter, and survive both conversations.
6. The part that stays human. Choosing what deserves to be measured is not an analytical act. It's a negotiation with the people whose work is being described, and getting it wrong damages trust in ways the dashboard never shows. I'll argue this is the coach's real job in this wave, and that it is getting harder, not easier.
A note on evidence: I'm deliberately not presenting a single-organisation case study. The numbers from any one company don't transfer, and I'd rather give you a method you can run than a result you can only admire. Every example I use is either published research, properly credited, or an anonymised pattern from across the organisations I've worked in.
Interactivity: I'll open by asking the room to write down, in one sentence, what their organisation currently claims about AI and productivity — and how they'd know if it were wrong. We come back to it at the end. Ten minutes for questions and challenge. Nobody has to speak aloud.
Key points — you will leave this session with…
A decomposition of "are we more productive?" into four observable signal groups you can actually collect
Four measurement traps that produce flattering results, with one practical check for each
A practitioner's read on what the published research supports and where it doesn't transfer to your context
A small, defensible measurement set you can take to both engineers and executives
A way to involve the people being measured in choosing the measures — and the case for why skipping that step costs you more than it saves