Ignore how much people use AI and look at what they do with its output. Strong AI users steer: they review generated code, exercise it before shipping and catch their own defects early. Weak usage has a signature too: model output committed unread, tell-tale debris in the diff, and impossible values shipping to production unnoticed.
The metric that stopped working
When one engineer in ten used an agent, usage itself was informative. When nine in ten do, it tells you nothing. On an AI-heavy team, the interesting variance moved one level down, into how people treat what the model gives them.
The signature of steering
Engineers who use AI well leave evidence. Their sessions show output being read and challenged: tests written against generated code, values checked against reality, follow-up prompts that push back rather than accept. Their defects tend to be caught by their own verification before production, which is the single strongest forward-looking reliability signal a team member can show. Their commits read like someone was present.
The signature of shipping unread
The opposite pattern is just as recognisable, and it is not about skill with prompts:
- Debris in the diff. Stray statement fragments, leftover scaffolding marked for removal, fragments of agent dialogue in comments. Each one is direct evidence a change went in without a human reading it.
- Unvalidated output as design. The archetypal failure is a feature that lets the model generate values with no human-authored check anywhere, discovered only when production shows data that cannot be true.
- Rework loops. The same subsystem patched again and again without a durable fix, because each patch was generated, shipped and never really understood.
Measuring it fairly
Two rules keep this honest. First, discount what the tooling does automatically: modern skill frameworks generate plans, tests and self-review commits by default, so those artefacts are evidence of tooling, not of judgement. Second, present findings as evidence rather than verdicts: cite the sessions and commits, run the sceptical pass before showing a negative conclusion, and leave the judgement to a human who knows the context. The point is not to catch people. It is to know where coaching would help, and who already deserves more trust than the commit count ever showed.
Frequently asked questions
Is high AI usage a red flag?
No. Usage volume is neutral. The differentiator is validation behaviour: whether output gets reviewed, exercised and checked before it ships.
Can validation behaviour really be seen in data?
Yes, when session records and git history are read together. Verification actions, self-caught defects and committed-unread debris are all observable patterns, each citable to specific sessions and commits.
Should this feed performance reviews?
As evidence with citations, alongside human judgement, and never as an automatic score. Negative findings in particular should survive a sceptical second pass before anyone acts on them.