How do you measure AI strategy success?

Two families of metrics

Measuring an AI strategy is like measuring any transformation programme: you need leading indicators (are we doing the work?) and lagging indicators (is the work paying off?). Most organisations default to one or the other. Both alone are a trap.

Leading indicators

• Pilots launched vs planned. Are you moving through the pipeline you committed to?
• Data pipelines built and monitored. No pipeline, no model in production.
• Users trained and actively using. Trained is not the same as using.
• Governance artefacts in place — model inventory, policies, review cadence.
• Time from idea to first production model. This should shrink as the team gets better.

Lagging indicators

• Cost savings realised.
• Revenue impact.
• Cycle time reduction.
• Error rate reduction.
• Customer or employee satisfaction change.

A worked example: flat lagging, informative leading

Take a customer service team piloting an AI triage tool. Six months in, the lagging metric — average resolution time — hasn't moved. On its own, that reads as failure. Kill it, right?

Look at the leading indicators first. Pilot rollout hit 100% of planned teams. Agent adoption is at four in five shifts, up from one in five at launch. Time from flagged issue to fix is down from three weeks to four days, because the team built a fast feedback loop with the vendor. What's actually happening: the tool is being used properly, and the team is iterating quickly, but resolution time is flat because ticket volume rose 15% over the same period — a seasonal spike nobody baselined against. Adjusted for volume, resolution time per ticket is down. Without the leading indicators, this looks like a failed pilot. With them, it's a working pilot mis-measured against the wrong lagging comparator. The fix wasn't to cancel the project — it was to correct the lagging metric to control for volume, and set a new baseline that accounted for it.

This is the point of running both families together: leading indicators tell you whether the flat lagging number is a real problem or a measurement problem.

The vanity-metric trap

Not every number that moves is worth watching. "Number of AI conversations had" tells you nothing about value delivered. "Models deployed" tells you nothing about whether anyone uses them. "AI awareness training completion" tells you people sat through a slide deck, not that behaviour changed. These metrics feel good to report because they always go up and never require a hard conversation. That's exactly why they're dangerous — they crowd out room on the dashboard for metrics that might say something uncomfortable. A useful test: if the metric can only ever improve regardless of whether the initiative is working, it's vanity. Cut it.

Rules for a healthy measurement layer

• Every initiative has a named metric and a named owner. No metric, no project. No owner, no accountability when the number stalls.
• Set a baseline before you start. "Improve resolution time by 20%" is meaningless without knowing the starting number, the period it covers, and what else was changing at the same time (seasonality, headcount, product changes).
• Report on both families, side by side. Leading without lagging looks like activity theatre — busy teams, no proof of value. Lagging without leading gives you no early warning and no diagnostic when a number goes flat.
• Revisit quarterly. Some metrics stop being informative once a process matures — "time to first production model" matters a lot in month three and much less in year two. Retire metrics that no longer discriminate between good and bad performance, and replace them.
• Watch for confounders. Volume changes, seasonality, org restructures and tooling changes can all move a lagging metric independent of the AI work. Note them when you set the baseline, not after the number surprises you.

What the board actually wants

Board reporting can be simple: three lagging metrics, three leading metrics, and a paragraph on what changed and why — including any confounders that explain a flat or noisy number. If your dashboard has 40 tiles, nobody reads it, and the one number that matters gets lost in the noise. Pick fewer metrics and defend them properly rather than reporting everything and defending nothing.

Next steps

1. List every live AI initiative and check each has exactly one named metric and one named owner.
2. For each metric, write down the baseline value, the date it was captured, and anything else that was changing at the time.
3. Pair every lagging metric with at least one leading metric that would explain a flat or noisy result.
4. Strip out anything that only ever goes up — if a metric can't fail, it isn't measuring anything.
5. Put a quarterly date in the diary to review and retire metrics that have stopped being useful.

Where Florence helps

Florence's Insights and Projects steps ask you to name a measure for every project before it's approved — not after, when it's convenient to pick one that already looks good. Each project brief carries its metric and owner forward into the Strategy view, so when a lagging number looks flat, you can see the leading indicators for that same project sitting right next to it, rather than hunting across separate spreadsheets to work out whether it's a real problem or a measurement one.