Data, governance, and risk: what AI strategy must cover

The uncomfortable questions

Every AI strategy has to answer these in writing, not in a meeting nobody minutes:

• Whose data are we using, and are we actually allowed to use it for this purpose?
• Which decisions can AI make on its own, and which need a human to sign off before anything happens?
• How do we notice bias, drift, or plain wrongness once a model is live, rather than months later?
• What happens if a model, a vendor, or a regulator changes overnight?

Skip these and legal, compliance, or the board will insert themselves later, usually at the worst possible moment, usually after something has already gone wrong.

A worked example

Take a mid-sized recruitment firm building an AI tool to screen CVs against job specs. On paper this is a good use case: high volume, repetitive, clear rules. In practice it needs answers before a line of code is written.

Whose data are we using: candidate CVs, often containing special category data (health conditions mentioned in gaps, religious affiliations via school names, nationality). Under UK GDPR that changes the lawful basis you need and the consent language on the job application form.

Which decisions can AI make: if the model auto-rejects candidates below a threshold with no human review, that's an automated decision with legal effect under Article 22 UK GDPR, and it likely falls into a higher risk category under the EU AI Act if the firm operates in the EU (AI systems used in recruitment and worker management sit in the Act's high-risk list). A model that ranks and a recruiter that decides is a different, much safer, design.

How do we detect bias: if the training data reflects ten years of a company's past hiring, it will reproduce that pattern, including whatever imbalance already existed. You need to test the model's outputs against protected characteristics before launch and on a schedule after, not just once.

What if the vendor changes: many firms buy this capability from a vendor's API rather than building it. If the vendor silently upgrades the underlying model, your bias testing is invalid the next morning and nobody told you.

None of this is exotic. It's the standard shape of the problem, and it applies whether you're screening CVs, flagging fraud, or drafting customer replies.

Risk-tiering, not blanket rules

Not every model needs the same scrutiny. A chatbot that drafts a first-pass email reply, reviewed by a human before sending, is a different risk to a model that decides credit limits unattended. Put every model into a tier, roughly:

1. Low — AI assists, a human reviews everything before it has any effect (drafting, summarising, search).
2. Medium — AI acts with spot-checking or exception-based review (routing tickets, flagging anomalies).
3. High — AI decisions have legal, financial, or safety effect on individuals, with no human in the loop, or a human who rubber-stamps rather than reviews.

Tier drives the level of governance: a low-tier tool needs a line in the model inventory; a high-tier tool needs documented testing, a named accountable owner, and a rollback plan. Most organisations over-govern the low tier and under-govern the high tier, because the high-tier tool is the one someone rushed into production to hit a deadline.

A minimum viable governance model

• Data policy. What data can go into an AI system, under what lawful basis, and where it's allowed to travel (does it leave the country, does it leave the vendor's infrastructure).
• Model inventory. Every AI model in production: owner, purpose, risk tier, and what it's allowed to decide.
• Human-in-the-loop policy. A clear, tier-specific line between what AI can do unsupervised and what needs sign-off.
• Incident process. Who gets called, what gets switched off, and how you tell affected people, when a model behaves badly.
• Third-party review. How you check vendor AI, including LLM providers, before you plug their API into something that touches customers.

You don't need a 200-page binder. You need five topics clear enough that a new joiner reads them in twenty minutes and knows what they can and can't do.

Common pitfalls

• Treating the risk tier as fixed at launch. Models get repurposed ("we'll just use the summariser for triage too") without anyone revisiting the tier.
• No owner named for a model, so when it drifts, everyone assumes someone else is watching it.
• Vendor contracts with no clause covering model changes, meaning you find out about an upgrade from a support ticket, not a notification.

Regulatory context

In the UK and EU, expect AI-specific regulation to keep tightening: the EU AI Act's phased obligations, sector regulators applying existing rules to AI decisions, and ICO guidance sharpening on automated decision-making. None of this should cause alarm. Build governance that's re-tunable rather than perfect on day one, and revisit the model inventory and tiers at least twice a year.

Where Florence helps

Florence's AI Readiness Assessment includes an optional ISO/IEC 42001 checklist covering exactly this ground: data handling, human oversight, and incident response. Gaps you flag there carry straight through into Strategy and Projects, so a missing governance control becomes a project brief with an owner, not a line in a policy document nobody reads again.