An AI governance framework that survives its first agent going to production
· 15 min read · Blog
An AI governance framework is only tested when the first agent reaches production. What holds, what breaks, and what a policy document cannot cover.
Most organisations building an AI governance framework start with the NIST AI Risk Management Framework or the ISO/IEC 42001 standard, and both are a good place to begin.
For AI that produces an output a person then acts on, like a model scoring loan applications, a classifier routing tickets or a recommendation engine, they cover most of what you need: policies, risk assessments, documentation and human oversight.
Agents change that. They call tools, connect to external systems and take actions, some of which can't be undone, often without a person reviewing each step. Both frameworks work at the level of the organisation's processes. They tell you to assess and oversee risk. They don't tell you how to control what an agent is allowed to do while it's running.
Regulators and standards bodies are starting to treat this as unfinished work. US bank regulators left generative and agentic AI out of the model risk guidance they issued in April 2026. NIST launched an AI Agent Standards Initiative in February 2026. And ISO's new guidance on human oversight of AI systems is at the final draft stage.
That gap, between governance on paper and control at runtime, is what the rest of this article covers.
What is an AI governance framework?
An AI governance framework is essentially a highly organized structure of policies, procedures, roles, and controls governing exactly how an organization designs, develops, procures, configures, and manages its AI systems.
That sentence has four key components, and many organizations miss the fourth one, controls, entirely. Policies will indicate what should take place. Procedures outline the specific sequence of events. Roles designate which person will undertake a particular responsibility.
Controls form the actual measures that prevent or demand an action: and they operate, are monitored by someone, create an audit trail and also have a well-documented failure mode recorded.
This distinction is crucial because if your framework contains just the initial three components you end up with mere documentation. This does not constitute actual governance. The difference really becomes apparent the very first time an AI system carries out an action it wasn't supposed to do so.
What should AI governance include?
AI governance should cover nine elements.
Inventory: A list of every AI system in use, starting with what each one can do. For each system, ask:
- What can it change?
- How much harm could that cause?
A system that can only read data sits in a different risk class from one that can write to your systems of record, like your CRM, ERP or finance ledger. Read-only isn't risk-free, though: a system that reads customer records can still expose them. Record what a system can do before you assign its risk tier, because the tier depends on it.
Risk classification: The tier each system is given, based on what it can do and the harm it could cause.
An accountable owner: A named person, not a team, who can show an auditor the controls that stop a non-compliant system from running.
Acceptable-use policy: A document that sets out which AI use cases are allowed, which need approval and which are banned. It should cover approved systems and shadow AI, meaning tools employees use that IT hasn't approved or doesn't know about.
Pre-deployment evaluation: The checks a system has to pass before it goes live, with stricter checks for higher risk tiers.
Documentation: Model cards, system descriptions and data lineage. This isn't paperwork for its own sake. If you go for ISO/IEC 42001 certification, the auditor will check it, and the NIST AI RMF calls for it.
Production monitoring: Ongoing checks on how a live system behaves, including drift detection, output sampling, anomaly flagging and, for agents, a log of every action taken.
Training and escalation: Everyone who builds, sets up or runs an AI system needs to know what they're allowed to do, what needs escalating and who to escalate to. A single session at onboarding isn't enough.
Audit and reporting: A regular review that checks the controls work as documented. It closes the loop between what the framework says should happen and what actually happens in production.
Is there an AI governance framework?
Yes, several, but it isn't a case of picking one. A regulated firm is usually working under all four of these at once.
A voluntary methodology. NIST's AI Risk Management Framework (AI RMF 1.0, released 26 January 2023) is intended for voluntary use. NIST added a Generative AI Profile in July 2024 and is now revising the framework as part of the White House AI Action Plan. "Voluntary" describes NIST's position, not yours. It becomes effectively mandatory the day a client puts it in a due diligence questionnaire, which is the trigger most teams actually face.
A certifiable management standard. ISO/IEC 42001, published in December 2023, is a certifiable management system standard. An accredited external auditor checks that you meet the requirements in clauses 4 to 10, and that the Annex A controls you declared applicable in your statement of applicability are actually operating. That's very different from checking you've written down your intentions. You can't document your way to a 42001 certificate.
ISO has since published two companion standards. ISO/IEC 42005 (May 2025) is guidance on assessing how an AI system could affect people and society. ISO/IEC 42006 (July 2025) sets the requirements for the bodies that audit and certify against 42001, including the competence their auditors need. Neither replaces 42001, which is still the standard organisations get certified against.
A law. The EU AI Act is a law, not a framework you adopt. It applies if you place an AI system on the EU market, or if a system's output is used in the EU.
A sector regime you're already examined on. In banking, that's model risk management. The UK's PRA SS1/23 expects banks with internal model approval to keep a model inventory, tier their models, validate them independently and monitor them. In the US, SR 11-7 and OCC 2011-12 were replaced in April 2026 by SR 26-2 and OCC Bulletin 2026-13. The new guidance is lighter and risk-based, and it leaves agents out entirely: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance." In the EU, DORA covers the ICT risk around these systems rather than the models themselves.
What the named frameworks were built for, and where they stop
The frameworks aren't naive about autonomy. Before looking at where they stop, it's worth being precise about what they cover.
Appendix C of the NIST AI RMF says AI systems "can autonomously make decisions, defer decision making to a human expert, or be used by a human decision maker as an additional opinion," and that human-AI configurations "can span from fully autonomous to fully manual."
MANAGE 2.4 of the AI RMF says:
"Mechanisms are in place and applied, responsibilities are assigned and understood to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use."
That's a kill switch requirement with named owners.
ISO/IEC 42001's Annex A controls on using AI systems (A.9.2 to A.9.4) cover processes and objectives for responsible use, and a defined intended use for each system.
ISO goes further on control in separate documents. ISO/IEC TS 8200 (April 2024) sets out principles for the controllability of automated AI systems, including how control is transferred. ISO/IEC 42105, guidance on human oversight of AI systems that builds on TS 8200, reached final draft stage in August 2026.
NIST is treating agents as open work too. In February 2026 it launched an AI Agent Standards Initiative, starting with a request for information on AI agent security and a concept paper on agent identity and authorisation.
The most rigorous academic framework covering this group of materials is the “Hourglass Model of Organizational AI Governance” developed by Mäntymäki, Minkkinen, Birkstedt, and Viljanen from the Turku School of Economics. It maps three layers (environmental, organizational, and the AI system) onto the OECD AI system life cycle and treats autonomy as a characteristic of the system layer.
MANAGE 2.4 requires a way to take a system out of operation. It doesn't say that an update to a system of record needs its own approval gate at the moment of the tool call, or that the gate has to be enforced by infrastructure rather than assigned to someone on paper.
A firm can hold a 42001 certificate and a properly documented oversight setup, and still have nothing between an agent and a change to a ledger entry, a customer message or a regulatory filing. Nothing in the certifiable material says where that gate goes or what has to enforce it. TS 8200 and 42105 move closer to the problem, but both are guidance rather than requirements a 42001 auditor certifies you against, and NIST's agent work began with public consultations this year.
How to build an AI governance framework
Building an AI governance framework follows a clear sequence.
1. Track down all your AI systems - both approved and unapproved ones. Central IT's list won't be the whole picture. Go and ask the business teams. AI tools that never appear on any register will likely be the ones causing issues later on.
2. For each system, figure out its capability first, before looking at its risk tier. What exactly can this system change? If it can only read data, then its risk profile will differ quite significantly from that of a system that actually writes to a key system of record. Every actual implementation sequence we see really starts with making an inventory and classifying, of the nine stages outlined over 30 pages by 2026, none puts capability first in the inventory. Changing this order is the real point of interest here.
3. Then work out the risk tier based on the capability. A system that simply reads customer information sits in a very different tier from one that updates customer records, which sits in a different tier from one that initiates actual payments. The specific tier will then determine which controls need applying.
4. If you're a heavily regulated company, add a capability column to your current model inventory held under PRA SS1/23 or SR 26-2 and then re-determine those tiers. Your governance framework is essentially a refinement, not a completely separate project stream.
5. Designate a single accountable owner for each system. It should be a particular person, not a whole team - who will be able to show an auditor that a system that didn't meet requirements couldn't possibly have been run.
6. Develop the acceptable-use policy and also set up the pre-deployment assessment gate. What's actually allowed? What will need approval right before deployment? And what's simply prohibited, come what may?
7. Now build the controls that enforce that policy. When the sequence reaches the stage of linking agents to external tools, an MCP gateway, the Model Context Protocol (MCP) is an open protocol creating a standard for how AI applications hook into tools and data is one key piece of infrastructure used to enforce things at a low level. You can also use platforms for ingesting existing agents or automations into infrastructure you own.
8. Establish your agent hosting system, production monitoring, audit routine, and escalation procedures. Where will you host all agents or AI your team build? Who will review the audit output? And who holds the authority to actually take down a system at 2 am if necessary?
An example of AI governance
An agent gets set up to draft and file updates to customer records. Here is what your governance structure would look like at the exact moment of action :
- During onboarding: The system gets recorded as being write-capable in the inventory.
- Risk level: High.
- Owner: Identified member of your first-line business team.
- Controls required: A pre-deployment evaluation, setting up logging within your infrastructure (and not by the agent), and having an approval gate in place right before any actual writing occurs.
- At deployment: The pre-deployment evaluation will confirm that no write paths are available to the agent that bypass the approval gate. This gate is actually implemented right within the infrastructure. It's not some kind of prompt to the model.
- In production: The agent creates a draft for updating a record. Just before the write is executed, the gate triggers again. A reviewer can see the proposed change in a format they can actually assess, rather than a raw tool call. They then either approve or reject it. If the reviewer is unavailable, the action will queue up but it won't just fail open.
Specifically, this example aims to demonstrate that the control fires right at the point of the actual action, just before the write is completed. That's quite different from most examples you'll see in the governance literature, which stop at saying "the model is monitored for drift."
Which parts are policy, and which parts are controls?
A policy simply expresses an intention. However, a control actively operates; it has an owner, leaves evidence, and has a well-defined failure mode.
The actual test auditors use is based on the following five questions applied to each part of your framework right now:
- Who performs this control, by name or by role?
- How frequently does it actually run?
- What evidence does it leave?
- What exactly happens if it does fail?
- Show me three instances from the last quarter
If there's absolutely no sign of it operating and yet you do have a policy, however it may be expressed in the document- then you've got problems.
Controls can be preventive, detective, or corrective, and this matters for agents in particular. A detective control works best when there is a very small time frame between when an action occurs and when it's detected.
For instance, for a person making a wrong entry into a record file, that time frame might be several hours long. But for an agent performing 1000 simultaneous record updates, that window will be closed at the same time as the subsequent reconciliation cycle. The real issue isn't just one single incorrect record, but all those created by the agent before anyone has even had a chance to look at them.
Documentation is not mere 'show' or display; it is essential to keeping an audit afloat. The point being made is that creation of documents and actual enforcement are two completely different tasks; which is what failure in deployed systems consistently looks like.
Where the approval boundary actually sits
The boundary isn't about read vs. write access. Rather, it concerns reversibility, where AI is hosted, the 'blast radius, ' and the data class of a given operation.
Low-value, logged writes that are reversible may still be performed but will then need to be reconciled afterward. Externally visible or irreversible writes, such as payments, communication with customers, or submitting information to regulators, or those changing permissions and entitlements will need a gate put in place before being executed.
Reads aren't always permitted automatically. A read isn't reversible if it involves taking out your data. An agent trying to read customer data to the wrong destination represents a reportable breach with a very short regulatory time frame. An agent trying to read across an information barrier is a serious market abuse control failure that doesn't even involve writing. Data class gates determine whether you can read; reversibility won't save you from needing these gates.
Approvals must come from policies and have to be pre-budgeted. Five percent of 100,000 actions a day equates to five thousand approvals a day. It's essential to document what follows: the approver becomes a rubber stamp, and a preventative control degrades into a recorded one. Tools available include rate limits, value ceilings, per-session caps, and also batched reviews of similar actions. Documenting each action at enormous scale isn't a control; it is a control that will undoubtedly fail right when it's most necessary.
If the approver is ever unavailable: fail securely or hold a queue. State who's holding that queue. An agent can never act as a valid approver of their own actions. And the actual approver needs to see what they can evaluate, approving a raw tool call in a completely unfamiliar system is essentially just adding a timestamp with zero governance behind it.
Who owns AI governance?
Accountability for AI governance actually falls onto the three traditional lines of defense. While the mapping might differ per company, the test remains the same: whoever can show an auditor that an out-of-compliance agent couldn't have executed will be the enforcement owner.
First line - The business team responsible for the particular use case runs the controls. Platform engineering builds and operates the actual enforcement point. These teams therefore own the risk they generate.
Second line - Sets the standards, defines the tiering method, and has an independent challenge function in place. In many banks and insurance companies, this sits under the risk department led by the Chief Risk Officer, with AI governance forming part of model risk management.
Third line - Internal audit will assure the framework. It is not an option, and it is not placed last on the list because it is the least important one. Within a highly controlled company, the third line really identifies the ungoverned agent: the system that wasn't recorded in any inventory, the approval gate that was never set up, the queue that will always be open. An ownership answer without third line isn't going to be considered an ownership answer to this reader.
When necessary, under the regulatory framework, an individual may be named as accountable under law. In the UK's Senior Managers and Certification Regime (SMCR), a senior manager will hold a documented statement of their responsibilities. The designation of a specific role won't be enough - there needs to be a specific person assigned.
What goes wrong in production?
Risk lists that only contain general risks won't be very helpful here. Three different failure modes can be described as follows, focusing on the mechanisms involved:
First: The absence of a preventive control over an irreversible action. Your agent writes to a critical system. The audit trail logs it. Your detective controls work as planned. Nothing intervened between the agent and the change until it had been completed. The issue isn't that the action wasn't logged; it is just that nothing required an approval request before running the action.
Second: The agent's actions aren't recorded at all because logging is included within the instructions given to the agent, not something carried out independently by the underlying infrastructure. When an agent doesn't log its activities because it's been tampered with, the instruction is overridden later, or the model doesn't follow it consistently, instruction-based logging fails.
Third: The governance framework only covers the systems that central IT is aware of. However, the agent with a SKILL.md file and MCP servers which caused the problem was created by someone in a business team, deployed into a shared or local environment, and wasn't recorded in any model inventory. It's a completely foreseeable outcome of making AI tools easy to access while taking a long time to get approval through central channels- your team ignores the rules.
When do you need one, and what does it cost to create an AI governance framework?
The trigger isn't headcount or your annual income. It is actually the very first time an AI system changes something in your system of record, or when a customer or auditor ever asks, whichever happens first. Both events will probably occur if you've set up any AI beyond being just a read-only assistant.
On cost: There's no trustworthy public figure to quote right now, as this is all so new. What the cost involves is:
Assessment work: Identifying your systems, capability mapping, risk categorization.
Documentation: Model cards, acceptable use policy, and statement of applicability if pursuing the ISO/IEC 42001 path.
Certification audit: If certification is your aim, then an external auditor accredited by the body in question.
Engineering time: Creating controls instead of creating policies is where most variation lies - and it's also something that governance budgets tend to miscount the most.
"Significant investment" is not an answer. The engineering time required to build an approval gate right at the tool-call boundary will be a very different figure from the engineering time spent making sure such a gate ought to exist in the first place. Or it will cost less if you bring in a platform that manages much of the risk.
Please specify whether you're planning to budget for the former or latter - because they are vastly different projects.
Conclusion
An AI governance framework constitutes a highly structured group of policies, procedures, roles, and controls.
NIST AI RMF 1.0 and ISO/IEC 42001 reasonably cover three of these four elements; the frameworks are sound, and adopting one would be a sensible decision.
The fourth, controls, especially where enforcement is located and at what level of detail, is where there's a gap between a governance document and governed agents live.
If you're trying to figure out exactly what your AI governance framework will need in terms of controls, not just in principles, Mindset AI works with mid-market & Enterprise teams in regulated companies. Mindset AI takes the agents, skills and MCPs your team have been creating in Claude, GPT etc to automate repeatable processes, to host on a platform that provides full audiability, governance and management to align to all frameworks discussed.
Common questions
Is there an AI governance framework?
Yes, there are a few, and they aren't mutually exclusive. The NIST AI Risk Management Framework is a voluntary methodology. ISO/IEC 42001 is a certifiable international standard, now supported by ISO/IEC 42005 on impact assessment and ISO/IEC 42006 on certification bodies.
What should AI governance include?
Nine key components: an inventory of your AI systems ordered according to their capabilities, a risk classification system, named accountable owners for each one, an acceptable use policy, prior deployment assessments, proper documentation, ongoing production monitoring, the setup of training and escalation paths \- plus audit and reporting procedures.
How to build an AI governance framework?
Locate every single AI system that you're currently using, including that 'shadow' AI that nobody really talks about. Then, for each one, establish the exact nature of the changes it will make before assessing risk. Create your controls based upon this assessment.