The AI Governance Gap NAIC Put on the Record This Summer

In the NAIC Big Data and AI Working Group summer meeting, AM Best drew a distinction between documentary governance (what should happen with an AI model) and operational governance (the true outcomes of the model in practice).

Achieving operational governance, however, depends on a second distinction that's not spelled out directly in the discussion, but sits underneath it: the difference between a system's risk profile (what decision it makes or supports, and what's at stake) and its controls (the technical mechanisms that keep it in bounds).

Here, we recap AM Best's key distinction from the meeting, while adding more of our own commentary on the risk profile and controls layer, with guidance on how to approach operational governance fully.

It’s easy to have an AI policy, but hard to show how it holds up in action

Most carriers have an AI policy by now, as many of them have adopted these alongside the NAIC Model Bulletin. Now, examiners are increasingly asking to see the work process behind the policy such as:  model inputs and outputs, the internal reasoning for whether to implement a system or restrict its use, and a model's change history as it’s monitored, tested, and updated over time. Often, this isn't well-documented, and at many carriers that record is especially thin. Regulators won't accept a new document request, particularly for initiatives that received a “no” previously.

Showing that work completely means understanding the difference between a system's risk profile and its controls. The risk profile is what decision the AI is supporting, and what's at stake for the person on the other end of it. The controls are the technical mechanisms, whether that is to support a claim payment, an adverse underwriting action, or a rate. These controls are the mechanisms that keep the system in bounds, like kill switches, rollback, and human review of the outputs. 

Two systems can carry identical controls, but still need different oversight, because they carry different risk profiles. A tool that summarizes notes for a desk adjuster to review is not the same as a system authorized to route and pay a claim on its own, even with the same safeguards in place.

Governance that stops at the controls can still leave a gap: kill switches and review thresholds get implemented, but never tied back to the specific decision they're protecting. If your program runs the same checklist on a customer service chatbot and also on an underwriting model that drives declinations, an examiner will notice this red flag and so should you. 

More thorough policies can’t fix a broken process

When regulators start asking harder questions, the initial instinct that most carriers have is to send the policy back for another round of revisions. However, more policy might give you even more to reconcile later, and it doesn't necessarily produce more evidence during the process. This is the main callout from the Summer Meeting: 

‍Documentary governance answers with policies. Operational governance answers with evidence.

In addition to clear and well-documented policies, you need a process that captures decisions and context in the moment as they change and move forward over time. That starts with the risk profile: what decision a given system supports, and what's at stake. From there, the controls you apply (and how closely you monitor them) should scale to match, not follow the same checklist regardless.

Here's what that looks like in practice:

• Inventory and classification (the risk profile): A complete, current list of your AI systems, classified by impact and autonomy, each with governance expectations attached.
• Validation and monitoring (the controls): Clear evaluation criteria and re-testing triggers before release, and a documented response when a threshold gets breached.
• Human oversight (the controls): Review that's substantive enough given the volume moving through it, with specific examples where that review changed an outcome.
• Vendor and agentic controls (the controls): Usable audit rights, tested rollback procedures, and a clear answer for what happens when a vendor update changes system behavior.

What operational governance throughout the AI lifecycle can look like in practice

Though the concepts of documentary and operational governance, and the risk and control levers underlying them might be newer for this summer, they’re a core part of AlignAI’s product philosophy and offering, and we help customers navigate this daily.

The banking ecosystem is an environment where model risk expectations have been formalized for years, making the industry a mature and focal reference point to follow. As an example, a  top-10 U.S. commercial bank  is especially no exception, but this bank now has the capability to capture 100% of AI initiative context at intake, with zero follow-up required. This includes the decisions a given system supports, what’s at stake, the risk profile itself, not just the paperwork around it. So a proposed AI system, model, or initiative’s change history exists from day one, whether it’s approved or rejected, the record is there. 

Because the risk profile is established up front, the controls that follow across validation, monitoring, human review, etc. scale to match it instead of applying the same checklist regardless of what's on the line. This made risk scoring for the bank drop from hours to minutes. Reviews had the evidence they needed instead of chasing it down after the fact. Reviewers also aren't blindly rubber-stamping something through, but making decisions with the risk context already in hand.

Reviews had the evidence needed, instead of getting chased down after the fact. This holds regardless of the number of reviews needed, because the time is saved in not chasing details down. This also means reviewers aren’t just blindly rubber stamping something to get it through, but making decisions with the information they need right at hand.

This same pattern applies to the carriers who are working toward the NAIC Model Bulletin. The ability to achieve this outcome comes from pairing a policy that defines the review process with a system that runs it, pulling in context from the tools your teams already are using, and keeping a record of what happened along the way. 

Are you ready for operational governance?

If you had to show the audit trail of last quarter’s AI initiatives, the decisions undergirding them, and the changes to any of the models or technology, could you reconstruct them at the drop of a hat? 

If that record only exists in someone’s memory, or scattered across the org in different spreadsheets or tools, or nowhere at all, it goes beyond an audit problem to become a structural problem. 

The carriers that achieve success are not only incorporating more governance. Rather, these carriers have dedicated time to strengthening and building the operational side for how AI initiatives are captured. This has been instrumental in their approach, from intake through production and beyond. See how AlignAI can help you do both.

FAQ

What’s the difference between documentary and operational AI governance? 

Documentary governance describes what should happen through written policies and procedures. Operational governance shows what actually happened across inventory and classification, validation and monitoring, human oversight, and vendor controls. This is increasingly what regulators and examiners ask to see.

Why do two AI systems built on similar technology need different governance? 

Governance requirements are set by the decision the system supports, not the underlying technology. Factors like decision criticality, degree of autonomy, explainability, speed of change, and third-party dependency determine how much oversight a given AI use case actually needs.

What does operational AI governance require insurers to show? 

A complete, current AI inventory classified by impact and autonomy; validation and monitoring with clear re-testing triggers; human oversight substantive enough to point to specific outcomes it changed; and vendor controls including usable audit rights and tested rollback procedures.

What should insurers document beyond model validation? 

Beyond validation and change history, insurers should document why a model was or wasn’t used for a given purpose, whether human review measurably changed any outcomes, and what happens operationally when a vendor update changes a system’s behavior.