Agentic AI Governance in Healthcare: An Honest Map

Status (updated 25 July 2026), next review 1 October 2026.

No regulator has published a framework written specifically for agentic AI in healthcare. That isn't a gap we can fill for you with a confident answer, and we're not going to pretend otherwise. What does apply is the existing obligation set, and health data and AI governance sets out how the pieces fit together. What follows is an honest map of how existing rules apply today, where they genuinely strain, and how to build a defensible governance posture in the meantime.

In short: No regulatory framework yet addresses agentic AI in healthcare directly. Agentic systems are assessed under existing rules: if the agent has a medical purpose it's a medical device under MDR, and if Notified-Body-assessed it's high-risk under the AI Act. Autonomy, tool use, and continuous change create open questions both frameworks strain to answer.

What makes agentic AI different for regulators

Most of MDR and the AI Act were written with a mental model of software as a fixed, bounded system: it takes a defined input, applies a defined process, produces a defined output, and that behaviour is validated once and then locked, changing only through a controlled, documented, re-validated update. An agent breaks that model in three specific ways.

Autonomy. A traditional clinical decision-support tool presents information; a clinician acts. An agent can be designed to take multi-step action toward a goal with less human intervention at each step, which changes where in the chain human oversight — a requirement the AI Act imposes directly through Article 14, and MDR reaches through its usability-engineering and risk requirements — actually has to sit.

Tool use. An agent that can call external tools, query other systems, or invoke other models introduces a composition problem. The agent's behaviour depends not just on its own training but on the behaviour of everything it calls, much of which may not have been validated as part of the same regulatory submission.

Continuous change. Agentic systems, particularly those built on top of large underlying models that are themselves updated by their providers, can change behaviour without the deploying organisation initiating or controlling that change in the traditional sense. That's a structurally different problem from the "we shipped a new model version" change management both MDR and FDA's evolving AI/ML frameworks were built to handle.

None of these three properties is unprecedented on its own. What's genuinely new is having all three at once, in a healthcare context, where the consequence of an unvalidated behaviour change is patient harm rather than a degraded recommendation feed.

How existing frameworks apply today

Regulators haven't published agent-specific rules, but that doesn't mean agentic systems in healthcare are unregulated. They're assessed under the same tests as any other software, applied honestly to a harder case.

Under MDR, the question is unchanged: does the agent have a medical purpose, diagnosing, treating, monitoring, or preventing disease, regardless of how it arrives at its output? An agent that autonomously adjusts a treatment recommendation, flags a clinical concern, or takes an action affecting patient care meets the same medical-purpose test as any other software, and classification under Rule 11 follows the same logic based on the significance of the decision it informs. MDR Rule 11: Why Software Lands in Class IIa covers that logic in detail; nothing about agentic architecture exempts a system from it.

Under the AI Act, if that agent is a medical device requiring Notified Body assessment, Article 6(1) makes it automatically high-risk, the same as any other AI-enabled device. Two Routes to High-Risk AI in Healthcare covers this in full, including where the Digital Omnibus timeline currently stands. The obligations that follow, risk management, data governance, human oversight, logging, apply to the agent the same way they apply to a non-agentic model, even though satisfying them is genuinely harder in practice, particularly human oversight and logging, given multi-step autonomous action.

The honest summary: existing frameworks do apply to agentic AI in healthcare. They weren't written with agents specifically in mind, so satisfying their underlying safety intent, genuine human oversight, meaningful traceability, is harder for an autonomous, tool-using, continuously-changing system than for a static model. That gap between "the rule technically applies" and "the rule's safety intent is actually satisfied" is where the real governance work sits.

The genuinely open questions

A few questions don't yet have settled answers, and we'd rather say so than manufacture false confidence.

Change control. If an agent's behaviour can shift because an underlying foundation model provider updates that model, who's responsible for re-validating the agent's clinical behaviour, on what cadence, and how does that fit inside MDR's change-management and significant-change notification requirements, which were built around the manufacturer controlling the change? FDA's predetermined change control plan framework, built for AI/ML devices, offers the closest existing template. FDA PCCP: Updating AI Models After Clearance covers how it works, but it presumes changes the manufacturer specifies in advance, a different problem from a dependency changing underneath you.

Human oversight in a multi-step process. Both MDR and the AI Act (Article 14) require meaningful human oversight. What that means for a single-step recommendation is well understood. What it means for an agent that takes several autonomous actions toward a goal, each individually low-risk but cumulatively consequential, isn't yet settled in guidance from any regulator we track.

Liability chains. When an agent calls a tool, and that tool's output contributes to an adverse outcome, existing product liability and MDR vigilance frameworks assume a single manufacturer accountable for a single product's behaviour. An agent composed of multiple models and tools, potentially from multiple vendors, strains that assumption in ways liability law hasn't yet caught up with.

We aren't aware of binding guidance resolving any of these three questions specifically for agentic systems as of this page's status date. Where we say "unsettled," we mean it: this isn't a rhetorical device, it's the actual state of the regulatory landscape.

A worked example: an agentic triage assistant

The questions above are easier to reason about with a concrete system in mind. Picture an agent deployed in a hospital's intake workflow: it reads a patient's reported symptoms, pulls their recent lab results from the EHR, checks those results against a reference range tool, and drafts a recommended triage category for a nurse to review before it's finalised.

Under MDR, this agent has a medical purpose, informing a triage decision, so it needs classification under Rule 11 like any other clinical decision-support software; given the consequence of a missed urgent case, it likely lands at Class IIa or above. Under the AI Act, once it's Notified-Body-assessed, Article 6(1) makes it automatically high-risk.

Here's where the open questions in this article stop being abstract. The agent's behaviour depends on the reference-range tool it calls, a third-party component whose provider can update it independently, which is exactly the change-control problem described above: nobody at the hospital controls when that update happens, but the agent's clinical output can shift because of it. The "nurse reviews before it's finalised" step is the human oversight control, but if the agent has already pulled data, applied a reference check, and drafted a recommendation before the nurse sees anything, the practical question is whether that oversight point comes early enough in the chain to be meaningful, or late enough that the nurse is mostly rubber-stamping a conclusion the system has already reached. And if that reference-range tool returns a wrong value that contributes to a delayed triage, the liability question, whose fault, the hospital's, the agent developer's, the reference-tool vendor's, doesn't have a settled answer under current frameworks.

None of this means the system can't be built or deployed responsibly. It means the responsible path runs through documenting these exact tensions and designing around them deliberately, rather than assuming existing MDR and AI Act paperwork templates will surface them on their own.

Evaluating a vendor-supplied agentic component

Many teams building agentic healthcare products aren't training their own foundation models; they're orchestrating one or more third-party models and tools inside an agentic framework they built themselves. This is the most common real-world shape of the problem, and it raises a practical question distinct from the theoretical ones above: what do you need from a vendor to defend your own regulatory position?

At minimum, document which specific model version or versions your agent relies on, whether and how the vendor notifies you of material changes to that model, what testing you've run against your specific intended use rather than relying on the vendor's general-purpose benchmarks, and what fallback behaviour your system exhibits if a vendor-supplied component becomes unavailable or behaves unexpectedly. None of this is a formal regulatory requirement written specifically for vendor relationships, but all of it is the kind of evidence a Notified Body or FDA reviewer will expect to see once they understand your product's architecture depends on components you don't fully control. Treating vendor dependency as a risk to document, rather than a detail to omit, is the more defensible position.

Why "move fast" doesn't work here, even for founders used to it

It's worth addressing directly why the usual startup instinct, ship something, learn from real usage, iterate, doesn't map cleanly onto agentic systems in healthcare. In most software categories, the cost of an early mistake is a bad user experience or a lost customer, both recoverable. In this category, the cost of an early mistake in oversight design or change-control discipline is a governance debt that compounds. An agent deployed without clear autonomy bounds, without logging fine-grained enough to reconstruct an incident, or without a documented rationale for where its human-oversight checkpoints sit, doesn't just carry ordinary technical debt. It carries evidence gaps that are difficult or impossible to fill retroactively, precisely at the moment a regulator, a Notified Body, or a plaintiff's lawyer is asking hardest for that evidence.

This doesn't mean moving slowly for its own sake. It means recognising that the specific categories of work described in this article, autonomy bounds, oversight-point documentation, granular logging, are cheap to build in from day one and expensive to bolt on later, which is a different trade-off than most early-stage engineering decisions.

What FDA's PCCP thinking suggests

FDA's predetermined change control plan framework, developed for AI/ML-enabled devices generally, is the most useful existing template for thinking about agentic change management, even though it wasn't written with agents specifically in mind. Its core idea, specify in advance the bounds within which a system is allowed to change, and get those bounds authorised as part of the original submission, so changes within them don't require a new submission, offers a structural pattern that could extend to agentic systems: define the agent's permitted tool set, action bounds, and escalation triggers in advance, and treat any behaviour outside those bounds as requiring re-validation. This is a suggestion about direction, not a claim that PCCP as currently defined already covers agentic systems. It doesn't, and applying it to an agent would require documentation and justification a regulator hasn't yet reviewed for this use case.

A defensible governance posture for builders now

In the absence of agent-specific rules, the defensible position is to apply existing frameworks conservatively rather than wait for guidance that doesn't yet exist. Classify your agent under MDR and the AI Act using the same rigour you'd apply to any other software, without treating agentic architecture as a reason for a lighter touch. Design human oversight into the workflow at points that correspond to genuine clinical decision moments, not just at the start or end of a multi-step process, and document why those points were chosen. Define and document the bounds of the agent's autonomy, what tools it can call, what actions it can take without escalation, what triggers a stop, as though you were writing a change control plan, even though no regulator currently requires you to submit one in this form. Build logging that captures not just the agent's final output but its intermediate steps and tool calls, since traceability requirements under both frameworks will be far easier to satisfy retrospectively if you capture this from the start rather than reconstructing it after an incident.

What we're watching

We track FDA and EU regulatory guidance specifically for agentic AI and will update this page when either publishes agent-specific rules, or when Notified Body practice on agentic systems visibly settles into a consistent pattern. One piece of the broader landscape has moved since this article was first drafted: the Digital Omnibus package affecting AI Act timing, discussed throughout this content set, was published in the Official Journal on 24 July 2026 as Regulation (EU) 2026/1744 and enters into force on 27 July 2026 — so the deferred AI Act high-risk dates (2 December 2027 for Annex III; 2 August 2028 for Article 6(1) medical devices) are now the legally binding ones, superseding the original dates, and the timing question elsewhere in this cluster is closed. It doesn't touch any of the three open questions specific to agentic systems described above; those remain unresolved regardless of the Omnibus. A second watch item: the Commission's MDR/IVDR simplification proposal (COM(2025) 1023, procedure 2025/0404(COD)) would move Rule 11 toward a risk-based model — a proposal, not law, but one that could shift the Class IIa assumption our worked examples rely on. This is a fast-moving area with genuinely low current competition for citable, honest content, which is precisely why we're publishing an honest map rather than a confident one. Status (updated 25 July 2026); next review 1 October 2026.

If you're building an agentic system in a healthcare context now and want to work through your specific governance posture, that's a conversation better had directly than answered generically on a page like this one. The right answer depends heavily on exactly what your agent does and where its autonomy sits.

Frequently asked questions

Is an agentic AI system automatically higher-risk than a non-agentic one under current rules? Not automatically, in the formal legal sense: classification under MDR and the AI Act follows medical purpose and risk significance, not architecture. In practice, though, the properties that make agents harder to govern, autonomy, tool use, continuous change, tend to make the underlying risk analysis more demanding, even where the formal classification outcome is the same as a non-agentic equivalent.

Can I wait for agent-specific regulation before building governance controls? We wouldn't recommend it. No regulator has committed to a timeline for agent-specific rules, and the practical risks described in this article, unvalidated behaviour change, unclear oversight points, unresolved liability, exist regardless of whether a rule names them yet.

Does using a well-known foundation model provider reduce my regulatory burden? It can reduce technical risk if the provider has strong change-management practices, but it doesn't reduce your documentation burden. You're still responsible for demonstrating your own system's safety and for showing you understand and have accounted for your dependency on a component you don't control.

Should agentic systems avoid healthcare entirely until rules catch up? That's a business and risk-tolerance decision, not one this article can make for you. What we can say is that the frameworks that do exist, applied conservatively, provide a workable, if imperfect, basis for responsible deployment today; several teams are already building on that basis rather than waiting.

How do I know if my product counts as "agentic" for the purposes of this article? There's no formal legal definition to apply, so we'd suggest a practical test: if your system takes more than one action toward a goal without a human approving each individual step, or if it can call external tools or other models as part of reaching its output, the governance questions in this article apply to you. A single-step model that produces a recommendation for a clinician to accept or reject isn't agentic in this sense, even if it's sophisticated; a system that gathers data, reasons across multiple steps, and acts on intermediate conclusions is.

---

Where next: When Medical Device AI Becomes High-Risk · Two Routes to High-Risk AI in Healthcare · FDA PCCP: Updating AI Models After Clearance

Talk to someone about your specific agentic governance posture. Book an expert conversation →