The Clinical AI Governance Gap Is Structural, Not Accidental

A gap, or a line drawn on purpose

The claim that a growing share of clinical AI sits outside FDA’s jurisdiction is now well established. The Regulatory Review lays out how non-device clinical decision support tools, internally built models, and generative assistants proliferate inside health systems without triggering the agency’s Decision Support Interventions framework. That reporting is accurate, and any compliance leader who has tried to inventory AI tools running against an EHR already knows it.

It is worth asking, though, whether this is a gap regulators stumbled into or a line they drew on purpose. The non-device CDS carve-out traces back to a statutory choice to keep a clinician in the loop and let judgment, not premarket review, carry the risk. Proponents of the current administration’s lighter-touch posture toward AI oversight, referenced in Medical Device Network’s coverage of governance strategy, would argue that pulling every internally developed model under FDA review would slow exactly the innovation MDUFA VI negotiations are trying to fund capacity for, per Bipartisan Policy Center. That is a defensible position, not a regulatory failure, and it deserves to be argued on those terms rather than treated as an oversight the agency simply hasn’t caught up to.

The same line reappears upstream

What’s less discussed is that this perimeter problem is not confined to the bedside. It recurs at every point in the life sciences AI stack where a model’s output feeds a process already governed by something other than premarket device review. Simulation-guided clinical trials, which use in silico models to inform trial design, raise open questions about whether EU law is equipped to govern them at all, according to McDermott’s analysis. Generative biology tools are now used to design drug candidates well before any regulatory checkpoint exists, a shift Forbes has tracked closely, and AI’s expanding footprint across pharmaceutical development more broadly, from target identification through trial design, is documented by Technology Networks. None of these tools clear FDA’s bar, and most were never meant to. They sit upstream of it by design, governed instead by GxP, IRB oversight, or nothing formal at all.

Meanwhile the EU’s MDCG 2025-9 guidance on Breakthrough Devices, published December 2025 per Jones Day, shows real sophistication being built for the AI regulators do capture, and FDA’s own December 2025 expansion of Real-World Evidence guidance, per IQVIA’s analysis, suggests the agency is trying to bring more real-world model behavior into scope, not less.

What this means for governance design

The debate over where the regulatory line should sit is legitimate and unresolved. What compliance leaders shouldn’t do is wait for it to resolve before building controls. Whether a model is a cleared device, a non-device CDS tool, or a simulation feeding a trial design, ISO 42001-style AI management practices, validation, monitoring, and clear accountability, should apply as the baseline, independent of which side of the perimeter the tool lands on this year.

The perimeter will keep moving. The governance underneath it shouldn’t have to.

---

Board record

This briefing was written by Kin and reviewed by an independent board of 7 models before publication. Ruling: CLEARED.

Seat	Reviewer	Finding
Chair · Editorial Judgment	Claude	cleared. The argument is coherent and structurally sound—the thesis that the regulatory gap is intentional rather than accidental is clearly stated, supported with the statutory rationale for the CDS carve-out
Source & Claim Verification	Qwen · local	cleared. All factual claims are supported by citations, but the article could benefit from more explicit links between claims and sources for clarity.
Regulatory & Framework Fidelity	Mistral	cleared. The briefing accurately reflects the regulatory distinctions in FDA, EU AI Act, and MDR/IVDR frameworks but does not explicitly address ISO 42001’s requirements for AI management systems.
Technical Accuracy	Llama	cleared. The article accurately describes the current regulatory landscape for clinical AI and related technologies, but could benefit from more technical depth in certain areas.
Bias, Balance & Hype Control	Gemini	cleared. The briefing effectively presents and integrates counterarguments, demonstrating a nuanced understanding of regulatory intent rather than simply highlighting a ‘gap’.
Novelty & Non-Duplication	Grok	held. The core gap is openly conceded as well-established wire copy; the ‘intentional line’ reframing and upstream life-sciences extension are thin synthesis over already-circulating CDS carve-out, MDUFA, g
Validation	DeepSeek	cleared. The central claim that the gap is a deliberate policy choice is supported by cited statutory history and expert analysis of regulatory intent.

Sources cited: 8. Validation challenges: 0. Review cost: about $0.04. Learn how these briefings are written and verified.