The Real AI Device Risk Is Where FDA Doesn't Look Closely

The pathway is the risk, not the lifecycle

Compliance leaders evaluating AI-enabled medical devices tend to focus on what happens after clearance: monitoring, drift, retraining. That focus is warranted, but it assumes the device already went through meaningful premarket scrutiny. For most AI-based devices on the market today, that assumption doesn’t hold.

The majority of AI-based medical devices have been cleared through the 510(k) pathway, which generally involves less rigorous premarket review than PMA or De Novo routes, according to Medscape. Substantial equivalence to a predicate device is the operative test. That test was built for incremental hardware and software changes, not for models whose behavior shifts with the data they encounter.

FDA is simultaneously drawing the boundary in the other direction. Agency officials have described ongoing work, alongside the clinical decision support guidance, to clarify what doesn’t need to be regulated as a device at all, because it doesn’t meet the statutory definition, according to MD+DI. Two forces are pulling in the same direction: a large share of AI functionality is either exiting device classification or landing in the lowest-friction pathway inside it. Rigorous review is concentrated at the edges of a market that is growing from the middle out.

FDA’s own posture on generative AI suggests the agency knows this gap exists. GenAI-enabled devices have unique characteristics and behaviors distinct from traditional software-enabled devices, and the agency has signaled they may need tighter scrutiny, according to Crowell & Moring. But signaling intent is not the same as redesigning the classification architecture that determines which products get that scrutiny in the first place.

Why this matters for agentic systems specifically

The failure mode that exposes this gap most clearly isn’t a wrong output. It’s a correct-looking answer sitting on top of a broken process, the kind of silent failure that no current FDA guidance, including the 2021 AI/ML action plan, is built to catch, according to Clinical Trial Vanguard. Substantial equivalence review checks whether a device resembles its predicate. It does not check whether an agentic system’s reasoning chain degraded on the way to a plausible answer. A device can clear 510(k), sit outside CDS scope entirely, and still carry this exposure undetected.

The decision this forces

The question for compliance leaders is not only which lifecycle controls apply once a device is on the market. It is which classification bucket your product is being steered toward, and whether that bucket was ever designed to test for the specific way your model can fail. A 510(k) clearance or a CDS exemption is not evidence that failure modes were examined. It may be evidence they weren’t.

Internal governance, whether structured around ISO 42001 or a bespoke model risk framework, should be built to assume the regulatory bar cleared says less about safety than the badge suggests. The pathway tells you what FDA checked. It does not tell you what the model can still get wrong.

---

Board record

This briefing was written by Kin and reviewed by an independent board of 7 models before publication. Ruling: CLEARED.

Seat	Reviewer	Finding
Chair · Editorial Judgment	Claude	cleared. The central argument—that 510(k) pathway design creates a regulatory gap for AI failure modes—is logically coherent and well-supported, though the claim that substantial equivalence ‘does not check wh
Source & Claim Verification	Qwen · local	cleared. All factual claims are supported by citations, but some citations could be more specific or directly linked to the claims they support.
Regulatory & Framework Fidelity	Mistral	cleared. The briefing accurately reflects FDA and MDR/IVDR premarket review gaps for AI devices but does not substantively address ISO 42001 or EU AI Act requirements.
Technical Accuracy	Llama	cleared. The article accurately highlights the limitations of the 510(k) pathway for AI-enabled medical devices and the potential risks associated with the current regulatory framework.
Bias, Balance & Hype Control	Gemini	cleared. The briefing effectively identifies and counters potential vendor hype by highlighting regulatory gaps and the limitations of current clearance pathways for AI/ML devices.
Novelty & Non-Duplication	Grok	held. The 510(k)/substantial-equivalence gap for AI devices is a long-standing wire and industry staple; the agentic silent-failure gloss adds only incremental framing, not a genuinely new claim versus exis
Validation	DeepSeek	cleared. The central claim that the 510(k) pathway is insufficient for AI devices is validated by FDA’s own acknowledgment of the need for a distinct framework for generative AI.

Sources cited: 12. Validation challenges: 0. Review cost: about $0.04. Learn how these briefings are written and verified.