In 2025, FDA cleared 295 AI/ML-enabled devices. Roughly 97% of those went through 510(k). At the same time, FDA's draft guidance on AI-enabled device software functions, posted January 7, 2025, carries a cybersecurity section that asks for testing most of those submissions almost certainly did not perform. Section XII, starting on page 34, is the part nobody is talking about, and it changes the scope of what "cybersecurity testing" means the moment a trained model is involved.
This is an awareness piece, not a prediction. The requirements are sitting in a public document. The gap is that the field is treating AI cybersecurity as a slightly bigger version of the same premarket testing everyone already does. It is not. It is a different surface, tested at a different time, on different infrastructure.
The September 2023 final guidance, "Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions," is the one teams have internalized. Threat model, SBOM, security risk assessment, architecture views, vulnerability scanning, fuzz testing, penetration testing, all anchored in a Secure Product Development Framework. That guidance has been stable and well understood since 2023. The activities are mature, the deficiency letters are predictable, and a competent team knows how to close them.
Here is the trap. All of that testing is aimed at the finished device. The runtime. The thing that ships. For a conventional device, that is the right boundary, because the attack surface is the deployed software and its interfaces.
A trained model breaks that boundary. The thing that ships is only the last stage of a pipeline, and the pipeline is where most of the AI-specific risk lives.
Section XII lists seven categories of AI-specific cyber threat as examples. I am paraphrasing FDA's own examples here:
Data poisoning. Injecting inauthentic or maliciously modified data to corrupt outcomes, including diagnosis.
Model inversion and stealing. Using forged or altered queries to infer details from a model or replicate it, hitting both performance and IP.
Model evasion. Crafting input samples that push the model to a wrong classification.
Data leakage. Exploiting the model to reach sensitive training or inference data.
Overfitting. Deliberately driving a model to overfit so it is more exposed to adversarial input.
Model bias. Manipulating training data to introduce or amplify bias, including embedded backdoors that trigger later.
Performance drift. Shifting the underlying data distribution over time to degrade the model or open it to attack.
Read that list against where each threat actually materializes, and the testing problem becomes obvious.
Only model evasion is primarily a runtime, on-device threat. It attacks the deployed model with crafted input, which is exactly the surface a conventional penetration test already touches. That is the one category a standard premarket pentest covers well.
The other six do not live on the device. They live in the parts of the system that never ship:
Data poisoning hits the data ingestion and preprocessing pipeline, before training ever starts.
Model inversion and stealing target the inference endpoint and the model artifact, including wherever the model is stored and served.
Data leakage targets the training data store and the inference path, which means access controls, encryption, and de-identification at the data layer, not the device binary.
Overfitting and model bias are properties created during training. You test for them in the training and evaluation environment, against the evaluation logic, not on the shipped firmware.
Performance drift is a lifecycle threat against the data distribution and the update or retraining mechanism over time.
So if your AI-enabled submission rests on a penetration test of the deployed model, you have evidence for roughly one of the seven categories FDA named. The other six are tested somewhere your current premarket process does not reach, on infrastructure your pentest scope does not include, at a point in the timeline that is usually long past by the time cybersecurity gets involved.
Beyond the 2023 baseline, FDA's draft asks AI-enabled submissions to include the AI-specific deltas across the same deliverables everyone already produces. The risk management report, threat model, risk assessment, and labeling all need the unique AI considerations folded in. Then it gets specific. At minimum, testing should include malformed input (fuzz) testing and penetration testing shown to be appropriate to the model's risks. A Security Use Case View covering the AI considerations. And a description of controls for data vulnerability and data leakage, specifically access controls, encryption, and anonymization or de-identification of sensitive data.
The practical read is that the premarket package should show coverage across the full AI lifecycle, ideally traceable test cases mapped to each of the seven threat categories, spanning both runtime inference and the upstream pipeline where these threats live. Fuzzing the deployed model does not demonstrate that you defended the training data. A pentest of the device does not show your model registry has access controls.
Conventional premarket cybersecurity testing happens late. It is one of the last gates before submission, and that works because the device is essentially done and the surface is fixed.
Six of these seven threats cannot be tested that way. You cannot assess data poisoning resistance after the model is trained and frozen. You cannot evaluate overfitting or embedded bias on a finished binary. Those properties were set during training, and the evidence has to come from the training environment, captured while it still exists.
That inverts the schedule. If an AI-enabled device skips this testing and the gap surfaces late, the remediation is not a patch. It is a retrain. Retraining means new data handling, new evaluation, new evidence, and potentially new clinical performance work depending on how much moves. That is schedule and budget risk on a different order than a typical cybersecurity deficiency, and it lands at the worst possible point, right before or right after submission.
The AI-enabled guidance is still draft. As of late June 2026 it has not been finalized, the comment period closed April 2025, and the public deficiency feedback that would tell us exactly how hard FDA enforces Section XII has not surfaced, mostly because these products are only now reaching submission in volume. So I am not claiming this is settled enforcement. I am claiming the requirement is written down, the testing it describes is fundamentally different from what teams do today, and the cost of being wrong is asymmetric.
If you are building anything with a trained model in it, and "AI-enabled" means exactly that, a model trained off-device and placed on the device for real-time use, the conservative move is to treat the upstream pipeline as in-scope for cybersecurity and to test it early, during training, while the environment and the data still exist. Doing it and not needing it costs you some test cases. Not doing it and needing it costs you a retrain.
The 2023 guidance told you how to secure the device. Section XII is telling you the device was never the whole system. Most of the 295 cleared last year were scoped to the device. That is the gap worth closing before the final guidance removes the ambiguity.
Jason