The AI Testing Newsletter · Issue 1

Myth about Mythos

This is the first issue of this newsletter, and the thesis is straightforward: AI is rewriting product security from end to end, the discovery side, the triage side, the tooling side, the regulatory side. Every issue going forward will dig into a piece of that. This one is about what's actually happening on the discovery side, and why most of the findings you're about to drown in don't matter.

If you run product security at a medical device company, this is for you. If you are in quality or regulatory, this is for you. If you're a researcher building the offensive tooling, this is also for you, I'd rather you read this than not.

If you are in product security at a medical device manufacturer, your week probably looks something like this: a CVE drops, someone forwards it to you, you spend two days figuring out whether it applies to your product, you put it on a row in your cybersecurity risk assessment spreadsheet, you move on. Maybe you do that fifteen or twenty times a month. Maybe more.

That model is finished. Not in five years. Now.

The volume of new vulnerability findings hitting product security teams has gone vertical in the last twelve months, and it isn't because the world suddenly got less secure. It's because AI agents are doing the discovery work that used to take a researcher a week, in an hour, against any target that's reachable. The same agents are reading your firmware, your SBOM, your protocol stack, and your public docs simultaneously, and producing findings faster than any human triage process was built to absorb. Legacy devices on Ebay? Yes, those are prime targets for runtime vulnerability discovery - even more lucrative than firmware.

We've spent more than a decade identifying vulnerabilities in medical devices. Thousands of vulnerabilities found by hand across hundreds of unique products. Everyone has heard the news about the Mythos. We have the backstory. Over the past year, we've been developing agentic vulnerability discovery specifically for medical devices (software, firmware, and hardware). Perhaps we are the current "experts" in AI testing for medical devices. Only time will tell. It's been hard work, but not without struggles. We faced challenges from Anthropic and OpenAI but were allowed back as a Certified Cybersecurity Partner. The truth is, it's not what you think. It's not a prompt; it's the harnesses and the agentic feedback loops that make it work. When done right, in concert, it's incredibly powerful. Everyone should take the time to understand its current state and its future potential.

There's a misconception worth clearing up before going further. When people hear "AI found this vulnerability," they picture a magic prompt to a magic model, drop the firmware in, get the bug out. The model is the whole show. It isn't. The frontier reasoning models are genuinely remarkable at this work, and that reasoning is the engine of everything described above. But reasoning isn't output.

To turn that reasoning into actual vulnerability discovery against a real target, you need the swarm around it, the agentic loop, the tool integrations, the shared filesystem, the coverage feedback, the domain skills that tell the model what a CRC actually is or how a DICOM frame is structured. That scaffolding is where the work happens. Plug a frontier model into a well-built swarm with the right tools and skills for your application and yes, the results are extraordinary. Hand the same model a prompt with no harness and you get a confident-sounding paragraph that misses the actual bug. The model is necessary. It is not sufficient. Anyone telling you otherwise is selling you the prompt and hoping you don't notice there's no engine behind it.

How AI agents are doing vulnerability discovery

The mental model most people still have is one model, ChatGPT, Claude, whatever, being asked "find me a bug in this code." That's not what's happening. What's actually being deployed is a swarm.

A recon agent enumerates the target. It pulls the SBOM, parses firmware metadata, maps the protocols, reads the public documentation. Its only job is to produce a structured model of the target.

A fuzzer agent mutates inputs against that target and logs crashes. It doesn't think. It generates traffic.

A coverage agent watches what code paths the fuzzer is actually hitting. When the fuzzer gets stuck, and it always does, usually on a CRC check, a length field, or a magic byte, the coverage agent figures out why and hands the fuzzer a structured fix. "Bypass the CRC at offset 0x4A2. Here's the polynomial." The fuzzer updates its mutator and breaks through within minutes.

A verification agent picks up the crashes and decides which ones are reachable in the real environment.

An orchestrator keeps the plan, the budget, and the success criteria.

None of these agents carries the others' context. They communicate through small structured handoffs and a shared filesystem. The filesystem is the team's long memory, progress notes, crash corpus, coverage maps, findings list. When any one agent's context window fills up, it compacts, reads its own progress file, and picks back up.

That's the whole trick. A team of focused workers, a shared scratchpad, a feedback loop that iterates faster than any human pentester can.

Who can do this now

The honest answer is: anyone with a credit card, but without prior expertise in vulnerability research, the results will "unexploitable" vulnerabilities. Still a vulnerability, but not something that can be exploited in the product today from the attack surface that exists - that's where this get wilder, requiring a pipeline of agentic workflows.

Anthropic shipped the Claude Agent SDK earlier this year. It's the same runtime that powers Claude Code, exposed as a library. Subagent orchestration, session persistence, automatic context compaction, fourteen built-in tools, MCP support for plugging in your own. OpenAI and Google have equivalents in various states of maturity. The plumbing that used to be hard, the agent loop, the tool execution, the context management, is now pip install.

A motivated security researcher running this against a public-facing target costs tens of dollars in compute. A motivated adversary running it against a niche embedded device, say, a hospital's MRI scanner exposed through a misconfigured VPN, costs a few hundred. There is no nation-state premium on this work anymore. The barrier collapsed sometime in the last nine months and most of the industry hasn't caught up.

What it actually means

A few things, in order of how much they should worry you.

One. The volume is going to keep climbing. CVE counts were already growing exponentially. They're about to inflect again. The teams I talk to are already seeing 2--3x the inbound finding volume they were dealing with a year ago, and that's before the second wave of autonomous tooling really lands. If your triage process was already strained, it's about to break.

Two. Most of those findings don't matter to your product. This is the part that doesn't get enough airtime. An AI agent finding a CVE in a library you ship is not the same as that CVE being exploitable on your device, given your trust boundaries, your network position, your clinical use context. NVD's CVSS score is generic. It assumes the vulnerable function is reachable, the attacker has the access they need, and the impact is what the researcher described. On a regulated medical device with a tightly scoped attack surface, the real number is often substantially lower --- and sometimes the vulnerability isn't reachable at all.

The unfortunate corollary: the volume problem and the relevance problem multiply. You're not just getting more findings. You're getting more findings that look serious and aren't. Your team will spend its hours triaging things that don't matter while the things that do get buried in the queue. Unfortunately, it almost doesn't; matter that they are not exploitable, you must still risk assessment them all.

Three. The current playbook can't scale through this. Manual triage by a security engineer, a quarterly pentest engagement, an annual threat model refresh, these were built for a finding volume that no longer exists. You can't hire your way out of this; the people who can do the work aren't on the market, and even if they were, the math doesn't pencil. The only path through is to absorb the volume with automation that's at least as sophisticated as the discovery tooling generating it.

Four. The regulatory bar is moving, but slowly. FDA 524B, EU CRA Article 14, NIS2 Article 23, IMDRF guidance, they all expect continuous postmarket vulnerability management. They don't tell you how to handle the new volume. Auditors are about to start asking, though. The first time a regulator asks you "how did you determine this CVE wasn't exploitable on your device" and your answer is "a security engineer wrote a memo," you're going to wish you had something better.

What to actually do

I'll cover the defensive side in detail in future issues, but the short version is this: the same primitives that make the attack cheap make the defense cheap. The agent swarm pattern is symmetric. The team that points it inward, at their own products, has a structural advantage over the team that's still running point-in-time assessments.

The teams that win the next 24 months will:

  • Build a continuous, AI-driven triage process that absorbs new CVEs the day they drop and tells the developers which ones are real on their specific products.
  • Verify exploitability with evidence, not opinion. "An engineer said it wasn't reachable" is not going to hold up. "We ran an automated verification against the firmware build and confirmed the vulnerable path is unreachable, here's the artifact" will.
  • Treat the threat model of every product as a persistent asset that gets smarter with every assessment, rather than rebuilding context for every engagement.
  • Generate regulatory evidence as a byproduct, not as a separate deliverable.

None of that is theoretical. But the gap between teams that have moved and teams that haven't is widening fast.

What this newsletter will be

Weekly. Short. Focused on what's actually changing in product security as AI rewrites both sides of the fence. New CVEs that matter, regulatory updates that do or don't, tooling shifts on the offensive and defensive side, and field notes from running real assessments against real devices.

No vendor pitches or "in today's rapidly evolving landscape" rhetoric. Just what I'm observing. Yes, we operate in this space. Clearly, we provide services and products. We are the largest (by volume) medical device pentesting company in the industry. Consequently, we possess valuable insights and feel a sense of responsibility to share the events that we believe are shaping the industry.

Jason Sinchak
CEO, ELTON
Exploitability management for medical devices. FDA §524B methodologyExploitability proven at runtime95% faster than legacy testing Book a Demo
Platform
Platform OverviewDigital TwinAutonomous TestingExploitability VerificationFind the 1%Remediation OptimizationELTON TestLink™SBOM, VEX & ReportingCVSSv4 MigrationProduct Tour
Solutions
Postmarket SurveillanceIncident ResponseSecurity EngineeringRegulatory AffairsFDA §524BEU MDR/CRAEU REDNIS2IMDRF N60 / N73Japan MHLW
Why ELTON
Why ELTONProof Over ProbabilityFind the 1%AI PentestingMDDT MethodologyCredentialsDevice ModalitiesPricingELTON vs. Legacy Testing
Resources
FDA Deficiency ListFDA Testing RequirementsFDA Cyber SOPs & TemplatesRemediation LibraryRegulatory GuidesWebinarsThe End of Legacy TestingThe AI Vulnerability ExplosionSecurity AdvisoriesWhitepapersIntelligence & Blog
Company
AboutLeadershipCareersContact Book a Demo