Tips to mitigate the risk that medtech teams keep leaving unowned as artificial intelligence moves into Class II and III medical devices.
Dr. Sarah Matt is a surgery-trained physician-executive who advises device and health-technology companies on clinical AI governance, commercialization, and adoption. [Photo courtesy of Dr. Sarah Matt]
By the time a clinician trusts a wrong output from your artificial-intelligence-powered device, every decision that made the trust possible was made years earlier.
How many decisions go into a wrong output? Design made a decision. Quality made one. Regulatory made one too. But none of them thought they were making an actual clinical call. The decision rooms of product development teams are often completely closed to the clinicians and true users of most of the innovations we create.
The risk of the tools providers use everyday are being pushed off the clinician and back onto the AI model. These models are built far upstream where no single function is assigned to own what it actually decides.
The question worth answering before clearance is simple: Where does that gap sit, and who closes it?
In a mechanical device, the clinician makes the clinical call at the bedside. In an Al-enabled device, that judgment increasingly happens upstream, inside the model itself. [Image courtesy of Dr. Sarah Matt]
The handoff nobody signs for
A mechanical valve or a piston do what they were machined to do. They do the same thing on the first use as the ten-thousandth time. A model doesn’t come with that same guarantee.
Once a device’s logic stops being fixed and starts being learned, part of the clinical judgment that used to live with the provider moves into the software. A tool that flags a finding or recommends a setting is making a “decision” an actual person used to make alone. With the increasingly massive number of tools, patients, and tasks, those decisions are often deferred to the device without a second thought.
Most companies are set up for the basics of piston quality and a “fix-it-and-forget-it” mentality. Design will treat the model as a component among many. Quality will treat it as either “documented” or “not.” The regulator will treat it as “cleared” or “not.” Our user — the clinician at the bedside — will assume someone upstream understood exactly what was shipped.
Everyone is partly right, which is exactly how the gap stays invisible.
Now what? Name a clinical-risk owner before submission, not after the first field complaint. One person on the org chart needs to be accountable for what the model does to a real patient.
Design, quality and regulatory, and clinical and commercial each treat the model as someone else’s responsibility. The clinical-risk gap sits unowned in the center. [Illustration courtesy of Dr. Sarah Matt]
An algorithm treated like a “part”
FMEA and hazard analysis are core ways we ensure quality standards are met and kept. But these processes were built for parts like a bracket that cracks or a seal that leaks. The failures are visible and repeatable so the tools handle them well.
AI, on the other hand, fails differently. It drifts as real-world data pulls away from the training set. It breaks when the distribution shifts and it meets a population or a care setting it never saw in development. It invites automation bias, where tired clinicians stop questioning it. So while the model can hold its accuracy, it can still lose calibration on the one subgroup that matters most, and nothing in a traditional risk file is built to catch it.
Now what? Treat drift, distribution shift and other changes as first-class risks, each with a named detection method and mitigation. Regularly monitor the disaggregated subgroups, not just the overall accuracy.
Traditional risk files catch visible, repeatable hardware failures. They have no equivalent column for drift, distribution shift, or automation bias. [Image courtesy of Dr. Sarah Matt]
Everyone owns it, so nobody does
Let’s walk through the life of a model. First it locks at submission, gets deployed into a hospital, and then drifts in month seven. Nothing looks off until it throws an error in month nine. Now we have a real issue.
So who acts, and how fast? Design says the model is “working as designed.” Quality says the complaint is logged. Regulatory says the clearance still stands. All while the commercial team is getting hammered by customers with hard questions. Each function “owns” it, so nobody does. The patient-facing risk sits between them, essentially unstaffed.
The FDA’s predetermined change control plan (PCCP) defines what is allowed to change. But it doesn’t go so far as to name the person who owns the consequences when it does. Everything is crisp at launch, but that can all change during the slow degradation that no one is truly watching. And nobody is really paying attention because “watching” was never written into anyone’s job description.
Now what? Assign an owner to each stage of the model’s lifecycle and decide upfront the degradation response well before you ship. Know what signals trigger action, who acts on them, and how fast. A field complaint should meet a well thought out plan, not a frantic fire drill.
Accountability is clear at submission and deployment. By the time drift produces a field signal months later, no function on the org chart claims ownership. [Image courtesy of Dr. Sarah Matt]
Build it in, or else …
None of this seems far-fetched. It’s the discipline device teams already apply to every physical failure mode, extended to the one part of the system that keeps changing after it leaves the building. The teams that decide who answers for the model before they submit will be the most successful.
Build that owner in before clearance, or face a public crisis later.
Vital Werks founder Sarah Matt, MD, MBA, is a surgery-trained physician-executive and former Oracle Health VP who advises device and health-technology companies on clinical AI governance, commercialization, and adoption. She provides internal medicine charity care and is the author of a national bestseller, “The Borderless Healthcare Revolution.”
Read more MDO Contributions and learn how to submit your own.
The opinions expressed in this blog post are the author’s only and do not necessarily reflect those of Medical Design & Outsourcing or its employees.
