Showing clinicians why an AI decided helped on average and hurt some readers

Explainable AI — models that show a human why they produced a given output rather than just producing it — is close to an article of faith in medical AI policy. The reasoning is that a clinician who can see the model's basis will trust it appropriately: relying on it when it is right, overriding it when it is wrong. A study published this month in npj Digital Medicine tested that assumption with actual clinicians and found the picture is less tidy than the argument [s1].

The design

The researchers adapted a prototype-based explainable AI model for estimating gestational age from ultrasound images, then ran a three-stage reader study with ten sonographers [s1]. Prototype-based explanation works by pointing to reference examples the model considers similar to the case in front of it, rather than producing a heat map over the image.

The three stages let the researchers separate two things that are usually confounded: the effect of seeing a model prediction at all, and the additional effect of seeing an explanation for it. They measured accuracy, self-reported trust, reliance on the model, and a novel measure of appropriate reliance — whether a reader followed the model when following it was the right call [s1].

The results

Model predictions alone cut clinician mean absolute error from 23.5 days to 15.7 days [s1]. That is a large improvement, and it is the least surprising part of the study: gestational age estimation from a single image is a task where an algorithm trained on a large labelled set can reasonably outperform an individual reader.

Adding explanations reduced mean absolute error further, to 14.3 days, but this additional improvement was not statistically significant [s1].

The finding the authors treat as the important one sits underneath that average. The impact of explanations varied across participants, with some performing worse with explanations than without [s1]. And although explanations increased participants' confidence, they had no significant effect on trust or on reliance on the model [s1].

That combination — more confident, not more trusting, not more reliant, and in some individuals less accurate — is close to the opposite of what explainability is generally assumed to deliver. Confidence that rises without accuracy rising with it is the specific failure mode that makes decision support dangerous rather than merely useless.

Why ten readers is both a limitation and the point

A ten-participant study cannot support strong general claims, and the authors do not make them. The sample is small, the task is narrow, and the model is a research adaptation rather than a deployed product.

But the design answers a criticism the paper opens with: that explainable AI studies often rely on automated metrics rather than user evaluation [s1]. Automated explainability metrics score how faithfully an explanation reflects the model's computation. They say nothing about what a human does after reading one. A study with ten clinicians that measures human behaviour tells you something an automated benchmark on ten thousand cases does not, which is that the between-clinician variance is large enough to swamp the average effect.

How this connects to what gets cleared

The regulatory context makes the finding harder to dismiss. A systematic review published a week earlier in JAMA Network Open examined every AI/ML device with FDA premarket authorisation from November 1995 to June 2024 and found that of 717 radiology devices with submission documentation, 56 — 8% — included a human in the loop in their testing [s2]. Most AI/ML devices are used in conjunction with a human, the review notes, yet only that small fraction were tested with any human operator [s2].

The review also cites a study of radiologists reading chest radiographs with AI assistance in which high performers maintained strong performance while low performers did not necessarily improve, and concludes that such heterogeneity raises questions about clinical generalizability [s2]. The npj Digital Medicine study is a direct instance of that heterogeneity, measured prospectively in a controlled reader study rather than inferred [s1].

What this does not mean

It does not mean explanations are harmful. On average they helped, and the direction of the point estimate was favourable [s1]. It does not mean clinicians should not see model reasoning; the study measures one explanation format, on one task, with one small group of specialists.

What it argues against is the assumption that explainability is a solved safety mechanism — that adding an explanation panel converts an opaque model into a supervised one. The paper's own framing is that these counterintuitive results highlight potential pitfalls in deploying explainable AI, and emphasise the need for human studies to capture clinician variability [s1].

What to watch

The measurable question is whether human-in-the-loop testing appears in a larger share of device submissions over time, or stays near 8% [s2]. The harder one is whether anybody starts reporting the distribution of clinician performance with a tool rather than the mean — because a device that improves the average reader while degrading the weakest ones is a different product from one that lifts everybody, and current evidence standards do not distinguish them.

Sources

• The human factor in explainable artificial intelligence: clinician variability in trust, reliance, and performance — npj Digital Medicine, 2025-11-14
• FDA Approval of Artificial Intelligence and Machine Learning Devices in Radiology: A Systematic Review — JAMA Network Open, 2025-11-07