Models beat German medical students on text — and fell apart on the picture questions

Benchmarks of large language models against medical exams have a credibility problem: most of the exams have been on the public internet for years, so a high score may be measuring memorisation rather than reasoning. A study published in npj Digital Medicine on August 31 tries to close that hole by using a question bank that is not public.

The authors evaluated proprietary and open-weight foundation models on 24 official German medical licensing examinations sat between 2019 and 2024 — 7,485 items in total, alongside item-level response data from 119,878 student sittings [s1]. The German papers are a restricted-access benchmark with, in the authors' phrase, reduced public exposure [s1]. That is the point of using them.

What the models scored

On the full benchmark for the first examination (M1, n = 3,747), Gemini 3.1 Pro was most accurate at 99.31% (95% CI 98.99–99.53), followed by GPT-5.4 at 98.48% (95% CI 98.03–98.82) and Claude Opus 4.6 at 96.93% (95% CI 96.33–97.44) [s1]. On the second examination (M2, n = 3,738), Gemini 3.1 Pro again led at 98.37% (95% CI 97.91–98.73), with Claude Opus 4.6 and GPT-5.4 both at 97.30% [s1].

Because not every model accepts image input, the authors also built a shared text-only subset (M1 n = 3,227; M2 n = 3,244) so all thirteen models could be compared on identical items [s1]. There, several open-weight systems were close behind the frontier models: on M1, GLM-5 reached 99.10%, Kimi K2.5 98.76% and DeepSeek V3.2-Thinking 98.73%, against 99.63% for Gemini 3.1 Pro [s1]. Even compact models exceeded mean student performance [s1].

For scale, mean student accuracy was 71.21% on M1 text-only items and 74.66% on M2 text-only items [s1].

The finding that is actually interesting

The headline accuracies are near-ceiling and largely uninformative. The modality analysis is not.

Items that contain an image were harder for everyone. Mean student accuracy fell from 71.21% on M1 text-only items to 64.27% on M1 image-present items, a gap of 6.94 percentage points, and from 74.66% to 70.74% on M2, a gap of 3.92 points [s1].

The models' drops were mostly much larger. On M1, Gemini 3.1 Pro fell from 99.63% to 97.31% — a gap of 2.32 points, the only model to lose less ground than students did [s1]. GPT-5.4 fell from 99.50% to 92.12% (7.39 points) and Claude Opus 4.6 from 99.26% to 82.50% (16.76 points) [s1]. Among open-weight models, Kimi K2.5 fell 14.72 points, Qwen3-VL 28.61 points, and Mistral Large 3 dropped from 94.45% to 46.15% — a gap of 48.30 points [s1].

Absolute percentage-point gaps are a poor comparison when one solver sits at 99% and another at 71%, so the authors also report an error-rate ratio: the error rate on image items divided by the error rate on text items, which is invariant to baseline accuracy [s1]. For students, that ratio was 1.24x on M1 and 1.15x on M2 [s1]. For proprietary frontier models on M1 it ranged from 7.2x to 23.5x [s1]. On M2 the model ratios were smaller — 3.8x to 5.0x for frontier models — but still several times the human figure [s1].

The authors are careful to say the ratio is itself sensitive to baseline accuracy and should not be used to rank individual models against one another [s1]. What it supports is the weaker, sturdier claim: image-present items impose a difficulty on models that is specific to them, over and above the difficulty those items also pose to human candidates [s1].

Model-hard is not student-hard

The other useful result is about what "difficult" means. Item-level human difficulty and model difficulty correlated only moderately — Spearman's rho 0.318 (95% CI 0.286–0.346) on M1 and 0.333 (95% CI 0.302–0.364) on M2 [s1]. The questions that trip students up are, to a large extent, not the questions that trip models up.

That matters for anyone planning to use these systems as study aids. A model tuned to answer what a model finds hard is not thereby tuned to help with what a student finds hard.

How much weight the numbers carry

This is a benchmark study, not a study of clinical performance or of education outcomes. Nothing here shows that a model used in teaching improves a student's result, and multiple-choice licensing items are a narrow proxy for clinical reasoning.

There is also a broader reporting problem in this literature. A cross-sectional audit published on August 4 assessed 102 large-language-model studies in high-ranking radiology and medical imaging journals against the MI-CLEAR-LLM reporting standard [s2]. Mean adherence was 51.2% (SD 14.7%, range 22.2%–84.2%) [s2]. The least-reported items were the models' training-data cutoff date, at 9.8%, and the rationale for prompt wording, at 15.6% [s2]; prompt execution setup was reported 29.4% of the time and stochasticity management 33.1% [s2]. Those are precisely the details that determine whether a reported accuracy can be reproduced — or whether a benchmark was in the training data.

The German study's use of a restricted-access item bank is a partial answer to the contamination half of that problem. It does not answer the reproducibility half.

What to watch

Whether the text–image gap narrows in the next model generation, and whether anyone repeats this design on a second national exam with unpublished items. One restricted benchmark in one country and one language is a single data point, however many items it contains.

Sources

• Evaluating foundation models on official German medical licensing examinations: Implications for high-stakes assessment and AI-assisted medical education — npj Digital Medicine, 2026-08-31
• Reporting Quality of Large Language Model Studies: A Cross-Sectional Audit of High-Ranking Radiology and Medical Imaging Journals — Korean Journal of Radiology, 2026-08-04