A multi-agent AI scored 100% on a gastroenterology benchmark. Specialists beat chatbots.
A small proof-of-concept study pitted several AI systems against gastroenterologists and emergency physicians on cholangitis exam questions. The gap between the best and worst AI performers was enormous.
A study published this month in BMC Medical Informatics and Decision Making offers a useful reminder that "AI" is not one thing with one level of medical competence — pitting several distinct AI architectures against each other, and against practicing physicians, on questions about a specific and serious condition: acute cholangitis, an infection of the bile ducts [s1].
What was tested
Researchers built what they describe as a genetic neuro-symbolic LLM system — a setup that orchestrates multiple AI agents using a framework called LangGraph, combining neural network-based language processing with symbolic, rule-based reasoning, specifically for cholangitis management [s1]. They tested it against three general-purpose AI chatbots — Claude 4.5 Sonnet, ChatGPT 5.2, and Gemini 2.0 Flash — as well as ten gastroenterology specialists and four emergency medicine physicians drawn from four tertiary medical centers in Turkey [s1]. All participants, human and AI, answered the same 30 case-based questions drawn from American Board of Internal Medicine gastroenterology subspecialty exam material, split evenly across diagnosis, treatment, and complications/prognosis [s1].
The results
The multi-agent neuro-symbolic system scored 100% (30 of 30) [s1]. Among the general-purpose chatbots, performance varied widely: Claude 4.5 Sonnet scored 90.0%, while ChatGPT 5.2 scored 60.0% and Gemini 2.0 Flash scored 63.3% [s1]. The human specialists performed strongly — gastroenterologists averaged 95.7% (± 3.2%) — while emergency medicine physicians, who don't specialize in this condition, averaged 84.2% (± 8.8%) [s1]. Among the humans, gastroenterologists significantly outperformed emergency physicians specifically on treatment-related questions (p = 0.012), and the gastroenterologists' overall performance was statistically non-inferior to Gemini 2.0 Flash's [s1].
Why the spread between AI systems matters
The headline comparison here isn't AI versus doctors — several individual chatbots actually scored below the human specialists, and one scored well below the emergency physicians too. The more informative finding is how much a single condition-specific engineering choice — combining multiple specialized AI agents with structured, symbolic reasoning rather than relying on one general-purpose model's free-form output — apparently mattered: the purpose-built system outperformed all three off-the-shelf chatbots, in some cases by a wide margin, on the same fixed set of questions [s1].
What this study is, and isn't
This is explicitly described by its own authors as a proof-of-concept study [s1] — a small, 30-question benchmark drawn from board-exam material, tested at four centers in one country, not a clinical trial measuring real patient outcomes. A perfect score on 30 exam-style questions demonstrates competence at answering exam-style questions; it does not demonstrate performance managing an actual patient with cholangitis through the messier, more ambiguous process of real clinical care, complete with incomplete information, evolving symptoms, and the kind of judgment calls exam questions are specifically designed to avoid requiring.
What the authors themselves caution
The study's authors are explicit that this remains a proof-of-concept suggesting multi-agent, neuro-symbolic architectures may offer a promising direction for AI-assisted clinical decision support in complex hepatobiliary conditions — not a finished product [s1]. They state directly that prospective clinical validation is required before broader implementation claims can be warranted [s1]. Readers should treat the 100% score as evidence of what a carefully engineered, condition-specific AI system can do on a structured exam benchmark, not as evidence that the system is ready to guide real cholangitis care.
Sources
- Performance comparison of a neuro-symbolic large language model system versus conventional AI models and human experts in cholangitis management — BMC Medical Informatics and Decision Making, 2026-06-01
Sources
- Performance comparison of a neuro-symbolic large language model system versus conventional AI models and human experts in cholangitis management — BMC Medical Informatics and Decision Making , June 1, 2026
A computer that grades each colonoscopy raised how often endoscopists found adenomas
A Danish stepped-wedge trial gave endoscopists automated feedback on their technique after every procedure. Adenoma detection rose from 43.4% to 48.6% — a different tool from real-time polyp AI.
When AI symptom-checkers disagree with patients, patients tend to walk away
In 6,772 users of a US health system's AI triage tool, people engaged about twice as often when the AI matched what they already planned — raising a hard question about what the tools are steering.
Most AI models that predict who will skip their medicines aren't ready for the clinic
A review of 41 studies found that the great majority of AI medication-adherence prediction models carried high risk of bias, and that fancier algorithms did not reliably predict better.
Clinical AI has 63 ways to measure fairness and one built for clinical use
A Lancet Digital Health scoping review found the field's fairness metrics fragmented and rarely clinically validated. A second review found that most studies don't measure fairness at all.