Key Takeaways
A Nature Medicine study stress-tested frontier multimodal models, including GPT-5, Claude 3.5, and Gemini 2.5 Pro, on medical reasoning.
The models aced ordinary benchmarks but leaned on shortcuts, fabricated reasoning, and faltered when key image details were removed or modalities swapped.
The authors tie the gap to test design rather than model size, and released perturbation tools and rubrics for others to reuse.
Leading AI models that ace medical benchmarks fall apart under small, deliberate changes to their inputs, according to an adversarial stress test published in Nature Medicine on June 26.
Researchers led by Hoifung Poon and Yu Gu put frontier multimodal models, including GPT-5, Claude 3.5, and Gemini 2.5 Pro, through perturbations designed to probe how they actually reason. Scripps Research cardiologist Eric Topol, who flagged the paper, said the models are not ready.
The systems often reached the right answer for the wrong reasons. They could guess correctly even with key inputs removed, then get confused by slight prompt changes while producing convincing but flawed reasoning traces. Removing image details or swapping modalities exposed the brittleness.
The authors tie the weakness to how models are tested, not to model size, and argue benchmark wins do not equal clinical readiness. They released their perturbation tools and rubrics so others can run the same checks on robustness and sound reasoning.
Not everyone agrees the test is fair. Max Hodak, the Science Corporation founder, argued the study leans on older models and draws too broad a conclusion about AI in medicine. The published record carries no model-specific scores or vendor responses, so the debate over which systems fail, and by how much, stays open.
The most notable part of this is the gap itself. A model can top a leaderboard and still reason like it is guessing, which matters far more in a clinic than in a benchmark. It is the same tension running through Midjourney's claim that its AI scanner can rival the MRI and through the money pouring into AI chronic care for older adults. The tools are improving fast, and so is the case for testing them harder before they reach patients. Worth watching whether vendors respond.
People Also Ask
Which AI models did the Nature Medicine study test?
It stress-tested frontier multimodal models including GPT-5, Claude 3.5, and Gemini 2.5 Pro on multimodal medical reasoning.
Why did the AI models fail the clinical stress test?
They leaned on shortcuts and reached correct answers for the wrong reasons, faltering when key image details were removed or prompts slightly changed.
Does this mean AI is not safe for medicine?
The study shows benchmark scores overstate readiness, though critics note it leans on older models, so the debate over clinical use remains open.
What did the researchers release?
The authors published perturbation tools and rubrics so others can test models for robustness, sound reasoning, and real medical demands.
Sources
Nature Medicine, Eric Topol