AIHealthcare Analytics

Knowledge Wiki

Wiki › LLM Clinical Reasoning Benchmarks — Performance Across Models and Domains

LLM Clinical Reasoning Benchmarks — Performance Across Models and Domains

COMPARISON
Rev 2 Sep 8, 2026 04:08 UTC 4 sources

Overview Benchmarking of large language models across clinical reasoning domains has accelerated substantially in 2025–2026, with studies covering fetal CNS MRI interpretation, ophthalmology bioethics, pharmacist licensing examinations, and patient information quality. Collectively, these studies reveal both the advancing capabilities of frontier models and the methodological complexity of meaningful clinical benchmarking. Recent Benchmark Studies A comparative study of ChatGPT, Gemini, and DeepSeek on fetal CNS MRI text-based reasoning represents one of the first systematic evaluations in this subspecialty domain. A Japanese pharmacist licensing examination study found near-saturation accuracy in frontier models but introduced the critical methodological finding that refusal behavior must be separately characterized — overall accuracy, accuracy excluding refusals, refusal rate, and refusal consistency are all necessary metrics. GPT-5.1 benchmarked against a human ethicist in ophthalmology demonstrated higher perceived clinical utility despite lower readability scores. Methodological Advances The refusal-aware evaluation framework introduced by the pharmacy examination study is a significant methodological contribution: models that refuse questions may appear highly accurate on answered items while underperforming on overall task completion. Single-pass evaluations miss run-dependent variability in refusal behavior. These findings have broad implications for how benchmark…

Sources: harvested
🔒

Subscribe to read full wiki analysis

Full wiki content — including source provenance, contradiction tracking, related pages, and analyst notes — is available to Subscriber members.

View Plans & Subscribe

New users get a 7-day trial — no credit card required.

← Back to Wiki Index