AIHealthcare Analytics

Overview

Large language models including ChatGPT (GPT-5) and Gemini have achieved expert-level aggregate performance on formal medical licensing examinations, including the German M2 (Zweiter Abschnitt der Ärztlichen Prüfung). However, headline accuracy figures mask a critical structural vulnerability: performance degrades systematically as question difficulty increases.

The Difficulty-Gradient Problem

Research evaluating GPT-5 and Gemini on the German M2 examination found that accuracy correlates inversely with question difficulty. While overall pass rates may appear impressive, this difficulty-based failure pattern means that the most clinically consequential and nuanced questions — those requiring complex reasoning or rare knowledge — are precisely where current LLMs underperform. This finding is consistent with broader observations that aggregate benchmark scores can obscure domain-specific weaknesses.

2-Tiered Rescue Protocol

A "2-tiered rescue" system was developed to mitigate difficulty-based failures. In this architecture, a primary LLM attempts questions normally; items flagged as high-difficulty or uncertain trigger escalation to a secondary resolution mechanism (e.g., a second model, ensemble voting, or retrieval augmentation). This protocol achieved >95% accuracy on the German M2 examination, substantially outperforming either model operating alone. The approach offers a generalizable template for deploying LLMs in high-stakes knowledge assessment contexts.

Implications for Medical Education and Credentialing

The findings raise questions about how LLM performance should be interpreted and reported in medical education research. Institutions and researchers are cautioned against treating aggregate pass rates as sufficient evidence of clinical readiness. Difficulty-stratified evaluation and multi-model rescue architectures are recommended as components of any rigorous LLM deployment framework in medical examination or training contexts.