AIHealthcare Analytics

Overview

Research evaluating LLM-generated responses about brain aneurysm examines performance across multiple quality domains and compares how patients (community participants) and physicians perceive and rate these responses relative to expert-generated content.

Performance Findings

LLMs demonstrated variable performance across key quality domains — a finding described as consistent with prior research on LLM medical information quality. ChatGPT showed apparent advantages in some domains, though physician reviewers highlighted the continued need for information oversight even for the best-performing models.

Divergent Perception: Patients vs. Physicians

A striking finding is the systematic divergence in perception between lay and clinical evaluators. Community participants (patients) consistently rated LLM responses as better than physician-generated responses. Physicians, by contrast, rated LLM responses as similar to or somewhat worse than what they themselves would have provided. This divergence has significant implications for patient trust calibration and the risk of over-reliance on AI-generated health information.

Implications for Health Communication

The patient-physician perception gap suggests that lay users may be systematically overestimating LLM quality in medical communication contexts — potentially due to factors such as language accessibility, apparent comprehensiveness, or empathetic tone rather than clinical accuracy. Physician oversight of LLM-generated patient education content is reinforced as a necessary safeguard, particularly for high-stakes neurovascular conditions where misinformation carries serious clinical risk.