Large Language Models vs. Human Coders—ChatGPT-4.0 Performance on NIH Grant Abstract Qualitative Analysis
COMPARISONA comparative study evaluated ChatGPT-4.0 against human coders in performing qualitative analysis tasks using NIH grant abstracts as a test case. The study focused on identification and description of research innovations in substance use research, using a sample of 125 NIH HEAL Initiative abstracts related to opioid overdose prevention. Study Design: - Parallel coding by human experts and ChatGPT-4.0 - Agreement analysis and thematic comparison - Evaluation of coding consistency, depth, and clinical/research relevance Key Findings: Large language models show promise for accelerating qualitative analysis workflows but remain underexplored relative to human coding performance in substance use and health research contexts. Implications: - LLMs may reduce time-to-insight in literature synthesis and grant portfolio analysis - Performance varies by task specificity and domain expertise requirements - Human-in-the-loop approaches (LLM-assisted coding reviewed by experts) may optimize efficiency without sacrificing quality - Validation required before LLM-only coding for high-stakes research decisions This evidence contributes to growing literature on LLM performance in research support roles, suggesting that while LLMs can augment human coding capacity, independent LLM coding requires domain-specific validation and quality assurance. ---
Subscribe to read full wiki analysis
Full wiki content — including source provenance, contradiction tracking, related pages, and analyst notes — is available to Subscriber members.
View Plans & SubscribeNew users get a 7-day trial — no credit card required.