Harvard study finds AI models can match or exceed internal medicine doctors in ER diagnosis accuracy

Mon, 04 May 2026 09:38:00 +0200

A recent Harvard Medical School study evaluated how OpenAI's large language models perform in medical diagnostics, specifically in emergency room settings. The study compared the performance of two internal medicine attending physicians to two AI models, o1 and 4o, using 76 patient cases from Beth Israel Deaconess Medical Center. Independent physicians assessed the accuracy of the diagnoses without knowing their source. The findings showed that the o1 model achieved the exact or nearly correct diagnosis in 67% of initial ER triage cases, outperforming the physicians who scored 55% and 50%. The study emphasized that the AI was provided the same text-based patient information as the doctors, without pre-processing. Researchers caution that while AI shows potential, it is not yet ready to replace human decision-making in life-or-death ER situations. The study calls for further prospective trials in real-world clinical environments. Experts also highlighted that comparing AI to internal me