AI Chatbots vs Ophthalmology Specialists

TOPLINE:

A large language model (LLM) chatbot demonstrated proficiency in ophthalmology, matching or outperforming specialists in the management of glaucoma and retina cases.

METHODOLOGY:

Researchers asked 12 attending physicians and three senior trainees from the department of ophthalmology at Icahn School of Medicine at Mount Sinai, New York City, to respond to 20 ophthalmological questions and manage 20 deidentified patient cases.They prompted an LLM chatbot, GPT-4, to respond to the same questions and cases “as a clinician in an ophthalmology note format,” the researchers reported.The investigators evaluated the responses on a Likert scale for medical accuracy (10-point scale) and completeness (6-point scale).

TAKEAWAY:

The chatbot had higher average scores for accuracy (506.2 vs 403.4) and completeness (528.3 vs 398.7) than glaucoma specialists.Against retina specialists, the chatbot again ranked higher for accuracy (235.3 vs 216.1) and completeness (258.3 vs 208.7).Participants rated responses from the chatbot more favorably than those from their human counterparts.

IN PRACTICE:

“This work shows that an LLM chatbot can synthesize clinical data and report an impression and plan comparable with seasoned subspecialists,” the study authors wrote.

SOURCE:

The study was led by Andy S. Huang, MD, of Icahn School of Medicine at Mount Sinai, New York City. It was published online on February 22 in JAMA Ophthalmology.

LIMITATIONS:

The study relied on a relatively small number of participants from one center. “Our findings, while promising, should not be interpreted as endorsing direct clinical application due to chatbots’ unclear limitations in complex decision-making, alongside necessary ethical, regulatory, and validation considerations,” according to the researchers.