AI modelled on a panel of doctors beats generalist physicians in a diagnosis benchmark

An artificial intelligence system built on frontier language models, the most capable of the AI programs trained on vast quantities of text, reached the correct diagnosis in more than four out of five cases in a new interactive test of clinical reasoning. The study, by researchers at Microsoft, is published in Science. Generalist physicians who took the same test were right far less often. The design of the test is as interesting as the scores. It tries to mirror how diagnosis unfolds in practice. A diagnosis is not a single answer to a complete case file but a sequence of decisions about what to find out next, and the cost of each step is counted.

The authors call this format sequential diagnosis. A diagnostician begins with only a brief clinical presentation, a short description of the patient and the problem that brought them in. They then choose which further information to acquire before committing to a diagnosis. Because every request is a choice, the format allows two things to be measured at once. One is whether the final diagnosis is correct. The other is how much the workup, the trail of enquiries and tests along the way, would cost. A test that hands over every detail at the outset can check only the first.

The team built an interactive benchmark around this idea. A benchmark here means a standardised set of cases on which both people and AI systems can be scored. Alongside it, the team built an AI system designed to tackle it. The system does not rely on a single model producing a single answer. Instead it simulates a panel of physicians. As new information comes in, it continually revises its differential diagnosis, the working list of conditions that could explain the patient's symptoms. It also selects tests with cost-effectiveness in mind.

The benchmark draws on two sources. The first is the clinicopathological conference cases of the New England Journal of Medicine. These are published case discussions in which a puzzling presentation is worked through to a final diagnosis. The second is MIMIC-IV-ED, a research database of emergency department admissions. On the first source, the system's accuracy was 83.2%. On the second, it was 80.3%. Generalist physicians scored 36.1% and 46.8% respectively. The two sources differ in kind, yet the system's accuracy was similar on both. The abstract reports figures for accuracy only. It gives no numbers for the cost of the workups, although the authors say the findings point towards diagnoses that are cheaper as well as more accurate.

The authors are explicit that these results were obtained in a controlled benchmark setting. A benchmark can capture something of the structure of clinical reasoning. The study does not, however, test how such a system would perform in day-to-day clinical care. The authors frame the promise of language models in terms of supporting diagnosis. How that support would work in practice remains to be shown.