AI And Doctors In Microsoft’s Paper: What Did Researchers Measure?

🔍 Read the full analysis: AI And Doctors In Microsoft’s Paper: What Did Researchers Measure? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI system scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a paper published October 22 in Science. The peer-reviewed study measured diagnosis and estimated testing costs in a text-based simulator, not performance treating real patients, so its results do not establish how the system would perform in routine clinical care.

The paper, “Sequential diagnosis with language models,” reports results on two sets of cases. For the 304 complex cases drawn from the New England Journal of Medicine, MAI-DxO reached 83.2% diagnostic accuracy. Generalist physicians scored 36.1% when they could consult textbooks and the internet, and 22.3% without those resources. On 325 emergency department cases, Microsoft reported scores of 80.3% for the system and 46.8% for physicians.

The benchmark tested a process, not just a final answer. Each case began with a short summary; the model or physician could request additional information, such as a test result, from a simulated case file. Requests carried estimated costs. The study therefore evaluated both the final diagnosis and the path taken to reach it. MAI-DxO organizes a language model as a panel of virtual physicians that propose diagnoses, select questions or tests, and challenge one another over several rounds.

Microsoft’s results also show that the underlying model accounted for much of the performance. The paper’s reported chart data put GPT-5 alone at 78.9% on the complex cases, compared with 83.2% for MAI-DxO. The source article reports an estimated testing cost of $8,409 per case for GPT-5 alone; Microsoft’s orchestration approach added about four percentage points in accuracy and reduced estimated test costs. Those cost figures are estimates, not observed hospital bills.

What the Scores Say About Clinical Use

The findings matter because diagnostic AI is often evaluated on questions where all relevant facts are already provided. This benchmark instead examined whether a system could decide what information to seek next, a decision that is part of real diagnostic work. It also attached estimated prices to tests, making the paper’s approach relevant to questions about both accuracy and resource use.

But the results are not evidence that MAI-DxO is ready to diagnose patients independently. The cases were historical and text-based, with no live patient interaction, physical examination, or imaging. The comparison was between a multi-agent AI setup and individual primary care doctors or hospital generalists; it did not test specialist teams or the collaborative process used in many difficult cases. Readers and health systems should treat the reported scores as benchmark performance, not as a direct measure of improved patient outcomes.

How Microsoft Built the Benchmark

The study appeared in Science, volume 394, pages 423–428, on October 22, 2026. The paper lists 15 authors, all affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. A version of the work appeared on arXiv in June 2025 as a preprint, before peer review, with different headline figures and a different model. The peer-reviewed paper is the basis for the updated numbers reported here.

Details about the benchmark and physician comparison in the supplied source are drawn in part from that earlier preprint, because the final Science article is behind a paywall; those details may differ from the published version. In the preprint, 21 generalists from the United States and United Kingdom took part: 17 primary care doctors and four hospital generalists, with a median of 12 years of experience. They were assessed on 56 held-out cases, while the AI was assessed on all 304. Microsoft’s published chart data lists physician averages across 764 case attempts without references and 662 with references.

The scoring and cost methods also carry limits. A model judged diagnoses against the published final diagnosis on a five-point scale; physician re-grading of a sample showed good, but imperfect, agreement. When a requested result was absent from a case file, a gatekeeper model generated a synthetic result. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, plus a fixed $300 per physician visit. The authors described these prices as first-order approximations.

“More than twice as accurate as physicians using textbooks and the internet on complex cases.”

— Microsoft, in its blog post “Building Medical Superintelligence”

What the Benchmark Cannot Establish

The study does not establish whether MAI-DxO improves health outcomes, reduces delays, or lowers total costs when used with real patients. It also does not show how the system performs with physical examinations, imaging, live records, or specialist teams. The cases were selected for complexity, and the source says the NEJM set contained no healthy patients or harmless explanations, limiting the ability to assess false alarms.

There are other qualifications. The benchmark used a model to grade diagnoses, and some requested results were generated synthetically when they were missing from the records. Cost estimates depended on a particular U.S. price list and assumptions about visit costs. The source notes that the final paper’s full text was not available to its author, so some methodological details drawn from the preprint may not match the peer-reviewed version. Peer review checks a paper’s methods and reporting; it does not amount to independent replication. No outside reproduction of the reported results is described in the supplied material.

A Study Alongside Patient Care

Microsoft’s next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source, consenting patients will interact with the system alongside their usual care. That setting could provide evidence on how the system works with actual patients, but the supplied material does not give a start date, enrollment target, study design, or planned outcome measures.

Until those details and results are available, the main confirmed finding remains a strong score on simulated case records under the paper’s benchmark conditions. Any assessment of clinical value will depend on what the prospective study measures, how clinicians use the system, and whether independent research can reproduce the results.

Key Questions

What did Microsoft’s MAI-DxO study measure?

It measured diagnostic accuracy on complex and emergency department case records, along with estimated costs for tests requested during the diagnostic process. The cases were handled in a text-based simulator, not during treatment of real patients.

How did MAI-DxO score on the complex cases?

The paper reported 83.2% accuracy on 304 complex NEJM cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources.

Were doctors compared with the AI specialists?

No specialist group or physician team is described in the supplied source. The comparison was with individual generalists, including primary care doctors and hospital generalists, working on challenging cases.

Does the study show MAI-DxO is safe for patient care?

No. The benchmark did not test live care or patient outcomes. It does not establish how the system would perform with examinations, imaging, real-time records, or the risk of false alarms in a broader patient population.

What happens next?

The source reports a planned prospective study at Beth Israel Deaconess Medical Center, where consenting patients will use the system alongside normal care. The study’s timing and detailed measures were not provided.

Source: ThorstenMeyerAI.com