🔍 Read the full analysis: The Science Paper Behind Microsoft’s AI Doctor Comparison on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Science paper published Oct. 22 reports that Microsoft’s MAI-DxO system achieved 83.2% diagnostic accuracy on 304 complex NEJM cases, compared with 36.1% for generalist physicians using reference materials. The study tested text-based historical cases in a simulator, not patient care, and Microsoft has not reported an independent replication.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 complex medical cases in a study published Oct. 22 in Science, compared with 36.1% for generalist physicians using textbooks and the internet, according to the paper. The peer-reviewed results concern historical, text-based cases in a simulator, not diagnoses made for real patients, so they do not establish that the system improves clinical care.
The study, titled “Sequential diagnosis with language models,” evaluated a benchmark designed to mimic parts of diagnostic reasoning. A case begins with a short summary; the system must request further information, such as answers to questions or test results, from a model holding the case file. The benchmark scores the final diagnosis and estimates the cost of the tests requested. Microsoft calls the benchmark the Sequential Diagnosis Benchmark.
Microsoft’s reported results include a second set of 325 emergency department cases, on which MAI-DxO scored 80.3% and physicians scored 46.8%. For the 304 complex cases, the physician score was 22.3% without reference materials and 36.1% with textbooks and internet access. These are study results for the specific cases and test conditions; they are not measured rates of accuracy in everyday medical practice.
The system combines a language model with an orchestration process in which virtual agents propose diagnoses, choose follow-up questions or tests, and critique one another. Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex cases, before the MAI-DxO system’s orchestration layer was added. That comparison suggests much of the measured performance came from the underlying model, while the added process contributed a smaller increase. The paper also estimates testing costs, but those figures rely on assigned prices rather than actual hospital spending.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Say About Diagnosis
The study matters because it tests more than whether an AI can select a diagnosis when all the facts are already provided. In the benchmark, the system has to decide what information to seek and when to stop, choices that form part of real diagnostic work. The authors also track estimated test costs, an effort to examine efficiency alongside accuracy.
But the comparison does not show that MAI-DxO is ready to replace doctors or improve patient outcomes. The cases were historical and selected for complexity, and participants interacted with a text-based simulator rather than patients. The physician group consisted of generalists working individually, while the AI system used multiple agents. As TechCrunch noted in its account of the paper, difficult cases may be referred to specialists or handled by teams in practice; those groups were not part of the reported comparison.
The findings could inform later studies and product development, but decisions by hospitals, insurers and patients require evidence about performance in clinical settings. That includes whether the system can handle ordinary cases, avoid unnecessary tests, recognize uncertainty and work safely with clinicians. The present scores do not answer those questions.
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
The Science article follows an earlier, non-peer-reviewed version posted on arXiv in June 2025. That preprint reported a different model and a larger gap between AI and doctors. The later paper appeared in Science, volume 394, pages 423–428, on Oct. 22, 2026. Its 15 listed authors are affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman.
The source material’s account of the full paper’s methods draws on the preprint, which it says may differ from the final version. In that earlier account, a model judged answers against the published diagnosis, while physicians re-graded a sample and showed good, but imperfect, agreement with the automated scoring. If a requested test result was absent from the case file, a gatekeeper model generated a synthetic result. Estimated test prices came from a 2023 U.S. hospital price list mapped to billing codes, plus a fixed $300 physician visit charge; the authors described those prices as approximations.
The physician comparison also has limits. The preprint described 21 U.S. and UK generalists—17 in primary care and four hospital generalists—with a median of 12 years’ experience. They worked alone and were not allowed to use search engines or language models in the closed-book condition. According to the source account, the doctors were scored on 56 held-out cases, while the AI was evaluated on all 304. The published Science figures include an additional open-book physician result.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
Limits the Trial Leaves Open
The published scores do not show how MAI-DxO would perform with live patients, physical examinations or imaging. The source material says the benchmark’s NEJM cases had no healthy patients or harmless explanations, limiting what the test could reveal about false alarms. It is also unclear from the supplied reporting how the final Science paper handled every methodological detail that appeared in the earlier preprint.
Microsoft’s results have been peer-reviewed, but the source material reports no independent replication. Peer review evaluates the paper’s methods and reporting; it does not mean outside researchers have reproduced its findings. The study also does not establish whether the system’s accuracy, test ordering or costs would hold across routine cases, different hospitals or patient populations.
A Planned Study With Patients
The next development identified in the source material is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their usual care. That setting could provide evidence closer to clinical use than a retrospective case benchmark, although the supplied material does not give a start date, sample size, study design or publication timeline.
Until those details and results are available, the Science paper should be read as evidence about performance on defined simulations—not proof of benefit in hospitals. Key questions for further testing include how the system performs against specialist teams, whether it changes care decisions, how often it prompts unnecessary testing, and whether its estimates of diagnostic accuracy and cost hold in real-world practice.
Key Questions
What did Microsoft’s MAI-DxO study report?
The paper reported 83.2% diagnostic accuracy for MAI-DxO on 304 complex cases, versus 36.1% for generalist physicians with reference materials. Those figures apply to the study’s simulated cases and conditions.
Was MAI-DxO tested on real patients?
No. The reported benchmark used historical, text-based medical cases in a simulator. The source material says a prospective study involving consenting patients is planned at Beth Israel Deaconess Medical Center.
Were the doctors in the comparison specialists?
The physician group described in the source material consisted of primary care doctors and hospital generalists working alone. Specialists and clinical teams were not included in the comparison described there.
Does peer review mean the results were independently confirmed?
No. Publication in Science means the paper underwent peer review, but the supplied reporting does not identify an independent replication of Microsoft’s findings.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
