Microsoft claims its AI beats doctors on tough cases, but experts ask if the math holds up

Microsoft is positioning its latest AI diagnostic tool as a step toward "medical superintelligence," claiming it can handle complex medical cases with significantly greater accuracy than human generalists. However, the path to this claim is fraught with caveats that suggest the technology may be more of a powerful calculator than a replacement for clinical judgment. The system, known as MAI-DxO, coordinates multiple AI agents to simulate a panel of doctors debating a diagnosis. In a simulation using difficult cases from the New England Journal of Medicine, the system was correct 83% of the time, whereas primary care doctors and general internists working individually were correct only 36% of the time. Yet, this comparison excludes specialists and physician teams, who typically handle such complex cases, raising questions about the baseline against which the AI is measured.

The push for AI-driven diagnostics has become a competitive race among major tech firms. By publishing these results, Microsoft joins Google and OpenAI in asserting that their models can match or outperform human expertise. However, the study’s own data indicates that much of the performance boost comes from using more powerful underlying models rather than the orchestration layer itself. Microsoft tested its system with models from OpenAI, Google, Anthropic, Meta, and DeepSeek. The highest scores were achieved using OpenAI’s GPT-5 and GPT-5.5, suggesting that the "panel" of agents is less important than the raw power of the individual model doing the heavy lifting.

Cost efficiency, a critical factor for widespread adoption, showed mixed results. When using GPT-5, the multi-agent system reduced the estimated cost per case by about $3,000 compared to running the model alone, but it only improved accuracy by four points. For emergency department cases, the improvement in either cost or accuracy was negligible. With the newer GPT-5.5 model, released in April 2026, the multi-agent approach did improve accuracy on complex cases from 72% to nearly 87% at a slightly lower cost. However, this advantage disappeared in emergency settings, where the system and the standalone model performed similarly at around 85% accuracy. The system proved most useful when paired with weaker models, such as Gemini 2.5 Flash, which saw its score jump from 46.4% to nearly 69% with the orchestration layer, though the gap between weak and strong models remained wider than the boost provided by the system.

Mustafa Suleyman, CEO of Microsoft AI, views this as foundational infrastructure for future breakthroughs. He stated that the company is collaborating with the Mayo Clinic to train a healthcare-specific foundation model, scheduled for release next year. Suleyman predicts that by 2027, the company will be close to a "coding style breakthrough" for medical records, allowing for highly accurate predictions of patient outcomes. He argues that progress in this field will accelerate rapidly once a "flywheel" of real-world usage and iteration begins, a pattern he says has already driven advances in coding and chat applications.

The next phase involves moving beyond simulations to real-world testing. Microsoft has partnered with Beth Israel Deaconess Medical Center in Boston to evaluate the system’s clinical reasoning across various hospital settings. In this study, patients will receive standard care while consenting to parallel evaluation by the AI. The goal is to compare the AI’s recommendations against those of human doctors in high-stress environments, such as a Friday evening emergency department. This approach mirrors a recent study by Google, which tested its own diagnostic AI in urgent care settings with physicians monitoring in real time.

Microsoft’s broader vision, outlined in a recent manifesto, is to move beyond simple benchmarking tasks like answering exam questions. The company aims to integrate its tools directly into healthcare systems to improve overall outcomes. Suleyman, who previously co-founded DeepMind and led its health division, sees this focus as a potential differentiator for Microsoft among frontier AI labs. The company already sells dictation and radiology tools and views this diagnostic system as a natural expansion of that portfolio, with the ambition to hold the "number one" position in clinical decision support for the next few years.