Microsoft AI tool tested against doctors in medical cases

Microsoft says its Diagnostic Orchestrator surpassed generalist physicians in diagnostic tests built from prior medical cases. The system acts like a group of specialists: multiple AI agents review symptoms and patient histories, select tests, consider possible explanations and settle on a diagnosis. OpenAI’s GPT-5 powers the tool, which was evaluated using Microsoft’s SDBench benchmark. The company intends to make that benchmark open-source.

Across hundreds of published case reports, the system reached about 83 percent accuracy. Generalist doctors scored roughly 36 percent when references were available and 22 percent without them. On emergency department cases from a large admissions database at Boston’s Beth Israel Deaconess Medical Center, the AI reached 80 percent accuracy, while physicians averaged 47 percent. Its diagnoses also generally carried a lower estimated cost. Researchers said the coordinated system performed better than individual frontier AI models, although the peer-reviewed study did not cover several newer models.

The results come from historical cases, so Microsoft says clinical validation is still required before broad use. Hospital testing with Beth Israel Deaconess Medical Center is expected to begin within the next couple of months, though Microsoft has not set a firm schedule or explained when the technology might be sold. For Spokane patients and families, any future use would depend on real-world performance and compliance with medicine’s regulatory requirements. Mustafa Suleyman and Dr. Dominic King say clinicians should remain central to patient care, with AI potentially reducing routine workload and allowing doctors more time with patients.