GPT-4o Improves Doctors' Clinical Reasoning by Up to 18%, But Human Judgment Still Leads

A new ILO-led international study finds that AI tools like GPT-4o can significantly improve physicians' clinical reasoning by up to 18% but work best as decision-support systems alongside human expertise, not as replacements for doctors. The report urges governments, development partners, and healthcare providers to combine AI adoption with strong regulation, workforce training, local validation, and digital infrastructure to safely strengthen healthcare systems worldwide.

Artificial intelligence may soon become one of healthcare's most valuable assistants, not by replacing doctors, but by helping them make better clinical decisions. A new International Labour Organization (ILO) Working Paper, prepared by researchers from Maastricht University, Universitas Indonesia, Aga Khan University Hospital in Nairobi, and the ILO, provides strong international evidence that general-purpose large language models (LLMs) such as GPT-4o can significantly improve physicians' clinical reasoning. Based on a randomized controlled trial involving 249 physicians across Indonesia, Kenya, and the Netherlands, the study concludes that AI has the potential to improve healthcare quality, particularly in countries facing shortages of skilled medical professionals. However, it also warns that AI's success will depend on responsible implementation, proper training, and strong governance rather than technology alone.

AI Boosts Doctors' Performance Across Different Healthcare Systems

The study comes at a time when the World Health Organization estimates a global shortage of around 11 million healthcare workers by 2030, with the greatest shortages expected in low- and lower-middle-income countries. Many healthcare systems already struggle with limited specialist availability, rising patient numbers, and constrained diagnostic resources.

Researchers tested whether GPT-4o could strengthen physicians' clinical reasoning by dividing participants into groups with and without AI assistance while they solved standardized patient case studies covering cardiovascular, respiratory, musculoskeletal, and infectious diseases.

The results were clear. Physicians using AI improved their clinical reasoning scores by 18% in Kenya, 10.7% in Indonesia, and 7.2% in the Netherlands, with all improvements being statistically significant. Kenya recorded the largest gains, suggesting that AI-assisted decision support may provide the greatest benefits in healthcare systems with fewer specialists and greater resource constraints.

Interestingly, GPT-4o itself outperformed the best physicians who worked without AI support. However, the highest-performing doctors who combined their own expertise with AI achieved even better results than the model alone, reinforcing that AI performs best as a clinical assistant rather than an independent decision-maker.

AI Can Strengthen Healthcare Without Replacing Physicians

One of the report's strongest messages is that fears of doctors being replaced by AI are largely unfounded. According to the ILO's occupational analysis, medicine remains a profession with relatively low automation risk because many essential tasks, including physical examinations, patient communication, ethical judgment, surgery, and treatment decisions, still require human expertise.

Instead, AI is most valuable as a productivity tool. Physicians reported that GPT-4o helped them generate broader differential diagnoses, suggested additional questions during patient history-taking, and supported decision-making in complex cases.

The research also found that AI delivered greater benefits to less-specialized physicians than experienced specialists. In Kenya, non-specialists experienced estimated performance gains of 20.4 percentage points, compared with 10.7 percentage points for internal medicine specialists. This suggests AI could reduce skill gaps, strengthen frontline healthcare delivery, and improve diagnostic quality in underserved areas without replacing healthcare professionals.

For governments seeking to expand universal healthcare while facing workforce shortages, this makes AI a tool for increasing productivity rather than reducing employment.

Governments and Development Partners Must Build the Right Ecosystem

The report stresses that providing AI access alone is not enough. Although average performance improved, some physicians using AI still performed worse than colleagues without it, highlighting risks such as automation bias, over-reliance, and reduced critical thinking.

To avoid these risks, governments are encouraged to establish clear regulatory frameworks before deploying AI widely. Recommended measures include structured clinical workflows, mandatory human oversight, AI literacy training for healthcare workers, continuous monitoring of clinical outcomes, validation using local medical data, transparent liability rules, and regular evaluation of AI performance.

International development partners also have a major role to play. Rather than focusing only on purchasing AI software, organizations such as multilateral development banks and global health agencies can support digital infrastructure, broadband connectivity, secure health information systems, workforce training, regulatory capacity, and regional AI evaluation centers.

The report also highlights the importance of digital sovereignty. Many developing countries currently depend on foreign AI platforms and proprietary technologies. Supporting locally adaptable AI models and national regulatory capacity could reduce this dependence while ensuring AI reflects local languages, treatment guidelines, and healthcare realities.

Private Sector Has Major Opportunities, But Safety Must Come First

The findings point to growing opportunities for technology companies, healthcare providers, medical software developers, cloud service providers, and digital health startups. Demand is expected to increase for AI-powered clinical decision-support systems, electronic health records, medical documentation tools, physician training platforms, and secure healthcare cloud infrastructure.

However, commercial success will increasingly depend on meeting stricter regulatory expectations. The report recommends independent clinical validation, continuous safety monitoring, demographic bias testing, transparent documentation of training data, and clear accountability before AI tools are widely adopted in hospitals.

Looking ahead, researchers recommend expanding trials into real clinical environments to measure patient outcomes rather than simulated case performance. Governments should also invest in physician training, stronger digital infrastructure, updated legal frameworks, and continuous monitoring of AI systems. Equally important is involving healthcare professionals in AI governance through ongoing social dialogue to ensure technology supports rather than undermines clinical practice.

The report concludes that AI should not be viewed as a substitute for physicians but as a powerful decision-support technology capable of improving healthcare quality, reducing diagnostic disparities, and strengthening overstretched health systems. Countries that combine sound regulation, workforce development, local validation, and responsible AI deployment will be better positioned to address healthcare shortages while maintaining patient safety and public trust.