Explainable AI gains ground as key to safe clinical decision support

The study revealed substantial differences between the perspectives of clinicians and regulators regarding AI adoption. Regulators emphasized the importance of global explanations, such as feature importance rankings, to assess model safety, fairness, and transparency. In contrast, clinicians prioritized understanding how AI systems integrated clinical knowledge and specific patient data to produce decisions.

The growing presence of artificial intelligence in clinical settings is raising concerns about trust, transparency, and safety in medical decision-making. A new study titled “Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations”, submitted on arXiv, explores how explainable AI (XAI) can help bridge the gap between algorithmic predictions and clinical judgment. The study was led by a team of data scientists, clinicians, and UK medical regulators including experts from Brunel University London, the Medicines and Healthcare Products Regulatory Agency (MHRA), and NHS England.

While AI systems, especially black-box models like artificial neural networks, can deliver high diagnostic accuracy, their lack of transparency undermines clinical trust. The study evaluates several AI and XAI models through multi-stakeholder workshops and a pilot study using synthetic cardiovascular patient data. The focus was on assessing how clinicians and regulators interact with global, local, and counterfactual explanations generated by models including logistic regression, random forest, and neural networks. Their goal was to ensure that these models are not only performant but also interpretable and clinically safe.

What insights emerged from clinical and regulatory engagement?

The study revealed substantial differences between the perspectives of clinicians and regulators regarding AI adoption. Regulators emphasized the importance of global explanations, such as feature importance rankings, to assess model safety, fairness, and transparency. In contrast, clinicians prioritized understanding how AI systems integrated clinical knowledge and specific patient data to produce decisions. While clinicians found local explanations helpful when aligned with clinical reasoning, they expressed concern over the interpretability of visualizations produced by common XAI methods such as LIME.

Clinicians engaged with real-time diagnostic tasks involving synthetic patient profiles, initially making their own assessments before reviewing the AI model’s predictions and explanations. In several instances, their diagnostic accuracy improved after reviewing XAI outputs. However, the study also uncovered signs of automation bias: in one critical case, clinicians altered their initial correct diagnosis to match an incorrect AI prediction based on a misleading confidence score. This raised important concerns about trust calibration, how clinicians adjust their reliance on AI based on confidence levels, even when the model is wrong.

The inclusion of counterfactual explanations also sparked debate. While clinicians found simplified versions of these outputs more interpretable, there were concerns about recommendations involving immutable characteristics such as age or chronic medical history. Regulators stressed that in clinical settings, counterfactual explanations must be grounded in feasibility and clinical logic, not just mathematical optimization.

How should AI tools be safely introduced into clinical settings?

The study proposes a set of clear recommendations for the responsible integration of AI/XAI systems into clinical workflows. First, simple yet effective models should be prioritized when they offer performance comparable to black-box algorithms. Random forest models, for example, provided strong accuracy in heart attack risk prediction while retaining a degree of interpretability. Logistic regression models, while less accurate, offered clarity through odds ratios, enabling clinicians to understand how specific features like age or angina history influenced the model’s output.

Second, training and professional development are essential. Clinicians must be equipped to interpret AI-generated explanations and understand the limitations of the data and algorithms. Education is particularly important for mitigating automation bias and ensuring that AI systems support rather than override human judgment. The study also recommends involving cognitive specialists in designing XAI outputs tailored to how clinicians naturally process information.

Third, XAI should not be a one-size-fits-all solution. Local explanations must be adapted to clinical constraints, ensuring they are actionable within fast-paced environments. For example, while LIME was useful in some scenarios, clinicians reported that its visual outputs could be too time-consuming to interpret under pressure. Similarly, counterfactual explanations must highlight realistic clinical actions rather than theoretical adjustments.

Regulators are encouraged to focus on model performance as well as transparency, fairness, and subgroup evaluation. This includes identifying patient subgroups where models may perform poorly and ensuring documentation of such limitations. Transparency requirements, the study concludes, vary by stakeholder group: regulators need evidence of model reliability and fairness, while clinicians require clarity on clinical relevance and applicability.