Explainable AI in healthcare may be creating new risks instead of trust

Healthcare is a complex socio-technical system, not a purely technical environment. Clinical decisions are shaped not only by algorithmic outputs, but also by time pressure, professional accountability, ethical responsibility, patient autonomy, and organizational constraints. Explainability must operate within this system, rather than being imposed as a standalone technical requirement.

Explainability is widely promoted as the solution to trust and safety concerns in medical artificial intelligence, but a new academic review finds that this assumption does not hold across healthcare settings. An in-depth analysis of deep learning systems used in medicine shows that explainability can improve clinical decision-making in some contexts while introducing new risks in others, particularly when applied without regard to clinical urgency, task type, or institutional responsibility.

Published in the journal Algorithms, the study titled “Explainability in Deep Learning in Healthcare and Medicine: Panacea or Pandora’s Box? A Systemic View” examines explainable AI through a systems-based framework. The paper concludes that explainability is not a universal safeguard and must be calibrated to specific medical use cases rather than imposed as a blanket requirement.

Why universal explainability fails in healthcare

Healthcare is a complex socio-technical system, not a purely technical environment. Clinical decisions are shaped not only by algorithmic outputs, but also by time pressure, professional accountability, ethical responsibility, patient autonomy, and organizational constraints. Explainability must operate within this system, rather than being imposed as a standalone technical requirement.

Using general systems theory, the author explains that explainability is an emergent property of the entire healthcare ecosystem. It arises from the interaction between three layers: the technical layer of algorithms and data, the cognitive layer of human interpretation and reasoning, and the organizational layer of governance, accountability, and regulation. Optimizing explainability at only the technical level does not guarantee better outcomes.

The paper shows that many current explainability approaches focus narrowly on technical transparency. Methods such as saliency maps, attention visualizations, LIME, and SHAP are designed to reveal which features influence model predictions. While these methods can provide insight into model behavior, they often fail to align with how clinicians actually reason about patients and diseases.

Clinical reasoning is contextual, causal, and narrative-driven. It relies on understanding disease mechanisms, uncertainty, and trade-offs. Feature attribution scores or heatmaps do not automatically translate into clinically meaningful explanations. As a result, explanations may appear informative while offering little support for real decision-making.

The study highlights evidence showing that explanations can increase clinician confidence without improving diagnostic accuracy. In such cases, explainability creates an illusion of understanding rather than genuine insight. This phenomenon undermines the assumption that transparency automatically improves safety and trust.

When explainability helps and when it harms

The author draws a clear line between clinical reasoning tasks and clinical function tasks, arguing that explainability requirements should differ accordingly.

Clinical reasoning tasks include diagnosis, treatment selection, prognosis, and risk assessment. These are high-stakes decisions where clinicians remain legally and ethically responsible. In these contexts, explainability can play a constructive role. It can help clinicians evaluate AI recommendations, detect errors, calibrate trust, and support informed consent with patients. Here, explainability functions as a stabilizing force within shared decision-making.

On the other hand, clinical function tasks include operational and logistical processes such as appointment scheduling, inventory management, image preprocessing, and system optimization. These tasks do not require clinician judgment at the individual patient level. The paper argues that imposing detailed explainability in these contexts offers little benefit and may even be counterproductive. Performance validation and monitoring are more appropriate forms of accountability.

The risks become most acute in time-critical medical situations. In emergencies such as cardiac arrest or acute deterioration, clinicians must act immediately. The paper warns that requiring explanations in these moments can delay intervention, increase cognitive load, and compromise patient safety. In such contexts, explainability shifts from being a safeguard to becoming a hazard.

Explainability should never be treated as temporally neutral. Timing matters. Explanations that are useful in reflective settings can be dangerous in urgent ones. This insight directly challenges regulatory and institutional approaches that mandate explainability uniformly across all medical AI uses.

Limits of current explainable AI methods

While acknowledging their technical sophistication, the author identifies consistent gaps between what these methods provide and what healthcare systems actually need.

Saliency maps and attention mechanisms are shown to be visually compelling but unstable and easily misinterpreted. They may highlight areas of an image or features in data without conveying why those features matter clinically. LIME offers flexibility but suffers from inconsistency and lack of reproducibility, making it unreliable for high-stakes decisions. SHAP provides mathematical rigor and consistency, but its numerical outputs often lack clinical meaning and fail to support causal reasoning.

Across methods, the paper identifies a recurring problem: explainability tools often serve developers and auditors better than clinicians and patients. They prioritize interpretability at the algorithmic level while neglecting the cognitive and organizational dimensions of healthcare.

The author also warns against “explanation theater,” where systems appear transparent without improving accountability or outcomes. Explanations can be used to satisfy regulatory demands or ethical expectations while masking deeper issues such as biased data, distribution shift, or inadequate governance.

Importantly, the paper stresses that explainability does not replace other safety mechanisms. Transparent systems can still be unsafe. Trustworthy medical AI requires performance validation, bias monitoring, institutional oversight, and clear responsibility structures in addition to explainability.