Opaque AI systems can be justifiable in urgent and serious medical cases

Accuracy and reliability are necessary conditions for the ethical use of black-box AI, but they are not enough on their own. The authors outline evidence showing that black-box models frequently outperform clinicians in tasks such as cancer screening, skin-lesion analysis, diabetic retinopathy classification, and ICU mortality prediction. With access to far larger datasets than any human could process, these models identify subtle patterns and statistical signals that clinicians may miss.

Regulators worldwide are tightening restrictions on black-box artificial intelligence (AI) in medicine, arguing that clinical decisions must be explainable to be ethical and safe. But new research warns that prohibiting opaque algorithms may deny patients faster diagnoses, earlier interventions, and more accurate predictions at moments when accuracy matters more than explanation.

The paper, "When Is Black-Box AI Justifiable to Use in Healthcare?" published in Big Data & Society, delivers a comprehensive ethical analysis of when clinicians can responsibly rely on AI systems that operate without causal transparency. Unlike recent debates that frame explainability as a strict requirement, the authors argue that ethical justifiability depends on a balance of factors: accuracy, urgency, seriousness of the condition, the nature of the decision, the presence of bias, and the degree of human involvement. Their findings challenge the growing trend in Australia, Europe, and other jurisdictions to regulate black-box AI out of clinical practice altogether.

Instead, the authors argue that black-box systems can be justifiable, and sometimes ethically required, when they deliver demonstrably more accurate and timely results than human clinicians, especially in serious or urgent cases. They call for a contextual approach that assesses each use case on its own merits rather than assuming that explainability must always take priority.

Accuracy as a necessary but not sufficient condition

Accuracy and reliability are necessary conditions for the ethical use of black-box AI, but they are not enough on their own. The authors outline evidence showing that black-box models frequently outperform clinicians in tasks such as cancer screening, skin-lesion analysis, diabetic retinopathy classification, and ICU mortality prediction. With access to far larger datasets than any human could process, these models identify subtle patterns and statistical signals that clinicians may miss.

The authors warn that dismissing such tools because they cannot provide causal explanations may sacrifice patient outcomes. In contexts where milliseconds matter or where faster detection can save lives, accuracy becomes an ethical imperative. The study cites scenarios such as predicting heart-attack risk, diagnosing life-threatening infections, or identifying early signs of chronic disease. In these situations, patients may value improved outcomes more than an explanation of the algorithm's inner workings.

Yet accuracy alone cannot justify every use of black-box AI. According to the authors, some decisions require transparency due to their ethical weight, even if accuracy is high. Decisions involving resource allocation, fairness judgments, or distribution of scarce treatments require clearer reasoning because they rely not only on medical data but on societal values. These include organ donation prioritization, transplant allocation, or any decision that determines life-or-death access to limited resources. In such cases, even a highly accurate algorithm may be inappropriate because the moral stakes require human accountability.

Clinical decisions grounded in empirical data, such as diagnosing pneumonia from imaging, are more compatible with black-box AI. But decisions grounded in values, such as determining which patient is more deserving of a scarce organ, require explanations to ensure fairness. This distinction becomes fundamental to determining justifiability.

The paper insists that accuracy is necessary in all cases but sufficient only when paired with additional contextual factors. Whether those factors justify using black-box AI depends on the nature of the decision and the impact on the patient.

Seriousness, urgency and the context of clinical decisions

The study analyses how seriousness and urgency influence ethical thresholds. The authors point out that the more serious and urgent a condition is, the stronger the justification for using highly accurate black-box AI. In emergency situations, clinicians already rely on rapid decision-making that accepts uncertainty, incomplete information, and limited patient consent. In these contexts, demanding a full explanation from an algorithm would slow care and potentially create preventable harm.

The authors highlight the ethical consistency between emergency practices and AI-assisted decisions. In trauma rooms, cardiac arrests, stroke units, or pandemic triage settings, the priority is stabilizing the patient with the most effective tools available. Waiting for an explanation from an AI system, or rejecting its use entirely, would be inconsistent with existing emergency ethics, where saving lives takes precedence over transparency.

Outside emergencies, seriousness continues to matter. Patients with chronic or historically neglected conditions may require different ethical considerations. The study uses cases such as endometriosis and chronic pain, where patients often endure long diagnostic delays and medical dismissal. For these patients, an explanation may hold deep personal significance, even if it comes at the cost of slight reductions in accuracy. Preferences vary, and ethical justifiability must respect those preferences when they relate to the patient's lived experience and long history of medical uncertainty.

The authors simultaneously warn that patients in remote or underserved regions may prefer access to any reliable clinical support over an ideal but unavailable human specialist. In rural areas facing the "rural mortality penalty," black-box AI may offer earlier diagnoses or better monitoring options that patients would otherwise lack. In these contexts, denying access to AI because of explainability concerns risks worsening inequality.

The context of the decision, therefore, becomes crucial. The same algorithm that is ethically acceptable in an emergency room may be inappropriate in an organ-allocation committee. A model suitable for rural remote patient monitoring may be unacceptable for resolving normative disputes in transplant ethics.

The authors call for evaluating each system within the specific context in which it is used. The ethical weight of urgency, seriousness, and patient access must be acknowledged rather than overlooked in favor of a one-size-fits-all transparency standard.

Human oversight, explainability alternatives and managing bias

The study also explores the role of human involvement and shared decision-making. The authors argue that explainability does not have to come directly from the AI system itself. Doctors can provide post-hoc rationalizations based on medical knowledge, cross-checking AI recommendations against established clinical practice. This approach preserves accountability while still harnessing the benefits of opaque algorithms.

The authors outline various forms of oversight: regulatory bodies testing for accuracy and safety, clinicians validating AI outputs, and patients making informed decisions with medical guidance. The presence of human interpretation acts as a bridge between opaque models and patient autonomy.

The paper also addresses bias, one of the strongest criticisms of black-box AI. The authors acknowledge that biased training data can lead to unequal performance across demographic groups. However, they caution against treating bias as a reason to reject black-box AI outright. In many cases, AI-driven decisions can be audited, monitored, and corrected more easily than human biases, which are often invisible and unregulated. Bias does not disappear, but its presence can be systematically identified through testing and inspection.

Bias becomes a prohibitive factor only when it meaningfully reduces accuracy or disproportionately harms disadvantaged populations. If an algorithm performs poorly for certain groups, it cannot be considered reliable. That makes the problem one of accuracy rather than explainability. Ethical use requires rigorous testing, regulatory scrutiny, and mechanisms for patient redress, not a universal ban on opacity.

Accountability and transparency, as the study stresses, must be maintained at the system level, even when individual AI decisions cannot be fully explained. Regulators, developers, and clinicians must share responsibility for ensuring appropriate use. When these structures are in place, black-box AI can be ethically deployed without undermining patient rights.