From Black Box to Accountability: Why Explainable AI Could Make or Break Autonomous Driving

When a human driver causes a crash, investigators can ask what the driver saw, knew or intended. When an autonomous vehicle causes one, the answer may be buried inside a complex AI system whose decision-making cannot easily be reconstructed.

A new review in AI & SOCIETY argues that this "black-box" problem could become one of the defining legal and governance challenges of autonomous mobility. In "Systemic explainability for autonomous driving: connecting ethics to law and society," University of Bologna researcher Marco Sanchi makes the case for explainability as a foundational requirement linking safer design with accountability, liability and social acceptance.

Safety Is No Longer Just About Whether the Car Works

Autonomous vehicles (AVs) promise safer roads, more efficient traffic and improved mobility, yet many of the technologies enabling those benefits also make the systems harder to interpret. Deep-learning architectures, particularly end-to-end systems, can generate driving decisions through complex internal processes that are not immediately understandable to developers, passengers or regulators.

The author argues that this opacity is not merely a transparency problem. It can directly affect functional safety because hidden biases, technical limitations or cascading failures may be difficult to identify before they produce dangerous behavior. Explainability can help expose how a vehicle interprets its environment, where its limitations lie and why particular actions emerge under specific road conditions.

It shifts the meaning of safety from pure performance toward inspectable performance. A vehicle that performs well in testing but offers little visibility into how it responds when conditions change may be harder to certify, supervise or improve. In that sense, XAI becomes part of the safety infrastructure itself, supporting debugging, risk identification, safety-case documentation and scrutiny of model behavior.

The study also emphasizes explainability-by-design, in which interpretability is built into the system from development rather than added only after deployment. Post-hoc explanations remain useful, particularly for existing black-box systems, but intrinsic explainability could provide a stronger connection between design choices, operational behavior and later regulatory review.

Human Oversight Means Little if the Human Cannot Understand the Machine

Policymakers frequently respond to autonomous AI by insisting that humans remain "in the loop." However, the review exposes a practical weakness in that principle: meaningful oversight depends on whether the human actually understands enough about the system to intervene effectively.

In safety-critical driving situations, fallback-ready users or supervisors may need to take control quickly. But if a person is unaware of what the vehicle has detected, why it has chosen a particular maneuver or what uncertainty the system is facing, formal human oversight may offer little real protection. The review therefore links explainability directly to situational awareness and interactive safety.

Real-time, context-sensitive explanations could help users understand the vehicle's limitations, recognize abnormal behavior and decide whether intervention is necessary. The study connects this directly with European AI governance requirements emphasizing human supervision, interpretable outputs and the ability of overseers to override or stop high-risk AI systems when necessary.

However, more information is not automatically safer. A driver confronted with streams of technical explanations during an emergency could become overloaded rather than empowered. Sanchi therefore highlights timing and relevance: explanations need to be understandable, personalized and concentrated around situations where they meaningfully support control.

Regulators may eventually need to move beyond simply requiring "human oversight" and define what information must be available for that oversight to be meaningful. Otherwise, humans risk carrying responsibility for decisions they cannot realistically understand or influence.

Explainability Could Determine Where Responsibility Lands

Traditional liability systems depend heavily on concepts such as causation, foreseeability, fault and control. Autonomous systems disrupt those assumptions because a harmful action may emerge from complex machine behavior rather than a straightforward human decision.

The review describes a potential "responsibility gap" in which no individual actor possesses sufficient knowledge or control to be an obvious bearer of responsibility. Developers may have contributed only one component, manufacturers may rely on multiple interconnected systems, and vehicle supervisors may be unable to predict the behavior that ultimately produces harm.

This creates another danger: scapegoating. A human overseer may be blamed simply because legal systems require a responsible person, even when the vehicle acted unpredictably and the individual lacked meaningful opportunity to intervene. Sanchi treats this as an ethical and legal problem arising when responsibility is assigned without a credible causal or fault-based connection.

Explainability cannot eliminate the unpredictability of autonomous systems, but it can make accident reconstruction more credible. Post-hoc XAI techniques can help investigators identify which conditions, inputs and internal factors influenced a vehicle's decision, allowing causal chains to be reconstructed even after an opaque system has acted.

For insurers, courts and transport regulators, this could become one of XAI's most important practical uses. Explainability may help distinguish a defective model from poor supervision, flawed design choices or genuinely unforeseeable behavior. As autonomous transport expands, the ability to answer "why did the vehicle do that?" could become as important legally as determining what physically occurred.

Trust Should Be Calibrated, Not Manufactured

Public acceptance is another area where explainability could have systemic effects. AVs operate in environments characterized by vulnerability and risk, meaning trust cannot simply be engineered through branding or reassuring performance claims. Users need an accurate understanding of what a system can and cannot do.

The review distinguishes trust from trustworthiness. Trustworthiness refers to qualities such as safety, robustness and lawful behavior; trust concerns whether people actually rely on the technology. Problems emerge when the two become misaligned. Too little trust can inhibit adoption, while excessive trust can encourage over-reliance and automation bias.

Explainability can help calibrate expectations by communicating limitations, uncertainty and reasons for particular actions. The literature reviewed by Sanchi links explanations with improved perceptions of safety, usefulness, transparency and accountability, while also suggesting that explanations can help restore trust following failures.

This makes explainability a social-governance issue as much as a technical one. Successful autonomous mobility will require manufacturers to demonstrate not simply that vehicles are capable, but that their behavior remains intelligible and contestable when decisions affect passengers, pedestrians and other road users.

For developing economies, this issue deserves particular attention. Autonomous systems may increasingly be imported into jurisdictions with different road conditions, infrastructure quality, legal systems and regulatory capacity. A model validated in one environment may behave differently in another, strengthening the case for explanations that local authorities and operators can scrutinize rather than relying entirely on manufacturer assurances.

The study proposes the idea of systemic explainability: XAI should connect vehicle development, safety assurance, human interaction, accountability, liability and social acceptance rather than addressing each problem separately. Explainability built early into a vehicle can produce benefits that cascade through deployment, oversight and accident investigation.

The review remains conceptual, however. It does not provide new empirical evidence showing that explainable autonomous vehicles will necessarily reduce crashes, settle liability disputes or accelerate public adoption; no new dataset was generated or analyzed. Its European regulatory focus also means the framework will require adaptation across jurisdictions with different insurance, tort and transport regimes.

Regardless, the future of autonomous driving will not be determined solely by whether machines can drive. It will also depend on whether societies can inspect their reasoning, challenge their decisions and identify responsibility when automation fails.