Relying on AI Without Understanding It: The Inevitable Tradeoff

The prevailing belief in AI research is that better explanations lead to greater trust. AI developers have invested heavily in explainable AI (XAI) frameworks designed to make machine learning models more transparent, assuming that people need to understand AI decisions before they can trust them. However, this study challenges that assumption, revealing that explanation itself is often unattainable, even when conditions for successful communication exist.

Artificial Intelligence (AI) is rapidly becoming an integral part of decision-making across industries, from healthcare and finance to governance and defense. As AI systems take on increasingly complex and high-stakes tasks, their lack of transparency has raised concerns about trust. Conventional wisdom suggests that explanation precedes trust - that people trust AI when they understand how it makes decisions. However, a new study challenges this assumption, arguing that trust in AI may be inevitable even when explanations are incomplete or unavailable.

A recent study titled “Why Trust in AI May Be Inevitable” by Nghi Truong, Phanish Puranam, and Ilia Tsetlin, published in Joint Proceedings of the ACM IUI Workshops (2025), explores the fundamental limits of explainability in human-AI interactions. Using a formal model, the researchers demonstrate that explanation can fail even under ideal conditions - where actors are rational, honest, and possess overlapping knowledge - due to the inherent difficulty of searching for shared understanding within finite time constraints. This has profound implications for AI adoption, as it suggests that trust in AI will often arise not from understanding but from necessity.

The limits of explanation in AI systems

The prevailing belief in AI research is that better explanations lead to greater trust. AI developers have invested heavily in explainable AI (XAI) frameworks designed to make machine learning models more transparent, assuming that people need to understand AI decisions before they can trust them. However, this study challenges that assumption, revealing that explanation itself is often unattainable, even when conditions for successful communication exist.

The researchers model explanation as a search process through knowledge networks, where the explainer (AI or human) must find connections between shared concepts and the information being explained. This process is challenging because:

1. Explanation requires discovering shared knowledge—not just possessing it.
2. The search for common understanding takes time, and explanation efforts may be abandoned before the connection is found.
3. Different knowledge structures between AI and humans make alignment difficult, especially when AI represents information differently from human cognitive models.

Even when explanation is possible, it often demands more time and effort than is practical, leading users to default to trust as a substitute for understanding. In essence, the study suggests that while explanation helps build trust, there are many cases where trust must exist before explanation can even begin.

How AI gains trust without full explainability

The study argues that, in practice, people often trust AI not because they fully understand it, but because they have no better alternative. As AI systems become more capable, people rely on them even when explanations are absent or incomplete.

For example, modern Large Language Models (LLMs) generate highly convincing responses but struggle to provide detailed reasoning for their outputs. Users may trust these models for practical tasks, such as text generation or medical diagnosis, even though they do not fully comprehend how the AI arrived at its conclusions. This creates a paradox: AI systems gain trust through repeated successful interactions rather than deep transparency.

This phenomenon is particularly concerning in high-risk domains like healthcare, law, and finance, where misplaced trust in AI could lead to serious consequences. The study highlights two key risks:

• Spurious explanations: AI systems may provide seemingly logical justifications that do not reflect their true decision-making process.
• Reduced demand for explanations over time: As AI systems become more embedded in daily life, users may stop questioning their decisions, leading to over-reliance on opaque models.

The trust-explanation tradeoff in human-AI collaboration

A key insight from the study is that trust and explanation are not always aligned - sometimes, trust is necessary before explanation can even be attempted. The researchers identify two main scenarios where this occurs:

First, in cases where AI knowledge is fundamentally different from human knowledge, explanation attempts may fail due to differences in how information is represented. For instance, an AI system analyzing protein structures in molecular biology may base its reasoning on vast amounts of statistical data that are difficult for a human expert to interpret in a meaningful way.

Second, in time-sensitive decision-making environments, such as autonomous driving or medical emergency response, there is no time for full explanations before action is required. In such cases, users must trust AI systems to function correctly, even if they do not fully understand them.

The study suggests that AI designers should shift focus from pure explainability toward building trust through reliability. Instead of relying solely on XAI techniques, AI systems should:

• Demonstrate consistent accuracy in real-world applications.
• Use verifiable external benchmarks to establish credibility.
• Integrate human oversight where necessary to provide safety checks.

What this means for the future of AI trustworthiness

The findings of this study highlight a critical challenge for AI adoption: trust in AI will often be based on necessity rather than understanding. While explainability remains a valuable goal, the reality is that AI systems will not always be able to fully justify their decisions in a way that humans can understand.

This has major implications for AI regulation and deployment. Governments, businesses, and researchers must ensure that AI trust is built on demonstrable reliability rather than blind faith. Strategies to achieve this include:

• Developing AI audit mechanisms to verify model performance.
• Implementing robust safety measures to prevent AI errors from causing harm.
• Educating users on AI capabilities and limitations, so that trust is placed appropriately.

Ultimately, the study suggests that AI trust is not just about explanation - it is about performance, safety, and ethical responsibility. As AI continues to shape the future, the challenge will not be eliminating trust gaps entirely but ensuring that trust is well-placed, informed, and justified.