Generative AI’s biggest risk may be that no one can fully explain it

A major new systematic review finds that explainability has become the weakest link in the generative AI ecosystem, with current methods struggling to keep pace with the complexity and societal reach of these systems.

The study, titled Explainable Generative AI: A Two-Stage Review of Existing Techniques and Future Research Directions, published in the journal AI, assesses how explainability is conceptualized, implemented, and evaluated in generative AI systems. Using a rigorous two-stage review process, the authors analyze both high-level review literature and empirical studies to map the state of the field and identify critical gaps that threaten the safe deployment of generative models.

Explainability lags behind the rise of generative AI

Explainability research has not kept pace with the rapid evolution of generative AI. While traditional explainable AI methods were developed for predictive models that classify or label inputs, generative systems operate differently. They do not produce single deterministic outputs. Instead, they sample from probability distributions, navigate latent spaces, and generate multiple valid outputs for the same input. This shift fundamentally changes what it means to explain an AI system.

Across 261 initially identified studies, the authors ultimately analyze 63 peer-reviewed articles published between 2020 and early 2025. The first stage synthesizes insights from 18 review papers, while the second stage examines 45 empirical studies that apply explainability techniques in real generative systems. Together, these analyses show that most explainability approaches used today are adaptations of older methods that were never designed for generative behavior.

Feature attribution tools such as SHAP, LIME, and saliency-based techniques remain dominant, despite their limited ability to capture how generative models produce outputs over time. These methods often focus on isolated tokens, pixels, or features, offering partial explanations that fail to reflect the stochastic and multi-step nature of generation. As a result, explanations may appear convincing without being faithful to the underlying model behavior.

The study highlights widespread conceptual fragmentation. Different research communities use inconsistent terminology to describe similar ideas, while identical terms are sometimes applied to fundamentally different techniques. This lack of shared definitions makes it difficult to compare results, replicate findings, or develop standards that regulators and practitioners can rely on. Explainability, the authors argue, remains an emerging concept rather than a mature discipline within generative AI research.

What current methods explain and what they miss

Next up, the review examines how explainability is actually applied in practice across generative AI domains. The authors find that empirical studies span a wide range of applications, including healthcare decision support, education technologies, cybersecurity, creative tools, industrial systems, and legal and policy-oriented use cases. Transformer-based models dominate this landscape, followed by generative adversarial networks and variational autoencoders.

Despite this diversity, common patterns emerge. Most explainability efforts focus on post hoc explanations, meaning they attempt to interpret model behavior after generation has occurred rather than embedding transparency into the model itself. These explanations are typically local, explaining individual outputs rather than global system behavior. While this approach can help users understand specific responses, it does little to illuminate how models behave across distributions, contexts, or edge cases.

The review identifies three recurring tensions that define explainability in generative AI. The first is transparency of the generative mechanism, which concerns insight into architectures, latent representations, and sampling processes. The second is user-centered interpretability, which focuses on whether explanations are meaningful and usable for humans with different levels of expertise. The third is evaluation fidelity, which addresses whether explanations accurately reflect what the model is actually doing.

Most current methods, the authors find, optimize for one of these dimensions at the expense of the others. Technical approaches that reveal internal mechanisms often produce explanations that are inaccessible to non-experts. User-friendly explanations, such as natural language justifications, may improve trust and usability but lack verifiable fidelity. Evaluation practices are inconsistent, with many studies relying on anecdotal evidence or limited user studies rather than standardized benchmarks.

These weaknesses are especially concerning in high-stakes domains. In healthcare, education, and law, explainability is often required not only for usability but for accountability and safety. The review finds that while some studies report improvements in user trust and task performance when explanations are provided, few assess whether those explanations prevent harm, reduce bias, or enable meaningful oversight.

Regulatory pressure and the path forward

The study identifies a growing disconnect between regulatory expectations and technical reality. Policymakers worldwide are advancing frameworks that demand transparency, auditability, and accountability in AI systems. The European Union AI Act is repeatedly cited across the literature as a key external driver of explainability research. Yet the review concludes that current explainability techniques are not yet capable of meeting these regulatory demands in a consistent and reliable way.

The authors argue that explainability must be reframed as a system-level property rather than a set of optional tools. For generative AI, explanations should account for training data influences, retrieval mechanisms, latent-space dynamics, and output variability. This requires moving beyond retrofitted explanations toward generative-native frameworks that are designed alongside model architectures.

The study outlines several priority directions for future research. These include developing model-agnostic explainability frameworks that work across different generative architectures, establishing standardized evaluation protocols that balance technical fidelity with human-centered assessment, and addressing the trade-offs between interpretability and model performance. High computational cost remains a barrier for many advanced explainability methods, particularly in real-time or large-scale applications.

Human-centered design is another critical gap. The review emphasizes that explanations must be tailored to different stakeholders, including developers, end users, domain experts, educators, and regulators. A one-size-fits-all approach to explainability is unlikely to succeed in systems that affect diverse populations and decision contexts. Interactive and adaptive explanation interfaces are identified as promising avenues, allowing users to explore model behavior rather than passively consume explanations.

The authors also highlight the potential of hybrid approaches that combine statistical methods with symbolic reasoning, provenance tracking, and audit mechanisms. Such systems could bridge low-level technical transparency with high-level accountability, making explanations both faithful and actionable. However, these approaches remain rare and experimental.