AI epidemiology offers practical path to trustworthy AI

The study diagnoses what it describes as a growing explainability crisis. As AI models have scaled in size and complexity, particularly with the rise of foundation models and large language models, traditional explainability techniques have struggled to keep pace. Methods such as feature attribution, attention visualization, mechanistic interpretability, and post hoc rationales often fail to provide stable, actionable insights into model behavior.

Despite years of research into model interpretability, institutions continue to struggle with understanding, auditing, and governing the behavior of large, complex AI systems in real-world settings. A new study argues that the problem lies not in insufficient technical tools, but in a flawed way of thinking about explainability itself.

That argument is laid out in the research paper AI Epidemiology: Achieving Explainable AI Through Expert Oversight Patterns, published on arXiv. The study proposes a fundamental shift away from model-centric explainability toward a population-level, governance-focused framework that treats AI behavior as a system to be monitored, audited, and corrected through expert oversight rather than fully decoded from within.

Why traditional explainable AI is failing in practice

The study diagnoses what it describes as a growing explainability crisis. As AI models have scaled in size and complexity, particularly with the rise of foundation models and large language models, traditional explainability techniques have struggled to keep pace. Methods such as feature attribution, attention visualization, mechanistic interpretability, and post hoc rationales often fail to provide stable, actionable insights into model behavior.

According to the research, these techniques suffer from three structural limitations. First, they are brittle. Small changes in inputs, training data, or model architecture can lead to large shifts in explanations, even when outputs remain similar. Second, they do not scale well across tasks, domains, or model updates, making them impractical for institutions that deploy AI systems in dynamic environments. Third, they focus on internal mechanisms that are often irrelevant to real-world governance questions, such as whether a system is safe, biased, or reliable in practice.

The paper argues that explainability research has been overly influenced by a desire to peer inside the model, assuming that transparency at the level of weights or activations is both achievable and necessary. In real-world settings, however, decision-makers rarely need a full account of how a model computes an answer. What they need is the ability to detect risk, identify failure modes, and intervene before harm occurs.

This gap between technical explainability and operational oversight has become especially pronounced in high-stakes domains. Regulators increasingly demand explainable systems, yet institutions struggle to comply in ways that meaningfully improve safety or accountability. The study positions this mismatch as a conceptual error rather than a technical shortcoming, arguing that explainability should be reframed as a governance problem rather than an interpretability problem.

Introducing AI Epidemiology as a Governance Framework

To address this challenge, the study introduces AI Epidemiology, a framework inspired by public health surveillance rather than cognitive science or machine learning theory. Instead of asking how a model internally produces an output, AI Epidemiology asks which outputs fail, how often experts override them, where errors cluster, and which observable features predict future failure.

In this framework, AI outputs are treated as a population. Each interaction between an AI system and a human expert becomes a data point. Expert decisions, such as accepting, correcting, or overriding AI recommendations, serve as key signals of risk. Over time, patterns emerge that reveal where and when AI systems are unreliable, biased, or misaligned with institutional standards.

Experts are not asked to label data, explain their reasoning, or provide additional feedback beyond their normal professional actions. Their routine decisions within existing workflows automatically generate oversight signals. This allows institutions to monitor AI performance continuously without increasing cognitive or administrative burden on professionals.

The study formalizes this process through the Logia protocol, which standardizes how AI interactions are recorded and analyzed. Each interaction is captured through a compact set of structured fields, including the task context, the AI recommendation, and the expert’s response. Additional assessment variables capture dimensions such as perceived risk, alignment with guidelines, and factual accuracy. These variables function as exposure indicators that can be analyzed statistically across large populations of AI outputs.

By aggregating these signals, AI Epidemiology enables institutions to identify failure clusters, predict high-risk outputs, and adjust deployment strategies accordingly. Importantly, this can be done independently of the underlying model architecture. Whether the system is rule-based, neural, or hybrid is irrelevant. What matters is how it behaves in practice and how experts respond to it.

From model transparency to institutional accountability

The study argues that this shift has significant implications for AI governance. By focusing on observable behavior rather than internal mechanisms, AI Epidemiology aligns explainability with institutional accountability. It allows organizations to demonstrate oversight, risk management, and responsiveness in ways that are meaningful to regulators, professionals, and the public.

One of the paper’s key claims is that explainability does not require full transparency. In complex systems, transparency can be both illusory and counterproductive. What institutions need is reliable early warning systems that flag emerging risks before they escalate. AI Epidemiology provides this by treating expert overrides as leading indicators of failure.

The study also focuses on resilience. Because the framework is model-agnostic, it remains effective even as models are updated, retrained, or replaced. This addresses a major weakness of many explainability tools, which must be re-engineered for each new model version. AI Epidemiology instead builds institutional memory around patterns of risk and correction, allowing oversight to persist over time.

To demonstrate feasibility, the paper presents a real-world application in a clinical setting. In ophthalmology workflows, AI outputs were standardized and evaluated using the Logia protocol. The results showed strong alignment between expert judgment and automated assessments, supporting the claim that AI outputs can be monitored at scale without disrupting professional practice. This case illustrates how AI Epidemiology can function in high-stakes environments where reliability and accountability are critical.

In addition to healthcare, the study argues that the framework is broadly applicable across domains where expert oversight already exists, including law, finance, education, and public administration. Anywhere humans routinely review, correct, or validate AI outputs, AI Epidemiology can transform those interactions into governance signals.