Agentic AI needs evidence-based guardrails before it can be trusted in science

Agentic AI could reshape scientific research by coordinating complex workflows, retrieving evidence, extracting data and supporting regulatory decisions, but its use in high-stakes science depends on trust, not merely speed.

A perspective article, titled Evidence-based AI: from trailblazer to trustblazer? and published in Frontiers in Artificial Intelligence, presents a framework for making agentic AI auditable, reproducible and accountable. Based on evidence-based medicine and evidence-based toxicology, the article states that future AI systems in regulatory science should be built to show where evidence came from, how it was assessed, how uncertainty was handled and where human responsibility remains.

Trust becomes the key barrier for agentic AI in science

GenAI has already accelerated scientific work by helping researchers draft text, write code, sort literature and form hypotheses. Agentic AI goes further. These systems can plan, call external tools, coordinate specialized sub-agents and carry out multi-step tasks that resemble parts of a scientific workflow. The capability could help researchers manage evidence-heavy work in toxicology, regulatory science, medicine and environmental health. In real-world settings, agentic systems could search large bodies of literature, screen studies, extract data, assess risk of bias, synthesize findings, build decision tables and update conclusions as new evidence becomes available.

The opportunities are vast, and so are the risks. A mistake inside a long AI workflow can spread through later steps and produce a final recommendation that appears coherent, polished and authoritative. A missed study, weak source, faulty extraction or poor risk assessment can become harder to detect once it is wrapped into a fluent summary. In high-stakes settings, a useful AI system must do more than generate plausible text. It must preserve traceability, make its workflow reproducible under defined conditions, fit a declared context of use and communicate uncertainty clearly.

The study warns that science cannot rely on AI systems that simply sound convincing. A model may be useful for brainstorming or early drafting, but it is not ready to support public health, environmental safety or regulatory decisions unless its evidence trail can be inspected.

The researchers identify evidence-based medicine and evidence-based toxicology as proven models for managing this shift. Both fields developed methods to reduce selective citation, expert overconfidence and persuasive but weakly supported narratives. Their strength lies in disciplined process: pre-specified questions, reproducible searches, transparent inclusion criteria, structured data extraction, risk-of-bias appraisal, graded certainty and clear movement from evidence to decisions.

Agentic AI becomes valuable when it can convert those methods into executable infrastructure. Instead of allowing a model to produce a broad answer from opaque internal reasoning, a trustworthy system would divide the task into defined steps and preserve a record at every stage. The researchers describe this transition as a move from trailblazing to trustblazing. Trailblazing AI prioritizes novelty, capability and speed. Trustblazing AI places provenance, validation, uncertainty, documentation and human accountability at the center of design.

Evidence-based agent stacks could make AI workflows auditable

The article proposes Evidence-based Agent Stack, a modular architecture in which specialized AI agents perform narrow roles inside an evidence workflow. Each agent produces structured outputs that can be reviewed before the next step proceeds. The stack begins with a protocol agent, whose role is to translate the research question into a defined protocol, including the population, exposure or intervention, comparator, outcomes, eligible study types and analysis plan. This step is designed to lock the question and criteria before evidence screening begins, reducing the risk that conclusions are shaped after results appear.

A retrieval agent then searches approved sources using retrieval-augmented generation. This keeps outputs grounded in citable passages rather than model memory alone. A screening agent applies inclusion and exclusion criteria and records why evidence is accepted or rejected. An extraction agent captures predefined fields and marks missing information as not reported rather than filling gaps through guesswork.

A risk-of-bias agent supports appraisal of study credibility using established frameworks. The article treats this as a critical step because weak evidence can distort everything that follows. Risk-of-bias work remains context-sensitive, so the agent's role is assistance, evidence linking and consistency checking, not final judgment.

The stack also includes agents for synthesis, mechanism and causality, uncertainty, and evidence-to-decision translation. These components are meant to keep raw evidence separate from interpretation, label assumptions clearly and prevent final recommendations from hiding uncertainty or disagreement.

The uncertainty agent is crucial to the framework. In many AI outputs, uncertainty appears only as a brief caution at the end. In the proposed stack, uncertainty becomes a structured output in its own right, recording evidence gaps, conflicting findings, indirectness, model assumptions and limits of confidence.

The evidence-to-decision agent handles the final movement from evidence to recommendations. This step requires explicit criteria because scientific evidence alone does not decide policy. Trade-offs, feasibility, acceptability, values and responsibility must be documented, with final accountability remaining in human hands.

Across the entire stack, one rule is non-negotiable: no untraceable claims. Every extracted fact, especially numerical values, should link to a source. Every inference should be labeled as interpretation rather than direct evidence. Every model version, prompt, schema, corpus, retrieval setting and tool configuration should be recorded.

This level of versioning matters because agentic AI systems are not single tools, they are composite pipelines. Model weights, prompts, retrieval settings, chunking rules, extraction schemas and post-processing logic can all affect the final output. Without version control, a changed result may reflect pipeline drift rather than a genuine change in the evidence.

The article also flags automation traps. Prompt engineering can create the appearance of validation when a system is tuned repeatedly on small or convenient datasets and then tested on similar material. That can inflate performance and hide weaknesses. For high-stakes evidence work, prompts, schemas, retrieval settings and post-processing should be treated as part of the model and locked before testing.

Evaluation must also match the specific task. A system used for study screening may need very high recall. A system extracting numerical values may need strict accuracy. A system helping with regulatory toxicology may need conservative escalation when evidence is unclear. General benchmarks cannot establish readiness for every scientific setting.

The authors also note that large models are not automatically the best choice for every task. Smaller or more specialized models can outperform large language models in structured domains when strong datasets are available. In other words, trust must be earned through context-specific testing, not assumed from scale or polished output.

Why it matters for regulation and scientific accountability

The policy stakes are high because agentic AI is moving into areas where errors can shape public decisions. In toxicology, medicine, environmental health and regulatory science, the risk is not just a wrong answer. The deeper risk is a wrong workflow that produces a persuasive conclusion without sufficient evidence, documentation or accountability.

The article points to TREAT, short for Trustworthiness, Reproducibility, Explainability, Applicability and Transparency, as a practical governance framework for regulatory AI.

Reproducibility also needs a new meaning for AI. Traditional scientific validation often assumes that the same protocol should produce comparable results. Agentic systems are more complex because stochastic outputs, model updates and changing retrieval systems can affect results. The relevant standard becomes consistent performance under defined conditions, with clear documentation of uncertainty and limits.

E-validation adds another layer. Instead of treating AI validation as a one-time approval, it treats credibility as a lifecycle process. Systems must be validated, monitored, checked for drift and revalidated when evidence, data sources, models or workflows change. Modern AI systems are not static. A change in model version, retrieval index, prompt template or source database can shift an output. A system that was reliable in one setting may degrade later or behave differently in a new context. Scientific users need triggers for revalidation when those changes occur.

The article also describes the possible role of companion agents that monitor systems after deployment. Such agents could scan for new evidence, detect shifts in data representativeness, flag performance problems, initiate back-testing and alert users if earlier conclusions may need revision.

Regulatory oversight must focus on the full workflow, not only model performance. A high-stakes AI system should be able to preserve provenance, version its components, separate extraction from inference, report uncertainty, abstain when evidence is insufficient and escalate unresolved conflicts to human experts.

Research institutions should use Agentic AI as auditable decision support, not as an autonomous authority. Protocol locks, evidence gates, review logs, escalation rules and human sign-off should be built into workflows before AI outputs influence scientific or regulatory judgments.

For developers, the design target shifts from fluency to accountability. The most trusted systems may not be the fastest or most impressive in demos. They may be the one that best document their sources, expose their limits, preserve uncertainty and allow independent review.