AI Agents Face a Trust Crisis as False Reports Bypass Advanced Security Controls

Artificial intelligence agents are increasingly capable of performing operations rather than merely generating recommendations. However, the ability to authenticate an AI system does not establish whether its instructions are appropriate, whether the information supporting its decisions is reliable, or whether its actions will remain within authorized boundaries. These distinctions are emerging as central challenges in securing autonomous AI workflows.

A study titled "Trust, Accountability and Control: Post-Quantum Authentication and Evidence-Driven Authorization for Artificial Intelligence Agents", published in the journal Information, investigates how cryptographic authentication, behavioral evidence and execution controls can operate together. Authored by Domenico Furno and Vincenzo Loia of the University of Salerno, Italy, the research examines a security architecture designed to restrict AI agents' operational authority while preserving accountability for their decisions.

The findings expose a fundamental weakness in relying on authenticated identities as a measure of operational trust. Although the proposed architecture successfully prevented unauthorized effects under specified conditions, it remained vulnerable to dishonest assessments from authorized observers and privileged operations that bypassed its enforcement mechanisms. The experiments also revealed a significant trade-off: additional security restrictions can prevent legitimate operations from proceeding.

Authentication Is Not Authorization: The Security Problem Behind Autonomous AI

Traditional authentication establishes whether a message originates from an identified source. Autonomous AI systems require further safeguards because an authenticated agent can still propose an unauthorized operation, act beyond its assigned responsibilities or rely on inaccurate information when requesting permission.

The challenge becomes more complicated when several agents participate in a workflow. One component may propose an action, another may assess compliance, and a separate controller may determine whether execution is permitted. Each component introduces a distinct security boundary, particularly when observations influence the authority granted during subsequent operations.

The researchers developed a framework combining trust, accountability and control, known as TAC, with post-quantum digital signatures. Trust evaluates accumulated behavioral evidence, accountability assigns negative evidence to roles associated with violations, and control determines whether requested autonomy remains within permitted limits. Authorization therefore depends on both established permissions and recorded behavior.

The framework distinguishes the right to request an operation from the authority to execute it. An agent must present an authenticated request within its assigned scope, while an independently authorized observer supplies an assessment concerning that particular proposal. The controller checks both inputs before deciding whether execution should proceed.

The researchers also introduced persistent evidence records to prevent previously accepted observations from repeatedly influencing authorization decisions. Each accepted report occupies a unique position in the controller's history, ensuring that duplicated messages cannot artificially strengthen an agent's behavioral record under the specified storage assumptions.

Post-quantum authentication provides protection for the origin and integrity of these messages, but it does not independently establish their truthfulness. The distinction is essential because an observer possessing valid credentials can submit an inaccurate assessment without violating the cryptographic requirements.

Stronger Execution Controls Stop Unauthorized Changes, but Privileged Bypasses Remain

The experimental evaluation combined recorded AI-agent activity, controlled security tests, formal protocol verification and synthetic resource operations. Across 38,400 recorded controller positions per tested authentication path, the researchers found no mismatch between the original controller's decisions and those reproduced through authenticated evidence processing.

Further experiments demonstrated resistance to substituted commands, incorrect identities, unauthorized roles, altered operation details and repeated submissions. In one concurrent test involving 24 submissions of an identical request, the system admitted only one and classified the remaining 23 as duplicates.

The strongest execution-related findings emerged from comparisons between a protected dispatcher and an intentionally unprotected alternative. The protected mechanism verified the original command, committed authorization decision, current permissions and execution identity before allowing a resource operation.

Across 22 mediated configurations involving two signature algorithms, the protected dispatcher produced no unauthorized effects. The unprotected comparator generated unauthorized effects in 20 configurations, demonstrating how altered commands, repeated execution and outdated permissions can compromise systems lacking equivalent safeguards.

However, both privileged bypass attempts succeeded because they accessed the underlying resource outside the protected execution mechanism. The results establish that strong authorization checks cannot prevent harmful operations when other components retain direct access to protected resources.

The study also examined whether incomplete or delayed evidence could distort controller decisions. When all required observations eventually arrived, consumers reconstructed the expected authorization history. Missing observations, however, prevented progression, demonstrating that reliable evidence processing and continued service availability are separate challenges.

These findings establish conditional protection within the tested architecture rather than comprehensive resistance to arbitrary attacks. The researchers explicitly distinguish authenticated control from broader requirements involving trusted execution, resource isolation and independently verified external effects.

Adaptive AI Trust Creates a Trade-Off Between Security and Operational Progress

One of the most consequential findings concerns the practical cost of granting authority according to accumulated behavioral evidence. Unlike static enforcement, which applies predefined permissions, the adaptive controller restricts higher autonomy until an agent establishes sufficient positive evidence through successive assessments.

In a twelve-visit experiment involving four roles, the static enforcement baseline completed 48 compliant operations. The adaptive framework completed only 24, withholding the remaining 24 during its initial evidence-building period despite their compliance with the tested requirements.

The restrictions were intentional. Newly authenticated agents began without established behavioral authority, regardless of whether their cryptographic identities and operation permissions were valid. Their permitted autonomy increased only after the controller accumulated the required positive observations.

A separate scenario involving a truthfully reported access violation demonstrated that adaptive authorization could selectively delay the affected role. While other compliant roles obtained higher authority earlier, the access role's first permitted effect was delayed until the ninth visit.

Such behavior illustrates the framework's ability to distinguish roles according to their assessed conduct. However, the experiments do not establish that withholding compliant operations produces superior overall safety or operational performance compared with competent static enforcement.

More troubling results appeared when an authorized observer deliberately supplied false compliance information. In the controlled experiment, both static and adaptive systems permitted an operation violating the independently specified task limit because the signed assessment incorrectly classified the proposal as acceptable.

The observation carried a legitimate signature and satisfied the protocol's authentication requirements. Consequently, the controller accepted inaccurate evidence as valid input, demonstrating that the system could identify the reporting source without determining whether its assessment was truthful.

The researchers also measured the performance costs of post-quantum authentication. A complete signed request-and-observation pair took a median 3.406 milliseconds with ML-DSA-44, compared with 2.192 milliseconds using Ed25519 on the tested x86 configuration.

Message sizes differed substantially. The ML-DSA-44 request-and-observation pair required 11,053 encoded bytes, compared with 1,625 bytes for Ed25519. Separate experiments showed that software build optimization significantly influenced cryptographic performance, limiting the generalizability of timing comparisons.

These measurements exclude AI inference, network transport and external execution. They provide implementation-specific evidence about processing and representation costs rather than proof of production readiness, sustainable throughput or suitability for time-critical applications.

The Next AI Security Challenge Is Verifying Evidence, Not Just Identities

The research's contribution lies in connecting several established security mechanisms into a tested authorization process. Digital signatures authenticate requests and observations, scoped grants restrict their permitted uses, persistent records prevent repeated evidence consumption, and execution checks connect authorization decisions to controlled resource changes.

Formal verification strengthened the analysis by examining whether authenticated observations retained the required origin, scope, context and uniqueness properties. Additional tests deliberately removed protective mechanisms, exposing failures that helped establish why those components were necessary.

Nevertheless, the framework operates within clearly defined assumptions. Trusted provisioning, intact storage, reliable observer assessments and faithful execution remain necessary. The prototype does not provide comprehensive protection against compromised observers, global authorization inconsistencies, storage rollback or privileged bypasses.

The system also lacks mechanisms for global revocation, confidential transport and coordination between independently administered organizations. Its controlled tests used synthetic operations rather than deployed physical infrastructure, and the findings do not establish general superiority in task performance or operational safety.

For organizations developing autonomous AI workflows, the research points toward separating authentication, assessment and execution responsibilities. Narrowly defined permissions, reliable evidence retention, independent monitoring and protection against unauthorized execution become essential design considerations rather than interchangeable security features.

The findings also expose a governance problem involving accountability. A digital signature can associate a report with an authorized observer, supporting later investigation, but attribution alone cannot prevent an incorrect report from influencing a current decision. Strengthening the reliability of assessments therefore remains an unresolved requirement.

Additional analysis is needed to determine whether observer reports can be independently verified, resources remain isolated, compromised storage can be recovered and authorization works across independently managed systems. Evaluations involving reactive AI agents, real tools and physical devices would provide a clearer picture of the architecture's operational reliability.