Human oversight of AI exposed as new security risk

Attack surfaces are not limited to the AI systems themselves. Oversight can be compromised via communication channels, infrastructure, or the humans tasked with supervision. This includes poisoning attacks that alter the training data, adversarial attacks that subtly manipulate outputs, or explainability attacks that feed misleading explanations while leaving outputs intact. Even if the system appears stable, oversight may be blind to the manipulation.

Human oversight has long been positioned as a cornerstone of safe artificial intelligence, but new research warns that oversight itself could become an attack vector. A team of researchers has published an analysis showing that human oversight systems may be manipulated or compromised, exposing organizations to risks that current AI regulations do not fully address.

The study, Secure Human Oversight of AI: Exploring the Attack Surface of Human Oversight, published on arXiv in September 2025, argues that simply mandating human supervision in high-risk AI systems is insufficient. The authors highlight how attackers can exploit both technical and human weaknesses in oversight channels, urging policymakers, organizations, and developers to treat secure oversight as a critical priority.

How human oversight becomes an attack surface

The research breaks new ground by shifting the focus from the effectiveness of oversight to its vulnerability. The authors note that oversight relies on four essential requirements: epistemic access to system operations, causal power to intervene, self-control to act appropriately, and fitting intentions aligned with oversight goals. If attackers undermine any of these, oversight loses its protective function.

Attack surfaces are not limited to the AI systems themselves. Oversight can be compromised via communication channels, infrastructure, or the humans tasked with supervision. This includes poisoning attacks that alter the training data, adversarial attacks that subtly manipulate outputs, or explainability attacks that feed misleading explanations while leaving outputs intact. Even if the system appears stable, oversight may be blind to the manipulation.

Moreover, oversight is vulnerable to direct attacks on human supervisors. Social engineering, coercion, and bribery are identified as serious risks, alongside insider threats from employees with authorized access but malicious intent. The authors argue that these vulnerabilities mean oversight cannot be considered inherently secure simply because a human is present in the loop.

What threat vectors put oversight at risk

The study maps a range of potential attack vectors that could erode the safety of human oversight. On the technical side, denial-of-service and man-in-the-middle attacks can disrupt or intercept communications between AI systems and supervisors. Poisoning attacks target training data to bias outputs, while adversarial examples subtly distort inputs in ways that trick the AI and mislead overseers.

Explainability is also an attack target. As oversight increasingly relies on explainable AI modules to interpret decisions, attackers may manipulate these explanations without altering outcomes. This creates a false sense of security for human supervisors, who may believe they are monitoring a transparent system while being fed distorted justifications.

On the human side, attackers can bypass technical safeguards altogether. Oversight personnel can be deceived through phishing, coerced into negligence through threats, or bribed into collusion. Insider threats, where employees misuse access for personal or political reasons, represent one of the hardest challenges to mitigate.

The authors argue that these risks will only grow as regulations such as the EU AI Act expand oversight requirements. By mandating human involvement in high-risk applications, policymakers may inadvertently create incentives for attackers to exploit oversight as the weakest link in the safety chain.

How oversight can be secured against attacks

To address these threats, the study outlines a series of hardening strategies that combine technical, organizational, and human-centered measures. On the technical front, secure network management, intrusion detection systems, and strong encryption are recommended to shield communications. Transparency tools can help identify manipulations, while red teaming exercises can stress-test oversight systems against simulated attacks.

Governance frameworks also play a vital role. Standards such as ISO/IEC 27001, along with accountability, auditability, and authenticity (the “AAA framework”), provide structures for securing oversight processes. Regular auditing and monitoring are emphasized to ensure that oversight functions as intended and that vulnerabilities are detected early.

Human factors receive equal attention. Training oversight personnel to resist manipulation, bribery, or coercion is considered essential. Organizations must also implement checks and balances to prevent single points of failure, ensuring that oversight decisions cannot be easily subverted by individual actors.

The authors propose that in the long term, AI-assisted “amplified oversight” may reduce vulnerabilities by simplifying oversight tasks for humans. By using AI tools to filter, prioritize, or explain system operations, supervisors may gain clearer insights and reduce the cognitive burden that attackers could exploit. However, this too requires careful governance to avoid replacing one opaque system with another.