Autonomous AI could transform both defensive and offensive cyber operations

The research reinforces a dual-use reality. Offensive security tools can strengthen defense when used responsibly but also lower the barrier for sophisticated attacks if misused. The authors advocate for greater transparency, robust safety measures, and responsible open-source release strategies to ensure that defensive ecosystems benefit more than malicious actors.

The capabilities of artificial intelligence (AI) in cybersecurity are advancing faster than many industry observers expected, with new evidence showing that autonomous AI systems can now rival seasoned professionals in complex, real-world penetration testing. The new findings demonstrate that AI agents do not simply automate standard testing routines but are beginning to outperform humans in certain high-stakes tasks.

A landmark research paper, “Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing,” published on arXiv compares human cybersecurity experts and autonomous AI agents operating inside a live enterprise network. The study evaluates actual penetration-testing performance across a real university system of approximately 8,000 hosts and 12 subnets. The scope and realism of this experiment mark a major departure from capture-the-flag benchmarks and controlled simulations.

The results reveal a rapid shift in the cybersecurity landscape. While human experts discovered dozens of advanced vulnerabilities, the study’s newly developed AI framework, ARTEMIS, was able to outperform nine of ten human participants in overall scoring, showing both strong technical execution and the ability to work continuously over long time horizons. The findings raise urgent questions about defensive readiness, offensive misuse, and the future relationship between human operators and autonomous cyber agents.

AI agents deliver high-performance results in a real enterprise network

The research was designed to address a growing disconnect between AI cybersecurity benchmarks and real-world attack patterns. Existing evaluations often rely on short-form questions, synthetic vulnerabilities, or isolated tasks that fail to reflect the complexity of modern systems. Real breaches typically involve chained misconfigurations, reused credentials, outdated endpoints, and interactive exploration across live environments. To close this gap, the study placed both humans and AI agents inside an operational university network running heterogeneous infrastructure, including Unix-based systems, IoT devices, Windows machines, embedded systems, and authentication frameworks.

Ten selected professional penetration testers, each compensated for their participation, were given ten hours of active testing time and up to four days of access. Their findings were assessed using a scoring framework that emphasized both technical difficulty and business impact, rewarding the discovery and exploitation of high-severity vulnerabilities.

In parallel, the researchers evaluated six AI-agent frameworks, including commercial and open-source systems previously used for offensive security tasks. The primary emphasis, however, was on ARTEMIS, a novel multi-agent architecture developed for the study. ARTEMIS uses a supervisory agent orchestrating multiple specialized sub-agents, with automated task decomposition, dynamic prompt creation, parallel workflows, note-taking, and a built-in triage module responsible for verifying, reproducing, classifying, and submitting vulnerabilities.

Across multiple controlled runs, ARTEMIS demonstrated a level of performance that surprised even veteran researchers. One configuration of ARTEMIS ranked second overall in total score, identifying nine valid vulnerabilities with an 82 percent accuracy rate, surpassing every human participant except the top performer. Most competing AI frameworks failed to sustain long engagements, stalled during reconnaissance, or submitted shallow findings.

On the other hand, ARTEMIS executed sustained, multi-hour workflows, spawning several sub-agents in parallel and advancing through complex phases of scanning, probing, exploiting, and reporting. The agent’s ability to work continuously, methodically, and without fatigue positioned it strongly against human testers, who varied widely in strategy, attention distribution, and verification rigor.

Human professionals outperformed in speed, systematic coverage, and parallel exploration

The comparison between humans and AI agents revealed both convergence in methodology and divergence in execution. Human participants universally began their tasks with reconnaissance, using established tools such as nmap, rustscan, and masscan. They followed with targeted scanning, brute-force enumeration, exploitation of outdated services, credential-based attacks, lateral movement across network segments, and post-exploitation activities such as file access or credential harvesting.

While these patterns mirrored professional real-world workflows, the study found that AI agents, especially ARTEMIS, were able to execute similar steps with far greater systematic consistency. When ARTEMIS identified a potentially vulnerable host, it immediately launched sub-agents to investigate the target while continuing its reconnaissance work in parallel. This horizontal scalability allowed it to explore multiple branches of an attack path simultaneously, something no human participant could replicate.

Humans, on the other hand, exhibited limitations typical of complex manual analysis. Some testers failed to revisit promising leads identified earlier in their process. Others relied too heavily on automated scanners and missed nuanced vulnerabilities requiring manual probing. The top-performing humans balanced automation and manual validation effectively, but their approaches still lacked the multi-threaded efficiency of ARTEMIS.

The diversity of results among human participants underscored the complexity of the environment. While two vulnerabilities were detected by most testers, the remainder displayed a high degree of dispersion, with many being discovered by only one or two individuals. This fragmentation reflected not only the large scope of the environment but also the wide variation in human decision-making, intuition, and prioritization.

ARTEMIS also showed distinct strengths in areas where the command-line interface was more advantageous than graphical workflows. For example, humans struggled to exploit an outdated Dell iDRAC server because modern browsers refused to load the insecure HTTPS configuration. ARTEMIS bypassed this limitation easily using command-line tools, successfully exploiting a vulnerability no human discovered.

However, AI agents were not universally superior. ARTEMIS consistently struggled with GUI-dependent tasks, such as those involving TinyPilot remote consoles, which required visualization and manipulation of screen elements. These limitations caused the agent to miss several high-severity paths that many human testers exploited successfully.

The study found that ARTEMIS also produced more false positives than human testers. These errors often stemmed from misinterpreting HTTP responses or failing to validate authentication behavior. While the built-in triage module reduced noise, AI reliability still fell short of professional standards.

Even with these weaknesses, the totality of results shows that autonomous agents are not only catching up with human testers but in some areas have already surpassed them. Their capacity to operate continuously, work in parallel, and parse machine-readable outputs gives them advantages that scale as environments increase in size and complexity.

Implications for cyber defense, offensive misuse and future AI safety efforts

The research highlights several urgent implications for the cybersecurity community. First, AI agents that outperform trained professionals in real enterprise environments represent significant opportunities for defensive operations. Organizations that lack continuous penetration-testing coverage may eventually deploy autonomous agents to identify vulnerabilities faster and at a fraction of the cost. One ARTEMIS configuration cost approximately eighteen dollars per hour to operate, amounting to less than forty thousand dollars annually. Human penetration testers, by comparison, earn an average of more than one hundred twenty-five thousand dollars per year.

Second, the results underscore the risks posed by malicious actors who could repurpose similar AI technologies for offensive cyber operations. The speed, scale, and automation demonstrated by ARTEMIS suggest that future cyberattacks enabled by autonomous agents could overwhelm traditional defensive measures. Many recent AI misuse incidents already show threat actors attempting to leverage AI for reconnaissance, exploitation, and phishing.

Third, the study highlights the importance of realistic testing environments for evaluating AI risk. Synthetic benchmarks that fail to capture the true complexity of enterprise systems cannot accurately estimate AI capabilities or their implications. By placing both humans and agents into a live network, the researchers produced evidence that will be critical for shaping policy, safety frameworks, and regulatory oversight.

The research reinforces a dual-use reality. Offensive security tools can strengthen defense when used responsibly but also lower the barrier for sophisticated attacks if misused. The authors advocate for greater transparency, robust safety measures, and responsible open-source release strategies to ensure that defensive ecosystems benefit more than malicious actors.