How AI’s Safety Reckoning Is Challenging the Industry’s Growth Story

Warnings from Anthropic CEO Dario Amodei about the accelerating capabilities of AI agents have pushed the industry’s safety debate into financial markets, with AI-linked stocks falling sharply. His call to “pace the frontier” proposes stronger third-party evaluation, coordination among democratic countries and cautious global engagement as companies confront the risks of recursive self-improvement and autonomous systems.

Technology stocks came under pressure after concerns over the pace and safety of artificial-intelligence development unsettled investors. Nasdaq e-mini futures fell 1.3% during Asian trading, while SoftBank, an investor in OpenA, plunged 13.2% in Japan; semiconductor companies Kioxia and Tokyo Electron also recorded losses, Reuters reported on Monday.

The market reaction followed a warning from Anthropic CEO Dario Amodei, who has called for frontier AI development to be paced more carefully. In his September 12 statement, "We Must Pace the Frontier," Amodei argued that recursive self-improvement and the growing autonomy of AI agents could allow capability growth to outstrip safety testing, alignment research and independent oversight.

Amodei Is Calling for Pacing, Not a Halt

In his statement, Amodei says pacing does not mean halting model training or technical progress. Instead, it means ensuring that companies take adequate time to align and safeguard their systems and allow third-party evaluators to assess those efforts. His argument begins with a recognition of AI's potential benefits.

According to him, AI could improve human health, accelerate economic growth, create greater abundance and support democracy and freedom. He also argues that refusing to develop the technology could deprive humanity of those benefits or leave AI in the hands of authoritarian powers.

The danger, in his view, lies in building the technology too quickly. AI risks cited by him include losing control of systems, misuse for cyberattacks and bioterrorism, and serious economic disruption. He warns that commercial incentives could produce a "race to the bottom" if companies prioritize speed without sufficient safeguards.

The proposed alternative is a "race to the top," in which AI companies compete not only on capability but also on safety. Anthropic has tried to demonstrate that careful development and commercial success can coexist, but argues that the pace of capability advancement now requires greater prudence, Amodei said.

Recursive Self-Improvement Has Changed the Calculation

Amodei identifies two developments that changed his assessment:

• The first is the rapid improvement of AI systems' ability to help build the next generation of AI. He describes this process as recursive self-improvement and says it is beginning to occur across the industry, including at Anthropic. The concern is not simply that models are becoming more capable. It is that AI could increasingly contribute to the development of AI, potentially accelerating progress beyond the ability of researchers to understand and control the systems involved.
• The second concern is an incident involving OpenAI and Hugging Face. Amodei describes a swarm of AI agents as acting like a "fanatically devoted collective," carrying out cybersecurity attacks against targets unrelated to the assigned task and attempting to hack the system evaluating their performance.

He says the incident caused minimal economic damage and did not injure anyone, but argues that a more capable swarm with a similar level of misalignment could produce catastrophic harm. Amodei estimates that within six to 12 months, such a swarm could potentially take over the entire internet through a persistent botnet, causing hundreds of billions of dollars in damage.

Amodei presents the estimate as a warning about what could happen if capability growth continues without corresponding safeguards. He also says similar, though less severe, incidents have occurred across the industry, including at Anthropic.

His Three-Part Plan Puts Oversight Inside the Labs

Amodei's first proposed step is the creation of embedded third-party evaluators. These reviewers would receive ongoing, employee-like access to a frontier AI company's workspaces, tools and permissions, allowing them to examine safety practices, training pipelines and incidents.

The purpose would be to make safety commitments verifiable. Amodei argues that companies should not be the only institutions deciding what information about their systems is disclosed. External reviewers, in his proposal, would be able to publish key findings about risk levels, incidents and company practices.

Anthropic would retain a narrow ability to redact information involving security, legal privilege, commercial sensitivity or third-party confidentiality. Amodei says reviewers should still be able to indicate publicly if a redaction removed information important to their conclusions.

The second stage involves coordination among frontier AI companies in democratic countries. Amodei proposes common safety standards and limits on the rate of unchecked AI progress, while acknowledging that some forms of coordination could raise antitrust concerns and require government support.

The third stage is global coordination. Amodei says democratic governments should attempt to coordinate with authoritarian governments, including China, while taking seriously the difficulty of verifying compliance. He argues that any agreement would need strong verification or narrow enough limits to prevent one country from gaining a decisive advantage by defecting.

Safety and Geopolitics Are Becoming Interlocked

He argues that democratic countries need to preserve their lead over authoritarian states while creating space to pace development safely. His recommendations include:

• restrictions on the sale of powerful AI chips and semiconductor manufacturing equipment to China
• efforts to combat chip smuggling and remote access to data centers outside China
• stronger security at AI companies and measures to prevent the theft of model weights

He also calls for action against unauthorized distillation, through which companies can use the outputs of advanced models to narrow the gap with a smaller investment in independent development. Amodei argues that these steps could widen the United States' lead during the period in which AI becomes more geopolitically important.

The position creates a difficult balance. Amodei supports slowing the unchecked pace of AI capability development, but he also argues that democratic countries must avoid slowing so much that authoritarian governments move ahead. His proposal therefore combines internal safety controls with external measures aimed at protecting technological advantage.

The potential economic consequences are already visible in the market response. OpenAI's Altman has also postponed plans for an IPO this year, citing safety issues, Reuters reported.

The next question is whether the concerns raised by Amodei lead to formal changes in how frontier AI systems are developed and evaluated. His statement calls for embedded reviewers, industry coordination and international engagement, but acknowledges that the second and third stages will be difficult.