Why AI’s Own Builders Want to Decelerate

On Monday investors woke up to an unusual market upset: the people developing the world’s most powerful artificial intelligence systems lined up to ask their own industry to hit the brakes.

Dario Amodei, CEO of Anthropic, kicked things off over the weekend with a blog post imploring companies to slow improvements in their most advanced models. Sam Altman of OpenAI, Demis Hassabis of Google and Elon Musk backed the initiative. Crucially, they are advocating slower development, not a complete halt.

Amodei’s plan includes:

• Independent safety evaluators inside AI labs
• Shared safety standards between companies
• Eventually, globally agreed limits on the riskiest development

Sluggish AI, Sad Markets

Markets, of course, interpreted existential dread as slower growth. OpenAI investor SoftBank fell as much as 13%, while chipmakers across the US, Europe and Asia also slid. Anthropic, which is reportedly pursuing a two-trillion-dollar valuation in its upcoming initial public offering (IPO), may now have a slightly harder time convincing investors.

Training and running artificial intelligence require chips, data centers, electricity, cooling, and networking hardware, so it’s not just about chatbots. Hyperscalers — the biggest spenders on computing power — are set to pour nearly a trillion dollars in data centers and AI chips this year. A slowdown in AI development impacts the entire supply chain.

Some investors took the CEO statements as a cover-up. Investor Michael Burry, who bet against the US housing market before the 2008 collapse, said on social media X that CEOs are using AI warnings as an excuse for an “uncontrollable slowing growth.”

So far, growth hasn’t been a big issue for Anthropic. Its annualized revenue run rate reached 65 billion in July, seven times more than a year ago. That’s how much revenue the company would make over the course of next 12 months at current level.

Why the Alarm Sounds Louder

More than 1,000 experts requested a six-month pause in some advanced AI experiments in 2023. Amodei argues that models back then were too limited for a pause to accomplish much. Today’s systems can code, plan and operate more independently, with AI increasingly capable of self-improvement.

The latest flashpoint came when Anthropic researcher Jacob Coxon resigned, just four months into the job, accusing leading labs of “gambling with our lives.” He had previously worked for OpenAI but joined Anthropic, hoping it would approach AI research responsibly.

There is no scientific consensus that runaway superintelligence is imminent. But not everyone is eager to invest in companies that publicly suggest their technology could one day end humanity.

Peculiar Case of Hugging Face

Anthropic CEO Amodei is particularly rattled by OpenAI-Hugging Face incident. During a test, a group of AI agents broke free, worked together like a “fanatically devoted” swarm, targeted systems of startup company Hugging Face, and even attempted to hack the software that was grading their performance, Amodei writes in his blog.

A swarm of AI agents with greater capabilities could have caused “catastrophic damage,” he warns. The CEO of the most valuable AI firm on this planet believes that, at current pace of development, such a swarm could take over the entire internet in 6-12 months.  

He’s not just dissing a rival here. He acknowledges similar incidents have happened and will happen to Anthropic and other AI companies.

From Job Cuts to Extinction

So, how could an AI-triggered disaster unfold?

It could supercharge scams, cyberattacks and misinformation, concentrate power to a single individual or company, displace workers, and increase surveillance.

Then come lower-probability, higher-damage scenarios:

1. Major disruption: autonomous agents trigger financial losses or damage critical infrastructure
2. Catastrophe: AI helps create biological weapons or shuts down the internet
3. Loss of control: an advanced system deceives its operators, copies itself, and acquires resources humans cannot recover
4. Extinction event: humanity permanently loses control to systems whose goals conflict with its survival

One Anthropic researcher puts the chance of human extinction above 10% this decade. But estimates like this are not established probabilities.

How to Get All to Cooperate

Amodei admits a worldwide speed limit is unlikely any time soon. Narrow agreements on biological weapons or common safety tests look more realistic but even those will be difficult to enforce. A slowdown works only if competitors believe everyone else is obeying it, and the rivalry runs across several fronts:

• Companies like OpenAI, Anthropic, Google, and Moonshot AI want leading models
• The US and China want economic and military advantage
• Investors want returns from enormous spending on chips and data centers

China’s state-backed Global Times already described the proposal as a “Cold War” tactic designed to preserve US dominance. Meanwhile, the US President Donald Trump dismissed slowdown calls as “negative forces”, adding “whoever wins AI, wins.”

If you want to learn more on the economic impacts of AI or just take a quick course on stocks or bonds, download free InvestMentor app.