Dario Amodei has published a lengthy post arguing that frontier AI development must be paced. There are already plenty of people disagreeing with the proposal he makes, and many are speculating about ulterior motives. Personally, I simply find the argument deeply unconvincing.

Bias alert: I’ve been working in the clinical IT space for over 15 years, including more than a decade in environments where advanced AI systems are developed to assist with clinical diagnosis or, in some cases, make diagnoses directly. I’m entirely sympathetic to arguments around risk management, safety engineering, validation, and objective performance measures.

The basic thrust of Amodei’s argument is this:

“Frontier AI deployment is becoming harder to control, so advanced systems should face stronger operational controls, security, independent scrutiny and release conditions.”

I agree with that to an extent, although in many ways it feels like a simple response should simply be, “don’t give AI control of things it’s not capable of using safely”. Don’t let ChatGPT drive your car.

What strikes me about Amodei’s argument is how anthropomorphic it is. The advanced AIs in question, primarily large language models, are not independent actors. When we talk about “agentic AI”, what we’re usually talking about is the interfaces we provide to these systems, not the underlying models themselves.

It is true that the steps a frontier AI may take to achieve a goal are becoming increasingly sophisticated. It can be difficult to steer these systems away from certain objectives. They have little ability to evaluate the broader “goodness” or “badness” of an approach, and because they have been trained on examples of both defensive and offensive techniques, they will often reach for the latter when the former prove ineffective.

However, Amodei then presents a series of actions that he claims will improve the situation. On closer inspection, many of them simply do not make sense. The systems he asks us to control are entirely under our control already: the problem is that we’re giving them free rein.

We must slow down progress to preserve the democratic world’s lead

This is the clearest, and arguably most nonsensical, argument he advances.

I’m going to leave aside the distinction between democratic and authoritarian systems, except to say that I broadly agree with the characterisation. The problem is the claim that pacing progress will somehow preserve a technical advantage. That simply does not follow.

Worse, throughout most of the essay Amodei seems to acknowledge that the proposal is impractical anyway. Authoritarian states may not comply, and he provides no compelling reason why they would. The regulatory mechanisms he advocates move slowly and would require a degree of international coordination that appears highly unlikely.

Beyond that, the central issue is that he is calling on Western governments to act. The government of primary concern here is the US government, and at best it has an “interesting” relationship with Anthropic, despite Anthropic being a US company.

For example:

  • Anthropic has repeatedly attempted to impose non-defence restrictions on the use of its models.
  • The US Department of Defense has designated Anthropic a supply-chain risk.
  • Ongoing legal disputes on First Amendment grounds have alleged unlawful retaliation against the company.

More broadly, the Republican Party, which currently controls the federal government, has repeatedly blocked state-level attempts to regulate AI development. While there are certainly voices within the party who are concerned about AI risks, the White House itself has shown little enthusiasm for regulating the sector.

Amodei knows all of this. That makes it particularly intriguing that so much of his essay is effectively a request for this administration to take an international leadership role in implementing restrictions that it has repeatedly indicated it does not support.

We must restrict supply-chain technology

Amodei proposes three primary constraints:

  • Do not supply powerful AI chips to China.
  • Do not allow China to distil US models.
  • Prevent industrial espionage.

He does not explicitly mention China in all three cases, but the desired outcome is clear: slow China’s progress. Given that China is currently the most significant AI developer outside the democratic world, that is at least a coherent objective.

Industrial espionage is already illegal, so it’s not entirely clear what additional measures he has in mind. That proposal is unlikely to be controversial.

Distillation is a different matter. I don’t think the term is especially helpful because it obscures what is actually happening. A better shorthand might simply be “broad use”. Distilling a model ultimately requires large volumes of prompts and responses. Realistically, I struggle to see any effective control mechanism short of completely banning Chinese access to frontier systems.

As for AI chips, export restrictions have existed for years.

I have two problems with this argument. First, Amodei is advocating measures that are already largely in place.

Second, he assumes that additional restrictions will slow Chinese progress over the long term. That assumption is questionable. Constraints often encourage innovation. They may create a short-term bottleneck, but they also create powerful incentives to solve the underlying problem.

This is basic Theory of Constraints thinking:

“Constraints, whether imposed by time, resources, rules, or even self-imposed boundaries, can ignite a surprising burst of creativity, driving us to find solutions that are both innovative and unexpected.”

From ThoughtLab’s discussion of constraint-driven innovation.

I’m fairly sure Amodei understands this dynamic. Yet he never discusses it. He assumes the constraint will work, and continue working, without also evaluating the likelihood that it becomes the focus of future innovation.

A useful counterpoint is this CSIS analysis of Chinese AI development, which highlights China’s heavy emphasis on efficiency, distillation techniques, recursive self-improvement, and domestic compute infrastructure designed specifically to work around export restrictions.

Chip smuggling certainly exists, but Chinese progress has continued despite years of export controls. Attempting to ban distillation may prove equally ineffective. To the extent one avenue is restricted, Chinese developers will simply pursue another.

The real lesson here is that there is no durable technical moat around AI models or model designs.

China’s strategy of broad distribution and open-weight releases directly undermines attempts to maintain pricing power through exclusivity. And I think this is the real concern here: the debate is less about slowing AI and more about protecting the valuations of incumbent US AI companies. It was one thing when China was commoditising consumer goods, toys, textiles, or basic manufacturing. Commoditising intelligence is an entirely different proposition.

Embedded evaluators

This is the proposal I find most technically problematic.

Amodei argues that alignment and safety would be significantly improved through the use of independent evaluators with deep internal access, including contracts with providers and the ability to see the work going on. In effect, he is proposing something akin to the International Atomic Energy Agency for AI: inspectors embedded within organisations who can identify dangerous developments and raise concerns.

Even if such an arrangement were practical, and I’m not convinced that it is, Anthropic is not currently behaving as though it believes in this principle.

Claude Mythos 5.1 was not provided to the UK’s AI Safety Institute (AISI) for pre-release evaluation, despite the evaluation framework already existing and having been used successfully for previous models.

The Responsible AI Institute has an interesting article on this Anthropic refusal, noting:

“The Financial Times alluded to pressure from the Trump Administration to adopt a more protectionist stance on AI products.”

It is entirely plausible that the US government views foreign access to frontier models as a national-security concern. However, that does not undermine my earlier point about the administration’s limited appetite for regulation. The same article continues:

“The idea of robust assessments carried out by a foreign government simply do not align with an increasingly robust America First position, nor with the realities of national security asset classification, which among other restrictions also places nationality caveats on engineers participating in US defence and national security contracts. Anthropic is unlikely to be the last major US AI provider to downgrade its open relationship with AISI as a result.”

In other words, the US may very well be unwilling to allow foreign governments access to frontier systems. It may even become unwilling to grant meaningful access to independent evaluators.

That is partly because the United States is not trying to slow progress. In fact, many influential policymakers remain deeply sceptical of guardrails, particularly where military or national-security applications are concerned.

At my most cynical, I see this proposal less as a mechanism for improving safety and more as a mechanism for redistributing liability. In a world with embedded evaluators, Amodei can reasonably argue that responsibility for safe deployment does not rest solely with Anthropic. If a dangerous system is released, responsibility becomes shared between the company and the body tasked with independently validating its safety.

What will happen?

My suspicion is that very little of what Amodei proposes will actually happen.

The US government appears largely uninterested in slowing frontier AI development. China has little incentive to participate in arrangements that would constrain its ability to compete. Export controls have produced far less impact than advocates hoped, while compliance and verification become harder as AI systems grow more capable and more widely distributed.

What I expect instead is continued acceleration.

Safety research will improve. Evaluation methods will improve. Interpretability techniques will improve. Operational controls will improve. But all of those improvements will occur alongside rapidly advancing capabilities, not in place of them.

Most importantly, history suggests that technologies with overwhelming economic and strategic value are not slowed by appeals for caution alone. They are adopted, commercialised, copied, refined, and eventually commoditised.

This does not mean safety is unimportant; actually, the opposite. The correct response to powerful technology is rigorous engineering, better testing, stronger security, clearer accountability, and careful deployment. Those are all arguments Amodei makes, and in many cases they are persuasive.

What I find unpersuasive is the leap from “we need better safety practices” to “we must slow frontier progress”. In many cases, the two are equated (which isn’t really true) - but he also jumps from one to the other quickly. Slowing progress on its own does little to improve safety.

I find it fascinating how often “free market thinkers” resort to state solutions when the stakes are extremely high. I’m no libertarian by any stretch, but I do believe that groups of people perform better decision-making (Whether independent or not) than individuals often do. This is particularly true in democractic countries - even though, from time to time, it feels like electorates make significant errors. They are directionally better decisions, overall, than authoritarian regimes can make.

More oversight is absolutely required. More insight into what the models are becoming capable of is also important; my main worry these days is that the performance of models is significantly oversold compared to what they are capable of, and this is the primary safety concern I have with them.

Cybersecurity is back on the agenda, big time, and I’m glad of that. AI has raised the stakes, but let’s be honest: business sees security as a necessary evil, and we often seek to minimize the cost of security. It’s not a value-add product in most cases. We should be worried that most businesses are significantly vulnerable: but we should also acknowledge that this is not news.