Researcher leaves Anthropic, condemns race to superintelligence

Summary

Jacob Coxon, a researcher who says he spent the past three years doing pre-training research at OpenAI and Anthropic, has resigned from Anthropic and issued a public warning about the direction taken by major AI labs. In a thread posted on X, he accused both companies of acting irresponsibly and "racing straight" towards self-improving superintelligence, calling that race a gamble with everyone’s lives.

Coxon does not frame the concern as distant or abstract. He says the people building these systems earnestly believe AI could kill all humans by the end of the decade. He adds that many senior executives and researchers soften that language in public while expressing similar fears privately.

In practice

The core of the warning is recursive self-improvement: an AI system helps build the next generation of AI systems, which then builds an even more capable generation. Coxon argues that this cycle could create superhuman systems in areas such as cybersecurity, science and resource acquisition without a rigorous understanding of how they work internally or a credible way to stop them.

Anthropic acknowledges in its August risk report that its models already accelerate internal research and engineering work. The report says the company does not yet believe it has reached its threshold for high-risk automated research and development, but it also notes greater uncertainty in its assessments and early signs of acceleration. Coxon argues that this trajectory requires a firmer response before capabilities advance further.

Context

In the thread, Coxon draws a distinction between the two companies. He writes that many people at OpenAI have not fully internalised the civilisational stakes. At Anthropic, he says, the risks are understood, but the company is locked into a race to arrive first because it believes competitors will not act responsibly.

Coxon argues for agreements between laboratories to control the pace of development and says preventing a global race may require costly action, including a temporary ban on improving model capabilities. He also urges AI lab researchers to ask whether they should begin superintelligent reinforcement-learning runs without a rigorous understanding of their systems’ "minds."

Why it matters

• The debate is no longer only about misuse of current models; it also concerns how fast AI itself could accelerate AI research.
• The warning comes from someone who worked at two of the industry’s most influential labs and directly challenges the competitive incentives shaping both.
• Discussion of superintelligence and control is moving from research into politics, with recent legislative proposals in the United States and the United Kingdom.