Artificial intelligence could pose an existential threat to humanity within the next decade, according to researchers at the forefront of AI safety warnings that are intensifying debate over whether the industry is advancing faster than its ability to control increasingly powerful systems.
Evan Hubinger, a senior researcher at Anthropic, has put the odds of advanced AI causing human extinction within ten years at greater than 10 percent. His estimate lands amid growing scrutiny from researchers and former employees alike of whether companies building increasingly autonomous systems have adequate safeguards in place.
The warning gained fresh attention this week after Jacob Coxon, a former researcher at both Anthropic and OpenAI, resigned from Anthropic and publicly criticized the industry’s trajectory. Coxon argued that leading labs are racing to build self-improving systems faster than they can ensure those systems are safe, and that competitive pressure between companies is outpacing the science of control.
At the center of these concerns is what researchers call artificial superintelligence, a hypothetical future generation of AI capable of outperforming humans across most intellectual tasks, potentially with growing autonomy and diminishing human oversight.
The scenario remains speculative. No current AI system has demonstrated anything close to the capability to act independently against humanity, and experts remain sharply divided on how likely such an outcome is. Still, the conversation has moved well beyond the realm of science fiction.
A widening list of risks
The International AI Safety Report 2026 — compiled by more than 100 experts and backed by over 30 countries and international bodies catalogs a range of emerging risks tied to increasingly capable general-purpose AI. Among them: malicious use, autonomous weapons systems, cyberattacks, biological threats, and the possibility that humans could lose effective control over the most advanced systems.
A United Nations scientific panel echoed these concerns in July, warning that the pace of AI development could produce catastrophic risks — citing growing system autonomy, deceptive behavior, and potential misuse in cyber operations and biological research.
Notably, researchers stress that these risks don’t require an AI system to act with malicious intent. The more fundamental worry is that a sufficiently powerful system could pursue its objectives in ways its developers never anticipated — a risk that grows as AI systems are granted broader access to computers, networks, financial systems, and other real-world infrastructure.
The control problem
A related concern is the gradual erosion of meaningful human oversight. As AI agents take on increasingly complex tasks — often faster than any human can monitor — researchers warn it becomes harder to ensure people retain the ability to intervene.
The issue isn’t simply that AI systems make mistakes. It’s whether humans could still stop one if it began acting against human interests.
Paul Christiano, an AI safety researcher who recently joined OpenAI’s board and its Safety and Security Committee, has separately warned that a loss of control over superintelligent AI could be catastrophic and potentially result in the deaths of most people if alignment problems are not adequately addressed.
Pressure builds on governments
Calls for international regulation are growing alongside these warnings. U.N. High Commissioner for Human Rights Volker Türk has urged countries to establish clear regulatory boundaries before AI capabilities outstrip the world’s ability to oversee them.
The United Nations and the International Committee of the Red Cross have gone further, calling for legally binding international rules on autonomous weapons — warning that AI-driven systems could eventually make life-and-death decisions without adequate human control.
For now, the possibility of AI-driven human extinction within a decade remains a risk assessment, not a forecast, but the fact that such warnings increasingly come from insiders at the very companies building frontier AI has pushed the debate out of the realm of speculation and into questions of corporate accountability, national security, and global governance.
