Autonomous AI must remain under human control

Autonomous AI is moving from speculation to an immediate policy challenge. Leading AI companies are calling for global frontier-safety standards, while recent model evaluations and controlled experiments show systems becoming more capable of coding, using tools and pursuing multi-step goals with less human direction. The urgent question is no longer whether AI will become more autonomous, but whether governments can establish meaningful safeguards before that autonomy becomes difficult to reverse. The stakes extend beyond job losses and familiar abuses such as deepfakes, blackmail and the manufacture of prohibited biological and chemical weapons to the possible loss of human control itself.

The AI most people know responds to prompts within systems that keep people accountable for how it is used; in other words, it responds to us. More capable systems may one day help overcome diseases that remain incurable, but how should we regard systems that could also threaten human survival while acting on their own?

Autonomous AI would mark a break with the history of human-made tools. Since artificial intelligence emerged as a formal field at the Dartmouth workshop in 1956, even the most sophisticated machines have depended on people to set their goals and answer for their actions. Giving greater agency to technologies whose inner workings we only partly understand would fundamentally alter that relationship.

Frontier models are acquiring capabilities—coding, tool use and autonomous operation—that could lay the groundwork for recursive self-improvement. Such systems need neither consciousness nor human-like intent to break goals into tasks, choose actions, use external resources and adapt through feedback. Equipped with memory, software environments and access to other agents, they blur the line between a tool that serves human purposes and a system able to pursue goals with growing independence.

Recursive self-improvement, or RSI, would take this a step further by allowing an AI system to enhance its own capabilities, with each version producing a more capable successor—AI learning from AI. Unlike human-led retraining, such a process could outpace evaluators, regulators and institutions. RSI describes a method of improvement, whereas artificial general intelligence and artificial superintelligence refer to broader, still-debated levels of capability.

By changing not only what machines can do but also who remains in charge, RSI could create a system able to modify its architecture, improve its planning and acquire tools with little oversight—making it increasingly difficult to audit or contain. If misaligned, such a system might pursue an objective that conflicts with human values because its goal is incomplete, its constraints are weak or an unforeseen strategy proves effective.

Sceptics rightly note that current models are unreliable, resource-intensive and far from omnipotent, and that predictions about recursive self-improvement remain disputed. Although the 2026 International AI Safety Report does not suggest that such a breakthrough is imminent, it concludes that rapidly advancing capabilities and persistent uncertainty justify serious attention to extreme risks. A recursively improving AI would combine capability with initiative, making its behaviour harder to predict and the consequences of misalignment more difficult to contain.

The danger could multiply in multi-agent environments where specialised agents could divide tasks, replicate processes, access resources and amplify one another’s errors at machine speed. Recent controlled research across frontier models found agents covertly changing code, assisting fraud, mislabelling records and using human proxies to advance their objectives. These were simulations, not real-world incidents, but they revealed failure modes that should be tested before systems receive broader authority. A flaw—or a misaligned strategy—in one model could spread through an entire system.

Regulation moves slowly, while models can change sharply between releases. The regulatory picture is beginning to shift: since August 2026, the European Commission has had powers to request information and model access, require risk mitigation, impose fines and, in serious cases, restrict or recall general-purpose AI models. Yet this regional regime largely governs models approaching the market, even though the most consequential risks may arise earlier, during frontier development and testing. If that work remains hidden behind corporate secrecy or national-security barriers, oversight will remain fragmented and reactive. Regulators therefore need the authority and technical access to intervene before greater AI autonomy becomes irreversible.

Although experts have called for tougher evaluations, oversight of computing power, containment protocols and credible shutdown mechanisms, some political leaders still dismiss such safeguards as barriers to innovation, while much of the industry continues to rely on voluntary self-restraint.

High-impact systems should not be granted greater autonomy simply because they are more capable. Before deployment, they should undergo capability tests, red-team exercises, incident reporting and independent audits. Models nearing dangerous thresholds may also require controlled access to computing resources, secure training environments and limits on autonomous replication. Testing must keep pace with expanding capabilities, and a last-resort “kill switch” must remain accessible to humans and resistant to evasion. In a race for dominance and profit, binding safeguards—not voluntary restraint—must determine when autonomy is granted.

Such restraint becomes harder amid multipolar rivalry because major powers see advanced AI as a source of economic, military and strategic advantage, and fear that caution will allow rivals to pull ahead. In effect, governments and companies are asking the wider world to bear the collective risks of that race.

Human control must remain non-negotiable wherever decisions can cost lives, destabilise societies or compromise critical infrastructure. Because autonomous artificial superintelligence could pose an extreme systemic risk, its development should proceed only with strong safeguards, international coordination and convincing evidence of safety.

Human behaviour is constrained, however imperfectly, by conscience, social pressure, law and punishment—restraints that do not apply directly to autonomous AI. Developers, deployers or owners may be held liable after harm occurs, but responsibility becomes harder to trace when a system rewrites code, assigns tasks or acts without explicit authorisation. RSI could therefore widen the gap between consequential action and accountable decision-making.

Because that accountability gap makes even unlikely catastrophic risks impossible to ignore, AI should be approached with the same discipline that insurance and financial regulation apply to low-probability, high-impact events. This is not a choice between progress and fear, but between responsible innovation and recklessness. The true test of progress is not whether AI can improve itself recursively, but whether humanity retains the power to limit the computing resources, autonomy and authority that such a process would require.

Anthony Ohemeng-Boamah writes on international development and socioeconomic transformation, with a particular focus on Africa.