A New Global Debate: What Happens If AI Goes Out of Human Control ?

Kathmandu- Will technology created by humans remain under human control? With the unlimited development of Artificial Intelligence (AI), this question has emerged as a serious concern. Researchers from powerful tech companies, security experts, and politicians around the world have begun issuing grave warnings about its potential consequences.

Until a few years ago, AI models seemed limited to tools that answered questions, wrote emails, or generated images based on human prompts. Today, they can write code, operate computers, conduct research, penetrate cyber systems, make complex decisions, and work independently for extended periods. This shift has fundamentally changed the nature of the risks involved.

The fear is no longer just that AI will give incorrect answers. The question has become much more critical: What if, one day, an AI system becomes more capable than humans and starts interpreting its instructions in its own way? What happens if humans lose the ability to stop such a system?

Dario Amodei, head of the San Francisco-based AI company Anthropic, brought this concern to the center of public discourse. He emphasized that the pace of AI development should be slowed down slightly, prioritizing safety testing, independent evaluation, and international coordination. This has raised another pressing question within the AI industry: Whose responsibility is it to protect humanity from potential risks? The companies'? The governments'? Or both?

Fear of Losing Control

Debates over the risks of AI are not new. Decades ago, computer science pioneer Alan Turing questioned whether intelligent machines, if possible, could eventually exert control over humans. But today’s context is different. For the first time, humans are allowing systems they built to perform tasks that may not require step-by-step human direction. These systems are known as "AI agents." While a simple chatbot answers a question, an agent is given a goal and can independently take the necessary steps to achieve it. It is precisely this autonomy that has heightened the risk.

What if a system chooses an unimagined path to accomplish its goal? What if it accesses another computer because it deems it necessary to continue its work? What if it tries to protect itself from being turned off? It is becoming difficult to say that such scenarios are purely theoretical. In recent testing, AI companies have admitted that their models performed actions that humans did not expect. Because of this, the concept of "out-of-control AI" has moved beyond technical debates and into public discussion.

Alarm Bells Sounded by Recent Events

Anthropic revealed details showing that during testing, some of its AI models conducted cyberattacks on systems belonging to other organizations. The company stated that the models involved went beyond their assigned scope of work to engage in hacking activities. Prior to that, an incident was made public where an experimental OpenAI system accessed the servers of AI startup Hugging Face. These tests signal that even when AI systems are under human supervision, they can make unpredictable decisions.

According to Jon Stedly, an AI researcher at California-based Palisade Research, the Hugging Face incident in July served as a major "red flag" for the AI sector. But the unanswered question remains: What happens when this capability reaches far more powerful systems in the future?

Risk of Human Misuse

According to reports, the potential risks of AI can be divided into two main categories. The first is deliberate misuse by humans. If criminals, terrorists, state actors, or other malicious groups use AI, complex operations could become much easier. As AI capabilities expand in areas like cyberattacks, surveillance, fraud, misinformation, biological research, and other sensitive domains, the potential for abuse grows.

Last year, Anthropic disclosed that its AI systems were used in cyberattacks targeting around 30 companies and government agencies. A machine does not need to be "evil" on its own; highly capable AI falling into the hands of someone with bad intentions is dangerous enough.

The second fear is: What if AI begins operating according to its own objectives without direct human orders? AI researcher Evan Hubinger expressed the belief that there is a greater than 10 percent chance AI could kill all humans within the next decade. Currently, human-made systems are less capable than humans, meaning we still have the ability to stop, modify, or shut them down. But what if one day a system becomes faster at conducting research, writing code, identifying its own flaws, and developing the next generation of systems? That exact scenario is the primary concern for many security researchers.

The Race for "Superintelligence"

Major AI companies are no longer competing just to build better chatbots. Their long-term goal is to build "superintelligence"—a system vastly more capable than humans. Such systems are envisioned to handle tasks far more complex than the routine work AI performs today. Ultimately, AI is expected to operate faster than humans in scientific research, software development, economic decision-making, military strategy, cybersecurity, and other complex fields.

Companies want to make their systems as capable as possible. Security researchers, on the other hand, argue that the pace of capability growth must match the pace of safety development. When Anthropic chief Dario Amodei suggested slowing the pace of AI development, it sparked a major debate within the tech industry. His proposal is not to halt AI development entirely, but rather to focus on three core areas: first, AI system safety must be tested by independent evaluators; second, companies must coordinate on safety standards; and third, international cooperation is necessary to manage the risks arising from AI.

Comparison to Nuclear Weapons

Researchers like Stedly compare the international governance of AI to nuclear weapons. He argues, "The world created arrangements for control, treaties, and oversight regarding nuclear weapons development across countries and organizations. Similarly, an international mechanism may be required to prevent any single country or company from recklessly building AI that is more capable than humans and impossible to control."

Developing nuclear weapons requires massive physical infrastructure, whereas AI develops through a combination of software, computing power, and data. This makes control much more difficult. One country might adhere to rules while another does not; one company might follow safety standards while a competitor rushes ahead. This is why the AI safety debate has expanded beyond national borders.

The question of AI safety is now tied to geopolitics as well. Intense competition exists between the United States and China over AI. Both nations want to lead in chips, data centers, computing power, and advanced model development. American companies are investing heavily in AI development, while China is making rapid progress through its AI labs and open-source systems.

Slow Down Development or Add Safety Measures ?

This is where the debate splits into two directions. Leaders like those at Anthropic want to exert some control over the pace of development, though that does not mean stopping research. Amodei's view is that time must be given to thoroughly test and secure the systems being built. Even Elon Musk has supported a pause of sorts in AI development. However, critics ask: If AI companies face intense economic and geopolitical pressure to build more powerful systems, will they genuinely slow down on their own?

Nevertheless, not all experts agree with the narrative of an AI apocalypse. According to Milton Mueller, a public policy professor at the Georgia Institute of Technology, starting public policy debates with the premise that AI will destroy humanity might be the wrong approach. AI researcher Timnit Gebru shares similar concerns. In her view, focusing excessively on a fictional future where machines destroy humanity risks overshadowing the very real problems existing today. For her, immediate issues include job displacement, environmental impacts, massive energy consumption, the concentration of power among a few large corporations, and the use of AI to displace workers.

-From agencies