The United Nations on Monday started a landmark conference on a distinctly alarmed note, as it moves towards creating global AI rules that could affect vendors and users alike.

"We may be the last generation able to set the terms on which humanity and machines coexist," Secretary-General António Guterres told the opening of the Global Dialogue on AI Governance after highlighting "killer robots" as his chief concern.

Other UN leaders warned of a technology "shaped by a few," risks to critical infrastructure, and emotional attachment to AI models.

Many also spoke of the transformative promise AI holds – usually after dwelling on the risks.

The Global Dialogue is midpoint in a consultation process the UN started in January 2025. A panel of 40 experts convened in February 2026 to provide national policymakers with evidence around potential AI regulations.

The panel's initial findings were published last week and this week's Global Dialogue summit will discuss the report. Another conference will be held in May 2027 to move towards international regulation.

That is unprecedented speed for a UN system that appears to be galvanised by the speed of AI development. "The UN was built for moments like this," Annalena Baerbock, the president of the General Assembly, told those gathered.

Even when the regulations are finalised, The UN has very limited direct enforcement power. It typically serves as a forum for the negotiation of treaties that states then choose to ratify, and adopt in domestic law.

Don't vide-code humanity

Guterres has been pushing for a broad approach to AI for years, and on Monday celebrated what he called the first time that "every country has a seat at the table."

He warned of "runaway speed" in AI development, a power imbalance "hard-wired into technology," and machine-enabled lies that erode trust.

"We cannot vibe-code the future of humanity," he told some 4,000 delegates. 

His alarm was backed up by the scientific panel including the likes of DeepMind senior director of research Joelle Barral, SkillBench co-founder Juho Kim and 38 others. The panel, responsible for the report, steered clear of policy recommendations but called for urgent and courageous regulation.

See also: EU kicks AI Act's strictest regs down the road

"Please remember, intelligence gives power," the panel's co-chair, Canadian computer scientist Yoshua Bengio, told the conference. "As this power grows, it can unlock great benefits if we act wisely... but let's also be clear, this power can also lead to many perils if the decisions taken are reckless or if they are made to benefit a minority rather than all of humanity."

The panel pointed to already familiar concerns about AI: The Mythos-style risk to security; models that seem intent on deceiving humans trying to keep them in check; inequitable access to AI advancements; and humans forming emotional bonds with bots.

It also touched on issues to which enterprise users are likely more sensitive than policymakers, such as vendor lock-in and the complexity of ensuring genuine returns from AI investment.

Unlike many in the industry, however, those leading the conference showed little to no faith that vendors and users can self-regulate.

"Without standardized, rigorous, independent third-party assessment, similar to what exists for the pharmaceutical and aeronautical industries, assurance of safety largely depends on developer goodwill," the panel warned on testing safety validation in the report.

Agentic AI in the crosshairs

The UN's scientific experts are particularly concerned about agentic AI, and the scale and speed of its use that is increasingly making human-in-the-loop guardrails into a form of security theatre.

Guterres on Monday reiterated that humans must remain in control – especially in decisions made by weapons systems and beyond. 

"In every high-stakes decision, in justice, in healthcare, in policing, machines can inform, but humans must decide, and answer," he said. 

It remains to be seen if that is the consensus among member states, but there are indications that UN leaders will be pushing for the May 2027 meeting to start considering an international treaty on AI governance.

The link has been copied!