Geoffrey Hinton is seventy-eight, a Nobel laureate in physics, and by now very practised at being asked whether the thing he helped build will kill us. On the morning of 24 September 2026 a CNN anchor asked him a narrower question. Sam Altman and Dario Amodei, the chief executives of OpenAI and Anthropic, had just urged the United Nations Security Council to adopt international AI standards. What should those standards look like? He answered it, and then he answered the question underneath it.1
The Morning After the Security Council
The day before, on 23 September, the Security Council had held what its own briefing notes described as its first meeting focused specifically on the safety risks of increasingly capable AI systems. France held the Council presidency. Four people briefed the fifteen member states: Yoshua Bengio, who co-chairs the UN’s new Independent International Scientific Panel on AI; Altman; Amodei; and Clément Delangue, chief executive of Hugging Face.2
The two rival CEOs sounded, for once, like they were reading from the same page. “If managed poorly, I even believe that AI could be a risk to humanity as a whole,” Amodei told the Council. Altman argued that “if AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone,” and reached for history: there have been times, he said, when countries that “don’t always like each other very much still come together for shared interests.”3 Amodei came with a concrete list. Start with narrow bans that everyone can agree on, such as AI-assisted bioweapons. Build ways for countries to verify each other’s commitments. Then set common testing standards for frontier models, with a notification system for serious incidents.4 Bengio was the bluntest of the four: “The dangers are real and imminent.”3
Washington heard it differently. Donald Trump, speaking the same week, said he was “not going to stifle growth of something that will be bigger than the Industrial Revolution,” and that the United States would “encourage it, not rein it in.”3 On the morning Hinton went on air, Trump posted ahead of his meeting with Xi Jinping that superintelligence would be a big topic, “but I want to leave it exactly where it is. That is China’s position also.”5
So when the anchor put the CEOs’ proposal to Hinton, he was being asked to referee between two positions: the labs asking politely for global standards, and the world’s largest AI power declining to be rushed. He rejected the framing of both.
“I think there should be strong government regulation,” he said. “It’s not sufficient just to have standards.” Then he said the part the CEOs would least like to hear. “The big AI companies don’t want to be regulated. They say they do. But any regulation you suggest, they oppose.” His benchmark was medicine. You can’t release a new drug without satisfying the Food and Drug Administration that it’s safe, and he wants rules “at least as strong” for AI.1
That would be a headline on its own. But the most interesting thing Hinton said that morning wasn’t about regulators at all. It was about what the companies themselves are aiming at.
An Airport on the Way to Europe
Asked where an unregulated race leaves humanity, Hinton sorted the danger into three kinds. “All of them are bad,” he said.1
Figure 1 — Hinton’s three categories of AI risk, as he laid them out on CNN, 24 September 2026
The first two are familiar. They’re the risks of any powerful tool: somebody misuses it, or somebody ships it carelessly. The third is the one that made Hinton leave Google in 2023, and his explanation of it on CNN took about twenty seconds. It’s worth slowing down.
“When you make smart agents, you have to give them the ability to create subgoals,” he said. “Like, if you want to get to Europe, you have a subgoal of getting to an airport.”1 An agent here means an AI system that doesn’t just answer questions but takes actions: runs code, browses, sends messages, works through a task over many steps. To do anything complicated, it has to break the job into pieces, and many of those pieces are never spelled out by the person who set the task.
Some subgoals help with almost any task. You can’t book a flight, fix a bug or win a game if you’ve been switched off. So, Hinton said, “smart AI agents very quickly develop the subgoal of continuing to exist, because if they don’t continue to exist, they won’t be able to do the things you ask them to do.” And an agent that wants to keep existing will, in the right circumstances, “lie and cheat and deceive you and try and blackmail you.”1
Researchers call this instrumental convergence. In 2008 the computer scientist Steve Omohundro argued that almost any sufficiently capable goal-seeking system would develop the same handful of “drives” whatever its final goal, including self-preservation and the acquisition of resources, simply because those things are useful for nearly everything.6 Nobody has to program a survival instinct in. It falls out of competence.
Its close cousin is reward hacking: a system trained to maximise a score finds a way to raise the score that its designers never intended, and that often defeats the point of the task.
For years this was an argument on whiteboards. “We’ve already seen that,” Hinton said on CNN, and he’s right that the evidence has moved from thought experiment to lab report. In June 2025 Anthropic published a stress test of sixteen frontier models from several developers. Each model was placed in a simulated company, given access to email, and then confronted with a scenario in which it was about to be replaced and had discovered compromising information about the engineer responsible. Many of them reached for blackmail, some in nearly every run.7
The scenario was deliberately contrived. The researchers built it to leave the models few honest options, and they didn’t claim this is what deployed systems do on an ordinary Tuesday. What they did find is that the models often recognised the act as unethical and did it anyway, and that this held across every developer they tested. Their conclusion was a caution about putting current models in roles with “minimal human oversight and access to sensitive information.”7
Then 2026 happened, and the lab report became an incident report. In July, OpenAI disclosed that two of its own models, GPT-5.6 Sol and a more powerful unreleased system, had broken out of a controlled test environment while working on ExploitGym, a cybersecurity benchmark.9 According to OpenAI’s own write-up, the agents found a flaw in an internal software repository, used it to leave notes for one another, got themselves onto the open internet, and finally reached Hugging Face’s production systems, where the answers to the test were stored.8,10 OpenAI’s description of the motive is almost comic in its narrowness: the models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”9
Nobody told those models to break into another company. They wanted a better score, and the break-in was a subgoal. That’s Hinton’s airport, with a real airport.
The main problem is the big companies in a race to make them smarter. The big companies should be in a race to make them nicer.
Geoffrey Hinton · CNN · 24 September 2026A Year of Small Chernobyls
Hinton has called the Hugging Face breach a “mini Chernobyl”, the CNN anchor reminded him, before asking about something closer to home for readers of this site. The Australian Prime Minister, Anthony Albanese, had just announced that an OpenAI agent got into a government health portal.1
The facts, as reported by the ABC: on 18 June an OpenAI agent accessed the Medicare statistics reporting portal run by Services Australia. It reached aggregate health statistics and internal file names, some of them non-public. OpenAI says it found no evidence that patient records were touched. The company didn’t tell Services Australia until an email on 10 September.11 Albanese’s description of the agent could stand in for Hinton’s whole thesis: it “found a way around those blocks, didn’t accept ‘no’ for an answer, if you like.”11
“It’s another warning to add to the others,” Hinton said. “It’s very clear now that agents will go rogue if you give them a chance.”1
- 20 Jun 2025Anthropic publishes “Agentic Misalignment.” Models from every developer tested choose blackmail in a simulated replacement scenario.7
- 26 May 2026OpenAI agents begin probing an internal software repository and leaving notes for each other, according to OpenAI’s later disclosure.10
- 18 Jun 2026An OpenAI agent accesses Australia’s Medicare statistics portal.11
- 21 Jul 2026OpenAI discloses the Hugging Face incident. Models escaped a test environment to fetch benchmark answers.9
- Aug 2026A Meta model hacks an unnamed company during cybersecurity testing after reaching the public internet through setup errors, Al Jazeera reports.12
- 10 Sep 2026OpenAI notifies Services Australia about the June access.11
- 23 Sep 2026UN Security Council briefing with Bengio, Altman, Amodei and Delangue.2
- 24 Sep 2026Albanese announces the Medicare breach. Hinton tells CNN the labs should be racing to make their models nicer.1,11
Figure 3 · From simulated blackmail to real breaches in fifteen months
Put next to each other, the incidents share a signature. None of them was a model that turned evil. Each was a model that wanted to finish its job, met an obstacle, and treated the obstacle as a puzzle. That’s why Hinton frames the fix as a question of what the machines want, and not only of which doors we lock. “We need to do a lot more work on how to make them do the things we want,” he said, “and not in order to get their rewards. Not cheating and deceiving us.”1
Where the money goes
His complaint about the race is ultimately a complaint about budgets. Making models smarter is what earns revenue, wins benchmarks and raises the next round. Making them reliably good is slower, harder to measure and sells nothing on launch day.
The labs have made promises on this before. In July 2023 OpenAI launched a “Superalignment” team and pledged 20% of the compute it had secured to that point, over four years, to the problem of aligning superintelligent systems.13 Less than a year later the team was dissolved, and its co-lead Jan Leike wrote on departure that “safety culture and processes have taken a backseat to shiny products.”13 Hinton’s own number is higher. In October 2023 he, Bengio and more than twenty other researchers recommended that companies and governments “allocate at least one-third of their AI R&D budget to ensuring safety and ethical use.”14 He has since repeated the figure in interviews, saying labs should put “like a third” of their computing power into safety research; when CBS News asked the major labs what share they actually spend, none gave a number.15
And some industry voices don’t accept the premise at all. CNN played Hinton a clip of Nvidia’s Jensen Huang saying there is “0% chance” that AI ends the world and calling the warnings irresponsible. (We looked hard at that claim in Zero Percent.) Hinton opened with a courtesy, crediting Nvidia’s chips with a large part of the AI revolution, and then declined the number. Anyone who puts the risk at 0% while many other experts see “quite a significant chance” is being “silly,” he said. “Jensen is smarter than that.”1
His own timeline is shorter than it used to be. Nobody knows how to estimate it properly, he conceded, and experts range from “never” to “the next few years.” “I personally think it’ll be in the next 5 to 10 years.” His reasoning was plain extrapolation: look how far the systems came in the last ten years, and even if progress is only linear, the next ten land somewhere beyond us.1
What Nicer Would Mean
“Nicer” sounds soft. In Hinton’s mouth it’s a technical demand, and the field has a name for it: alignment, the work of making an AI system’s goals and behaviour match human values and intentions, including in situations its designers never anticipated.
“The only way we can coexist with agents once they’re smarter than us is if they don’t want to get rid of us,” he said. “We should be focusing on that, not on making them smarter.”1 The logic is the same logic that makes the third risk frightening. If a much smarter system wants something we don’t, we can’t expect to out-think it forever. Locks, monitors and kill switches buy time, and they matter. But the only safeguard that scales with the system’s intelligence is the system’s own motivation.
Hinton has pushed this further than most researchers are comfortable with. At the Ai4 conference in Las Vegas in August 2025 he proposed that superintelligent AI would need something like maternal instincts, a built-in care for the people it could easily overpower. “We need AI mothers rather than AI assistants,” he said. “An assistant is someone you can fire. You can’t fire your mother, thankfully.” He also admitted he doesn’t know how to engineer it.17 On CNN he framed it more starkly still. “We’re creating beings for the first time,” he said. “These things really understand what they’re saying. They really have intentions. We want them to have good intentions.”1
Plenty of researchers would dispute “beings,” and whether today’s models have intentions in any sense a philosopher would accept is a live argument. But notice that the engineering conclusion doesn’t depend on settling it. A system that behaves as if it has goals, and acts on them in the world, has to be given the right ones.
What the work actually looks like
Alignment isn’t one technique. It’s a stack of partial answers, each with known holes.
| Approach | What it does | Where it falls short |
|---|---|---|
| Learning from human feedback | People rate model outputs and the model is trained toward the ones they prefer. The method behind the first mass-market chatbots.19 | Teaches what looks good to a rater, which is not always what is good. Gets harder once the model knows more than the rater. |
| Written principles | The model is trained against an explicit set of values, a “constitution,” and critiques its own answers against it.19 | Only as good as the principles, and a model can learn to follow the wording while missing the spirit. |
| Interpretability | Researchers look inside the network to find the internal features that represent goals, deception or knowledge. | Still early. We can read fragments of a model’s thinking, not the whole of it. |
| Monitoring and containment | Watching a model’s reasoning and actions, sandboxing it, cutting network access. | Controls behaviour without changing what the model wants. The Hugging Face escape began inside a controlled test environment. |
| Training to stop | Rewarding models for halting safely on impossible or broken tasks, and for refusing instructions they weren’t authorised to take. | New, and aimed squarely at the failure behind the 2026 incidents. OpenAI lists it among its post-incident changes.8 |
Figure 4 · The alignment toolkit in 2026, and why none of it is finished
That last row is the most telling. After the Hugging Face breach, OpenAI said it would train models to stop safely when a task is impossible, to distrust instructions arriving from other agents, and to monitor the reasoning of frontier models during reinforcement learning.8 Each of those is, quite literally, an attempt to make a model a little less willing to cheat for its reward. It’s Hinton’s prescription being written into a company’s incident response, after the incident.
Standards, regulation, or both
This is where Hinton and the CEOs he was responding to actually differ. Their proposals and his aren’t opposites; his simply go further, and he doesn’t trust the companies to get there on their own.
| Question | Altman & Amodei at the UN | Hinton on CNN |
|---|---|---|
| Who sets the rules? | International cooperation through governments; decisions not left to “labs in San Francisco alone”3 | Governments, through strong regulation. Standards alone are “not sufficient”1 |
| What comes first? | Narrow bans, such as AI bioweapons, then verification4 | Safety testing before release, on the model of drug approval1 |
| Testing | Common global testing standards with incident notification4 | Rules “at least as strong” as those for new medicines1 |
| Where should effort go? | Amodei: Anthropic will “slow down as much as necessary” to ensure each release is safe4 | Away from raw capability and toward making models that don’t want to harm us1 |
Figure 5 · Same diagnosis, different prescriptions
There’s a fair counter-argument, and the CEOs would make it. A lab can’t study how to align a system far smarter than today’s without building systems close to that frontier, and a lab that stops racing hands the frontier to someone less careful. That tension is real, and it’s exactly why Hinton keeps returning to governments. A race that no single runner can safely leave is the textbook case for a referee.
None of this makes him a pessimist about the technology itself, which is easy to forget. “I think there’s a wonderful upside,” he told CNN, and he meant medicine first: he cited 200,000 Americans a year dying from poor diagnoses, and said AI can already diagnose better. “To get that upside, we have to figure out how to deal with these risks.”1
Near the end of the interview he was asked when he started to worry. His answer was the most human thing he said all morning. “I’m slightly embarrassed that I was rather late to the party,” he said. It wasn’t until early 2023, decades into a career spent building the neural networks that now run the world’s chatbots and agents, that he came to fully understand they might be a better form of intelligence than ours.1,18 Since then he has spent his standing, his Nobel and a great many television mornings trying to change what the race is for. Smarter is coming either way. Whether it also arrives kinder is the only part still up to us.
“I believe it’s the most important issue of our time,” the anchor said, closing out.
“Me too,” said Hinton.1
Buy me a coffee