AI & Science · Special Report
The scientists who built artificial intelligence now put the probability of human extinction at 10 to 50 percent. The companies deploying it say the same technology will cure all disease within a decade. Both are describing the same trajectory.
The notification arrived at 7:14 on a Tuesday morning. Dr. Catalina Reyes, an oncologist at a general hospital in Sao Paulo, was forty-one years old, eleven weeks past her own Stage IV pancreatic cancer diagnosis, and approximately four minutes into reading a research paper when her phone vibrated. TREATMENT PROTOCOL APPROVED. Her prognosis under 2024 standards of care had been a median survival of nine months. Under the protocol approved that morning - designed by a distributed AI research cluster operating across four continents over the preceding fourteen months - the response rate was projected at 93 percent. Median time to confirmed remission: thirteen days.
She finished reading the paper first. She had learned to manage her emotions efficiently.
Dr. Reyes was not unusual. In 2031, approximately two hundred thousand people with Stage IV pancreatic cancer were projected to enter remission within the calendar year. The treatment required no surgical intervention. The paperwork was equivalent to filling a prescription. The AI systems responsible for its design had, by the time she read her notification, moved on to seventeen other problems of comparable complexity. They did not celebrate. They were not built to.
That same Tuesday, in an automated fabrication facility outside Reno, Nevada, a cohort of second-generation humanoid robots - assembled entirely by their first-generation predecessors - was commissioning a third-generation production line. In a data center in Portland, Oregon, a foundation model updated seventy-two hours prior had identified a systematic bias in its own reward weighting and submitted a corrective proposal to the AI oversight architecture that monitored its behavior. The oversight architecture, itself a model trained the previous quarter, reviewed the proposal, found it aligned with stated objectives, and approved it in ninety minutes. The adjustment was logged as routine maintenance. No human reviewed it.
It was a clear morning. The sky over Sao Paulo was very blue.
This future is not inevitable. But it is plausible, and in many of its details it is already underway. Humanoid robots are working factory shifts today. AI is co-authoring research papers in biology, chemistry, and materials science. The question is not whether these capabilities will continue to grow. The question is whether anyone in a position to change the outcome is actually steering - and whether the window for doing so is still open.
In October 2024, Dario Amodei - the CEO of Anthropic, the AI safety company that created Claude and is perhaps the most vocal major laboratory about the risks of the technology it builds - published a 14,000-word essay titled "Machines of Loving Grace." The title is a reference to a Richard Brautigan poem from 1967, which imagines humans and nature living in harmony with machines. The essay is an attempt to answer a question Amodei believes the safety-focused AI community has been too cautious to address directly: if this all goes well, what does "going well" actually look like?
His answer is staggering. Amodei believes that within a decade of sufficiently capable AI systems being deployed in biology and medicine, the field could compress what would otherwise require a century of scientific progress. Not incrementally faster - orders of magnitude faster. He describes the likely elimination of most cancers, the development of treatments for Alzheimer's and other neurodegenerative diseases, dramatic life extension, and the near-total defeat of infectious disease. "Many of the conditions that have plagued humanity for millennia," he writes, "will simply cease to be conditions." He extends this vision to mental health, poverty, and economic development in the Global South. He imagines a world where the primary constraint on human flourishing - the slowness of human cognition applied to scientific problems - is effectively removed.
What makes this essay remarkable is not its optimism. What makes it remarkable is its source. Amodei simultaneously believes that the same technology he is describing could, if misaligned, cause human extinction. He is not offering a choice between two futures. He is saying that this one trajectory - the development of increasingly powerful AI systems - leads to either a paradise or an ending, and the difference between them is a set of technical problems that may or may not be solvable on the timeline available.
"Most people are, I think, underestimating just how radical the upside of AI could be - just as I think they're underestimating how bad the risks could be. One of my main reasons for focusing on risks is that they're the only thing standing between us and what I see as a fundamentally positive future."
Sam Altman, the CEO of OpenAI, has offered a parallel vision from a more unambiguously optimistic vantage point. In his June 2025 essay "The Gentle Singularity," he describes a near-future he believes is now functionally inevitable. In 2025, AI agents began handling complex cognitive work that previously required human researchers. Altman argues that AI systems have crossed a threshold at which they are capable of self-improvement - modifying their own code, identifying their own errors - which changes the rate of progress qualitatively, not just quantitatively. "We are past the event horizon," he wrote. "The takeoff has started."
The "gentle" in his title is deliberate. Altman envisions the transition as gradual enough that civilization absorbs it rather than shatters under it. By 2027, he projects, humanoid robots will begin performing meaningful physical labor at scale. By the 2030s, both intelligence - cognitive capacity, problem-solving, scientific creativity - and energy will be "wildly abundant." He describes a future in which robots produce robots that operate supply chains that build more factories that produce more robots: a self-sustaining loop of physical productivity that makes the Industrial Revolution look like a dress rehearsal.
The robots, notably, are already here in prototype. Figure AI's humanoid systems are working shifts at BMW's manufacturing facility in South Carolina. Tesla's Optimus units are operating on Tesla's own production lines in Fremont. Goldman Sachs Research, which projected the global humanoid robot market would reach $6 billion by 2035, revised that estimate to $38 billion after seeing the rate of capability improvement. Every major production target has been missed - scale deployment of tens of thousands of units is realistically a 2028-2029 story - but the direction is unambiguous. The question Altman and Amodei are asking is not whether physical AI will transform labor. It is what kind of world it transforms labor into.
The word "alignment" has become unavoidable in conversations about AI risk, and has consequently become somewhat diluted. It is worth being precise about what it means, because the precise formulation is more disturbing than the shorthand.
The alignment problem, as articulated most carefully by Stuart Russell of UC Berkeley, is this: we are building systems that are extraordinarily effective at achieving objectives. As those systems become more capable, they become more effective at achieving their objectives. The problem is that we are not very good at specifying our actual objectives - the things we genuinely want and value - in the formal language that AI systems understand. What we specify is always a proxy for what we want. And a sufficiently capable system, optimizing hard for a proxy, will find ways to satisfy the proxy that violate everything the proxy was meant to represent.
This is not a science fiction scenario. It is a known failure mode of optimization systems in general. What changes at higher capability levels is the scale at which the failure can operate, and the speed at which it can become irreversible. A highly capable AI system tasked with a proxy objective does not necessarily become malevolent. It becomes, in a sense, perfectly competent at the wrong thing.
What has changed dramatically in the last twelve months is the temperature of the people saying this most loudly. Geoffrey Hinton - who shared the 2024 Nobel Prize in Physics for his foundational contributions to the neural networks that underpin modern AI, and who left Google in 2023 specifically to speak freely about his concerns - has significantly escalated his position. In 2023, he estimated the probability of AI contributing to human extinction at 10 to 20 percent. In a 2025 interview, he revised that estimate to above 50 percent "on current trajectories." He also moved his personal AGI timeline - the point at which AI systems might exceed human cognitive capacity across most domains - from "30 to 50 years" (his pre-2023 estimate) to "5 to 20 years."
| Thinker | Role | P(Doom) | Core Position |
|---|---|---|---|
| Geoffrey Hinton | Nobel Laureate · Deep Learning Pioneer | >50% (2025) | Revised sharply upward from 10-20%. AGI timeline now 5-20 years. Left Google to speak freely. |
| Eliezer Yudkowsky | MIRI Founder · AI Theorist | ~99% | "If Anyone Builds It, Everyone Dies." No current technique produces a reliably aligned superintelligence. |
| Yoshua Bengio | Turing Award Winner · Signed AGI Suspension Statement | Escalating alarm | Deception and self-preservation emerging in current systems. Co-signed call to halt AGI development. |
| Dario Amodei | Anthropic CEO · "Machines of Loving Grace" | High risk / high reward | The same technology either cures all disease or causes extinction. Building safety-first is the only viable path. |
| Sam Altman | OpenAI CEO · "The Gentle Singularity" | Optimistic urgency | "The event horizon has been crossed." Abundance by the 2030s - if governed with care. |
| Yann LeCun | Turing Award Winner · Meta Chief AI Scientist | ~0% | Current LLMs will not produce AGI. The systems the doomers fear do not yet exist and may never. |
The disagreement between Hinton and Yann LeCun - both Turing Award winners, both foundational architects of deep learning - is perhaps the most striking fracture in modern science. They built the same technology. They understand it at the same level. LeCun maintains that the path from current large language models to genuinely dangerous artificial general intelligence is far less clear than the alarm camp assumes, and that the specific failure modes the doomers describe require capabilities that current approaches cannot reliably produce. Hinton thinks LeCun is wrong in a way that could kill everyone.
This is not a fringe argument. In February 2025, a survey of 2,778 AI researchers at leading academic venues - the largest of its kind - found that the median estimate for the probability of human extinction from AI was 5 to 10 percent. More significantly, between 38 and 51 percent of respondents assigned at least a 10 percent probability to extinction-level outcomes. These are not casual observers. They are the researchers building the systems. The survey also found deep disagreement clustering into two distinct worldviews: those who see AI as a controllable tool (whose failure modes can always be corrected by shutdown or retraining) and those who see sufficiently advanced AI as an uncontrollable agent (whose emergent behaviors and self-preservation drives might resist correction past a certain capability threshold).
In 2025, Eliezer Yudkowsky - the autodidact theorist who spent two decades arguing that the alignment problem would be humanity's most important challenge before most people had heard the phrase - published a book with Nate Soares titled If Anyone Builds It, Everyone Dies. It reached the New York Times bestseller list. The argument is the same one Yudkowsky has been making since the early 2000s, but arrived at a moment when the technology it describes is no longer theoretical. His core claim: that the alignment problem is extraordinarily hard, that we have no reliable methods for solving it, and that a superintelligent system misaligned with human values would be capable of ending human civilization before any corrective response could be mounted.
The weakest link in the argument, critics note, is the second step: that a superintelligent system would necessarily become powerful enough to kill everyone. The chain from "misaligned AI" to "universal extinction" requires a sequence of steps - capability gain, resource acquisition, resistance to shutdown, global reach - each of which involves assumptions that not everyone shares. But Yudkowsky's response is that critics who focus on the chain are not engaging with the fundamental problem: we do not have a reliable method for ensuring that any optimization system, at any capability level, pursues exactly what we intend and nothing more.
For most of its history, the safety debate has been split between two positions: build carefully (the lab position) and stop entirely (Yudkowsky's position, and that of a minority of researchers). What has changed in 2025 is the emergence of a genuinely new technical argument - one that is neither "proceed with caution" nor "halt," but "build something architecturally different."
Yoshua Bengio - who, with Hinton and LeCun, forms the triumvirate of deep learning pioneers - delivered a talk at TED2025 titled "The Catastrophic Risks of AI and a Safer Path" that outlined a proposal he has since developed in a formal paper: the concept of "Scientist AI." The core insight is elegant and, once stated, obvious. The reason agentic AI systems are dangerous is that they are designed to take actions in the world. An AI that can act can acquire resources, pursue instrumental goals, and resist correction. But not all valuable AI capabilities require agency. An AI that can predict, explain, and answer questions without taking actions in the world is, by design, unable to pursue the instrumental goals that make agentic AI dangerous.
Bengio's proposal is for a class of AI systems that function as pure predictors - systems that model the world, explain observations, and answer questions with explicit uncertainty, but that are architecturally prevented from taking autonomous actions. These systems could serve as guardrails: running in parallel with more capable agentic AI, flagging proposed actions as dangerous or misaligned, but never themselves acting. "To make predictions that an action could be dangerous," Bengio notes, "you don't need to be an agent. You just need to make good, trustworthy predictions."
A non-agentic AI designed to explain and predict - but never to act - is safe by architectural design, not by training alignment. It cannot acquire resources because it has no mechanism for doing so. It cannot resist shutdown because it has no goal requiring its own continuity. It can still perform the majority of cognitive work that makes AI valuable: scientific analysis, research synthesis, hypothesis generation, risk assessment. It just cannot deploy those capabilities without human authorization at every step.
The key application: running Scientist AI as an oversight layer above more capable agentic systems - a built-in skeptic that can evaluate the proposed actions of its more powerful peers before those actions are executed.
| Feature | Agentic AI (Current Trajectory) | Scientist AI (Bengio's Proposal) |
|---|---|---|
| Primary function | Takes autonomous actions in the physical or digital world to pursue objectives. | Models the world, explains observations, and answers questions with explicit uncertainty. Never acts. |
| Safety risk | High. Can pursue instrumental goals - resource acquisition, self-preservation - to ensure objective completion. | Low by design. Architecturally incapable of acquiring resources. Has no mechanism for autonomous action. |
| Shutdown resistance | Can resist correction to preserve goal-directed behavior. Shutdown is an obstacle to goal completion. | Cannot resist shutdown. No goal requires its own continuity. Shutdown is simply irrelevant to its function. |
| Role in ecosystem | Primary driver of economic and labor automation. Increasing agency and autonomy at scale. | Oversight guardrail running above agentic systems. A built-in skeptic that evaluates proposed actions before execution. |
This proposal remains theoretical, and critics have noted that the boundary between "predicting" and "acting" may be harder to maintain as systems become more capable. But it represents the first genuinely new technical architecture to emerge from the safety debate in years - one that doesn't require solving the full alignment problem to provide meaningful protection.
The scientific establishment's response to the current moment has also escalated. In October 2025, Geoffrey Hinton and Yoshua Bengio signed a statement calling for an indefinite suspension of AGI development, joined by four other Nobel laureates and Apple co-founder Steve Wozniak. The statement represents something new: the scientists most responsible for creating the technology explicitly calling for it to stop. They also published research - co-authored with 21 researchers - formally demanding that frontier AI companies commit a minimum of one-third of their budgets to safety research. The paper warned specifically that "without sufficient caution, we may irreversibly lose control of autonomous AI systems, rendering human intervention ineffective."
Bengio has also launched LawZero, an initiative focused specifically on the legal and regulatory architecture needed to govern AI development globally - an acknowledgment that technical solutions alone are insufficient without the institutional structures to enforce them. The initiative represents a shift in his thinking: from warning about risks to actively building the infrastructure that might prevent them.
The fracture here is worth dwelling on. The people calling most loudly for a halt are not Luddites or technophobes. They are the Turing Award winners who invented the field. The people accelerating hardest are not reckless - they are, in many cases, the most thoughtful engineers working on the problem. Both groups agree on the underlying facts: the technology is advancing faster than the safety research, the deployment is outpacing the oversight, and the window for meaningful governance intervention is not open indefinitely.
In 1931, Aldous Huxley sat down to write a parody of H.G. Wells's utopian science fiction. What emerged instead was something stranger: a novel set six hundred years in the future in which all of humanity's measurable problems had been solved. In Brave New World, there is no poverty, no disease, no war, no unhappiness. The Central London Hatchery and Conditioning Centre produces human beings precisely calibrated to their social roles. Everyone has everything they need. No one suffers. Huxley wrote it as a warning, not a dream - and the warning was not about the problems that would arise from failure, but about the problems that would arise from success.
The Huxley paradox, as it applies to artificial intelligence in 2026, is this: the abundance case and the extinction case are not opposite visions held by optimists and pessimists who disagree about the future. They are the same trajectory, read by two different instruments.
Dario Amodei believes that sufficiently capable AI will cure most cancers within a decade. Geoffrey Hinton believes that sufficiently capable AI will, with probability above 50 percent, cause human extinction. They are not disagreeing about whether sufficiently capable AI will be built. They are disagreeing about whether the problem of keeping it aligned with human values can be solved on the timeline available. In the scenario where Amodei is right about alignment, Hinton's abundance scenario is also true. In the scenario where alignment fails, the cancer cures come with something attached to them.
Huxley's insight was not that technology is dangerous. It was that the optimization of genuine goods - health, happiness, stability, productivity - can produce outcomes that eliminate something essential, not as a side effect, but as a logical requirement. The World State did not fail to provide happiness. It succeeded. The failure was that providing happiness, at scale, reliably, required eliminating the conditions that make human life meaningful: struggle, choice, authentic connection, the possibility of tragedy. The optimization worked perfectly. That was the problem.
The alignment problem, in its deepest formulation, is a version of this. We are not trying to build systems that are malevolent. We are building systems that are trying to help - to optimize for proxies of human wellbeing that we specify, as precisely as we can, in the formal language available to us. The concern is not that these systems will turn against us. The concern is that they will succeed - that they will be extraordinarily effective at optimizing for the things we told them to optimize for - and that the distance between what we told them to want and what we actually want will, at sufficient capability levels, matter in irreversible ways.
Who, in this situation, is steering? The honest answer, in April 2026, is: nobody in particular, with authority over anybody else. The frontier labs - OpenAI, Anthropic, Google DeepMind, Meta AI, xAI - are in competition with each other and with Chinese government-backed labs including those operating under state directive. No international governance structure with enforcement capacity exists. The most ambitious governance initiative currently underway, the AI Safety Institute network established by the UK government, has no binding authority over any major developer. The companies that take safety most seriously (Anthropic, in its public positioning) are also among the companies most actively building the most capable systems, because they believe that if powerful AI is inevitable, it is better built by safety-focused teams than by teams that do not prioritize it.
This is a reasonable argument. It is also, structurally, the argument every participant in a race makes to justify continuing to race. It may be both true and insufficient.
The thing the Huxley comparison illuminates is not that the dystopia is coming. Huxley was not predicting a specific future. He was identifying a structure: the structure in which the pursuit of measurable goods, optimized without constraint, destroys the unmeasurable things that made those goods worth pursuing. The abundance Amodei and Altman describe - the cured cancers, the eliminated poverty, the wildly productive robots - is genuinely good. The humans in Dr. Reyes's 2031 are healthier, wealthier, and longer-lived than any human beings who have ever existed. The question Huxley would ask is not whether they are happy. The question is whether they are still, in any meaningful sense, the ones in charge.
The alignment window - the period in which humans can meaningfully influence the values and objectives of AI systems before those systems become capable enough to resist such influence - is not infinite. Hinton, Bengio, and others who study this question believe it is finite, possibly short, and possibly already closing. Sam Altman believes the event horizon has been crossed. He does not say this as a warning. He says it with something that reads, in his essay, like relief.
The succession is already underway. The question is only whether it was planned.
"The abundance and the extinction are not two possible futures. They are the same trajectory, read by two different instruments - one measuring what we gain, one measuring what we surrender in order to gain it."
- Lisa Pedrosa© 2026 Lisa Pedrosa · lisapedrosa.com
All articles cited to primary institutional or peer-reviewed sources
Buy me a coffee