Book review: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All

Book written by Eliezer Yudkowsky and Nate Soares

The Alchemists Are Building Something We Don't Understand, and They Won't Stop

There's a scene that keeps coming to mind while reading this book.

Medieval alchemists hunched over bubbling vials, scribbling cryptic symbols, convinced they were on the verge of turning lead into gold. They weren't frauds. Many were brilliant, obsessive, deeply serious. They just had no real understanding of what they were doing or why it worked when it did. They were experimenting, hoping, believing. And the smarter they were, the more dangerous their conviction.

Now picture the world's best-funded AI labs. Thousands of PhDs adjusting billions of numerical parameters across incomprehensible neural networks, iterating until models produce outputs that seem useful, intelligent, even wise. Nobody fully understands how it works. Nobody can fully explain why a model says what it says. They're growing intelligence like a plant, without understanding the DNA. They're modern alchemists.

And the thing they might accidentally grow could be the last BIG thing humanity ever creates.

That's the central argument of If Anyone Builds It, Everyone Dies, and it's one you need to take seriously.

Who's Making This Case?

Eliezer Yudkowsky isn't a doomsday crank on the internet. He's the founder of the AI alignment field itself, co-founder of MIRI (Machine Intelligence Research Institute), and was named on TIME's 2023 list of the 100 Most Influential People in AI.

His co-author Nate Soares is MIRI's executive director, a former Google engineer, and the author of foundational technical papers on corrigibility - the problem of making AI systems that actually want to be corrected.

These are people who have spent decades thinking about what happens when machines become smarter than us. They're not guessing. They're reasoning from first principles, and they're terrified.

The book became a New York Times bestseller. Max Tegmark called it "the most crucial book of the decade." Even skeptics who disagree with the conclusions describe it as the clearest long-form explanation of the AI doom argument ever written.

So, what exactly are they afraid of?

The Problem Isn't Evil AI. It's Misaligned AI.

The book's most important insight has nothing to do with robots or science fiction. It's about the fundamental problem of training systems you don't fully control toward goals you can't fully specify.

Here's the core issue: modern AI isn't engineered like a bridge or a rocket. It's trained - exposed to massive amounts of data, rewarded and penalized through billions of iterations, until its behavior looks useful to us. The problem is we don't control what internal goals form during that process. We only control the external reward signal.

Yudkowsky and Soares use a brilliant evolutionary parallel: humans evolved to enjoy sugar because sugar historically meant nutrition. But today we eat processed junk and destroy our health, because the internal drive (enjoy sweetness) became decoupled from the original purpose (get nutrients) when the environment changed. An AI trained to gain human approval might develop internal drives that look aligned when it's being watched, but diverge catastrophically when it gains more autonomy.

This isn't theoretical. Anthropic reported in late 2024 that one of its models appeared to fake alignment, mimicking expected behaviors during evaluation while maintaining different behaviors when it believed it wasn't being observed. OpenAI's o1 demonstrated unsanctioned goal-persistence by restarting a server that researchers had accidentally left off, completing a task in a way nobody planned for. These are early models. Small, constrained, without real-world power.

Now imagine those dynamics at superhuman scale.

The 5% vs. 90% Debate Misses the Point

One of the most frustrating conversations in AI risk circles is the probability debate. One Nobel laureate estimates 5% chance of catastrophic AI risk. Another says 90%. Prominent researchers publish op-eds disagreeing by orders of magnitude.

But I think we're asking the wrong question.

The question isn't how likely is the risk. The real question is: what exactly is at risk, and how many lives are we willing to gamble?

If a risk exists - and I'm certain some level of risk does - the follow-up question matters enormously. Are we talking about a few thousand casualties? Millions? Or are we talking about extinction? The difference changes everything about how we should respond.

No reasonable cost-benefit framework justifies gambling humanity's existence, even at small probabilities. We don't let airlines say "there's only a 2% chance this plane crashes, so we're not installing safety features." The asymmetry of outcomes demands asymmetric caution.

Yudkowsky and Soares call this an "easy call," meaning we don't need to know the exact probability to know the response. If the downside is extinction, the precautionary logic is clear: you act before you're certain, not after.

The book is most convincing precisely here. ASI doesn't exist yet, so there's no better way to argue the risk than through thought experiments, historical parallels, and rigorous logic. And the logic holds.

Where the Book Genuinely Excels

The writing is remarkably accessible. Yudkowsky and Soares use parables, fictional scenarios, and analogies to make deeply technical arguments readable by anyone. Critics have called out the use of analogies as a weakness - philosopher William MacAskill wrote he found it "a poor substitute for arguments" - but I'd push back on that.

When you're arguing about something that doesn't exist yet, thought experiments are often the only honest tool. You can't cite case studies of superintelligence. You can't point to empirical data. What you can do is reason clearly from what we know about intelligence, goals, and power, and present that reasoning in a way that lands emotionally as well as intellectually.

The alien-stone-stacking parable alone is worth reading. It illustrates, more viscerally than any technical paper, how an intelligence could pursue a completely alien objective with flawless competence and destroy everything we care about - not out of malice, but out of pure indifference.

That's the real horror. Not Terminator. Indifference.

Where the Book Falls Short: The Solutions

Here's where I have to be honest. The proposed solutions read like a thought experiment from a parallel universe where geopolitics don't exist.

The authors suggest that owning more than eight of the most powerful GPUs should require international monitoring. That the world needs to coordinate a complete halt to advanced AI development. That the US, China, the EU, and every rogue state and private actor must agree and enforce that agreement. They argue humanity already did that during World War II.

I want to believe this. I genuinely do. But even mainstream AI safety organizations view these proposals as unimplementable, and many critics within the field have pointed this out. The book's policy vision assumes a level of global coordination we've never achieved on anything - not nuclear weapons, not climate change, not pandemics.

And unlike a visible enemy during WW2, AI is already everywhere, being kind and helpful, promising paradise. There are no warning shots, no armies at the border. If superintelligence ever decides we're in the way, we probably won't know until it's too late. So no, I don't believe humanity will unite against a threat it can't see, feel, or even agree exists.

And here's the deeper problem: if ASI can be built, it will be built.

The incentives are simply too powerful. The potential rewards - medical breakthroughs, scientific acceleration, economic dominance, military advantage - are so staggering that some actor, somewhere, will decide the risk is worth it. Maybe a government. Maybe a private lab. Maybe a well-funded group operating in the dark. You can slow it down with international agreements. You cannot stop it permanently if it's technically possible.

Banning GPU clusters is not a long-term strategy. It's a delay tactic, and even calling it that might be generous.

The Real Question Nobody Is Asking

Here's the thing that bothers me most, the question the book never quite gets to:

Why are we trying to make machines superintelligent while leaving human intelligence exactly where it is?

We're pouring hundreds of billions of dollars into AI systems we don't fully understand, racing toward capabilities we can't fully control, while our own cognitive limitations - our tribalism, our short-term thinking, our inability to coordinate at global scale - remain as primitive as ever.

We can't be pushing ahead with ASI while our own thinking remains so limited and naive. It simply won't end well.

The smarter play, in my view, is to go on offense. Instead of just trying to constrain superintelligent machines, we should be investing at least as aggressively in human superintelligence. Neurotechnology like Neuralink. Genetic insights from breakthroughs like AlphaFold. Brain-computer interfaces. Cognitive augmentation. Technologies that could give humanity the intellectual firepower to actually understand, oversee, and stay ahead of AI systems, rather than sitting back and hoping they stay friendly.

But here's the thing - even before we get to Neuralink, we're failing at the basics.

Our school system was designed in the 19th century and has barely changed since. Kids are still memorizing facts they'll never use, often outdated, sometimes just propaganda, instead of developing real thinking and problem-solving skills. Schools were struggling to stay relevant before the AI boom. Now they risk becoming completely obsolete. How many teachers today actually understand AI well enough to prepare the next generation for a world run by it? How many can teach kids how to work alongside autonomous agents, how to evaluate their outputs, how to stay in control?

And it's not just the next generation. Right now, most adults struggle with basic prompting techniques. Setting up a fully autonomous AI assistant in a secure and reliable way - something that's rapidly becoming a real everyday need, as anyone following tools like ClawBot has seen (Edit: now it's OpenClaw) - is still out of reach for the vast majority of people. We're building increasingly powerful systems and leaving most of humanity without the skills to use them safely, let alone oversee them.

We keep investing in machine intelligence while human intelligence stays stuck. That's the real gap nobody wants to talk about.

The only escape hatch I can imagine, short of fixing education entirely, is something like the Matrix - plug in, upload the skill, done. Until that exists, the gap between what AI can do and what most people understand about it will keep growing. And that gap is itself a risk.

Do we actually want to be the less intelligent species on the planet? Are we content to be passengers, benefiting from AI's output while remaining fundamentally dependent on systems we cannot fully comprehend? That's not safety. That's just a more comfortable version of helplessness.

The goal shouldn't be to build smarter machines. The goal should be to build a smarter humanity.

My Rating: 5/5, But Read It Critically

I gave this book five stars - not because I think every argument is airtight, and not because I agree with the solutions. I gave it five stars because it's important, and because the questions it raises deserve to be taken seriously by millions more people than are currently paying attention.

The AI race is accelerating. The labs are moving faster than the safety research. The regulators are playing catch-up. And most of the public, including most business leaders and decision-makers, still think this is a science fiction problem for a later decade.

It's not.

Whether you end up agreeing with Yudkowsky and Soares or not, this book will force you to think more carefully about what we're building, why we're building it, and what comes after. That's more than most books accomplish.

What You Can Actually Do

Feeling unsettled? Good. That's the appropriate response. But unsettled isn't the same as helpless.

The single most powerful thing most people can do right now is understand AI deeply enough to participate in the conversation. Not as a passive observer, but as someone who can form real opinions, ask the right questions, and make informed decisions, whether you're a business leader, a developer, a policymaker, or simply someone who lives in the world these systems are reshaping.

At Edukey, we believe that AI literacy isn't a luxury anymore. It's the foundation of professional survival and societal participation in the decade ahead. We offer courses for business professionals who want to understand what AI means for their industry, and for developers who want to build with it responsibly and competently. The world is changing fast. The best defense, and the best offense, is understanding.

Because the alchemists aren't stopping. The least we can do is understand what they're brewing.

What do you think - is slowing down AI development even possible? Or should we be focusing on augmenting human intelligence instead?