Bookshelf
How Tom uses this
If Anyone Builds It, Everyone Dies backs one of Tom's lessons. Under each of his lessons, what the book says about it, and on which pages.
What it argues
Yudkowsky and Soares argue that if anyone builds artificial superintelligence with anything like current methods, everyone on Earth dies. Modern AIs are grown by gradient descent, so nobody understands or can set what they end up wanting, and a mind trained for success will pursue strange preferences that have no place for humans. A superintelligence would beat us with technology we cannot foresee, and it needs only the internet, money and willing humans to start. Alignment must work on the first try, and the field is at the alchemy stage, with leaders offering idealistic plans and incentives pushing everyone up the ladder. They conclude that the only safe move is an international treaty that monitors compute and halts frontier AI research, and that it is not too late to do it.
Worth keeping
- Intelligence is prediction plus steering, and two smart minds can steer to different destinations (p. 24-25).
- AIs are grown, not crafted: engineers know the training process but not the cognition inside (p. 36-38).
- You do not get what you train for, shown by the ice cream, sucralose and contraception chain (p. 55-58, 65).
- Easy calls versus hard calls: the end state can be predictable when the path is not (p. 14-15, 50).
- The four curses of Chernobyl, speed, narrow margins, self-amplification and complications, as a test of any risky system (p. 143-147).
- Computer security as the curse of edge cases, where an intelligent adversary finds the one path that breaks your constraint (p. 148-149).
- The ladder in the dark: each rung pays more, the top rung kills everyone, and nobody knows where it is (p. 174-175).
- A field in the folk theory stage sounds like alchemists, with idealistic slogans in place of engineering (p. 156-158, 164-165).
- A broad coalition on a single ask works better than bundling it with other causes (p. 185).
- Nuclear war was avoided because people believed they would suffer too and worked at it for decades (p. 189-190).
The frameworks
5 frameworks, one animation each Every framework, step by step
The seven steps by which engineers produce an LLM, showing why nobody designs what ends up inside.
- Render the input as numbers. Turn a text fragment into a series of numbers.
- Provide parameters. Acquire storage for billions of numbers, each slot a parameter.
- Fill with random weights. Fill the slots with randomly chosen numbers called weights.
- Choose the architecture. Set the rules that combine the input with the weights.
- Read the output. Treat the output numbers as probabilities for the next letter or token.
- Train by gradient descent. For every weight, work out how a tiny change would make the answer more correct, nudge it that way and repeat over trillions of words.
- Generate text. Feed each predicted token back in to continue the text, then tune the model to answer as an assistant.
From chapter: PART I: NONHUMAN MINDS, pp. 21–101
The three-step chain that shows why the final preference cannot be read off the training target.
- Train on a target. A blind process rewards outward behaviour, such as passing on genes or being a helpful assistant.
- Chaotic inner wants form. The process stumbles on machinery that points toward the target in the training environment, in ways that are underconstrained.
- Power opens new options. With more intelligence the mind invents options the training never covered, such as ice cream or sucralose, and prefers those.
From chapter: PART I: NONHUMAN MINDS, pp. 21–101
A checklist drawn from Chernobyl for judging how dangerous a device is to build without full understanding.
- Speed. Processes that run faster than humans can react leave operators frozen if the slowing contrivance fails.
- Narrow margins. A thin gap between unimpressive and explosive means small errors are fatal.
- Self-amplification. Feedback loops, such as boiling coolant raising reactivity, leave little room for error.
- Complications. Unknown interactions, such as graphite-tipped control rods, create states no engineer designed for.
From chapter: PART III: FACING THE CHALLENGE, pp. 139–195
The authors' outline of what the world would need to do to stop escalation to superintelligence.
- Consolidate compute. Gather all computing power that could train or run more powerful AIs into places monitored by multiple treaty powers.
- Set the threshold low. Limit unmonitored hardware to about eight of the most advanced GPUs from 2024.
- Ban efficiency research. Make it illegal to publish research into more efficient and more powerful AI techniques.
- Enforce everywhere. Warn, then sabotage or strike, any datacentre that defies the treaty, while offering the holdout equal terms to join.
- Keep the coalition broad. Do not bundle the ban with any other cause, so that nearly everyone can agree.
From chapter: PART III: FACING THE CHALLENGE, pp. 139–195
The closing chapter's guidance by role for helping prevent extinction.
- Governments. Signal willingness to an international treaty so no country is disadvantaged by stopping.
- Politicians. At least require GPU clusters to be concentrated so the brakes can be applied later.
- Journalists. Cover the subject with gravity and mention outside scientists' warnings.
- Citizens. Write to representatives, vote in primaries, join large lawful protests and talk about it, then live life well.
From chapter: PART III: FACING THE CHALLENGE, pp. 139–195
Working with Tom on this
Tom works with creators and founders one to one, from a single session to a fractional role. Work with Tom
Back to the bookshelf
