---
title: "Train, want, prefer"
description: "The three-step chain that shows why the final preference cannot be read off the training target."
canonical: https://tomsharman.com/bookshelf/if-anyone-builds-it-everyone-dies/train-want-prefer/
updated: 2026-11-03
---

# Train, want, prefer

> The three-step chain that shows why the final preference cannot be read off the training target.

Eliezer Yudkowsky and Nate Soares's framework, from If Anyone Builds It, Everyone Dies, pp. 21 to 101.

Framework 2 of 5 from [If Anyone Builds It, Everyone Dies](https://tomsharman.com/bookshelf/if-anyone-builds-it-everyone-dies/index.md) by Eliezer Yudkowsky and Nate Soares. Category: Leadership.

## The steps

1. **Train on a target.** A blind process rewards outward behaviour, such as passing on genes or being a helpful assistant.
2. **Chaotic inner wants form.** The process stumbles on machinery that points toward the target in the training environment, in ways that are underconstrained.
3. **Power opens new options.** With more intelligence the mind invents options the training never covered, such as ice cream or sucralose, and prefers those.

## Where it comes from

From chapter: PART I: NONHUMAN MINDS, pp. 21–101

## The animation

24 seconds, 4:5, no sound.

- MP4: https://tomsharman-downloads-006143767029.s3.eu-north-1.amazonaws.com/downloads/bookshelf/if-anyone-builds-it-everyone-dies/train-want-prefer.mp4
- MP4, dark: https://tomsharman-downloads-006143767029.s3.eu-north-1.amazonaws.com/downloads/bookshelf/if-anyone-builds-it-everyone-dies/train-want-prefer-dark.mp4
- Still: https://tomsharman-downloads-006143767029.s3.eu-north-1.amazonaws.com/downloads/bookshelf/if-anyone-builds-it-everyone-dies/train-want-prefer.png
- Still, dark: https://tomsharman-downloads-006143767029.s3.eu-north-1.amazonaws.com/downloads/bookshelf/if-anyone-builds-it-everyone-dies/train-want-prefer-dark.png