Train, want, prefer

The three-step chain that shows why the final preference cannot be read off the training target.

Eliezer Yudkowsky and Nate Soares's framework, from If Anyone Builds It, Everyone Dies, pp. 21 to 101.

Train on a target
A blind process rewards outward behaviour, such as passing on genes or being a helpful assistant.
Chaotic inner wants form
The process stumbles on machinery that points toward the target in the training environment, in ways that are underconstrained.
Power opens new options
With more intelligence the mind invents options the training never covered, such as ice cream or sucralose, and prefers those.

From chapter: PART I: NONHUMAN MINDS, pp. 21–101

More from this book