The four curses of hard engineering

A checklist drawn from Chernobyl for judging how dangerous a device is to build without full understanding.

Eliezer Yudkowsky and Nate Soares's framework, from If Anyone Builds It, Everyone Dies, pp. 139 to 195.

Speed
Processes that run faster than humans can react leave operators frozen if the slowing contrivance fails.
Narrow margins
A thin gap between unimpressive and explosive means small errors are fatal.
Self-amplification
Feedback loops, such as boiling coolant raising reactivity, leave little room for error.
Complications
Unknown interactions, such as graphite-tipped control rods, create states no engineer designed for.

From chapter: PART III: FACING THE CHALLENGE, pp. 139–195

More from this book