Summary: complex systems and emergent risk
Summary
Section titled “Summary”L5 introduced the Swiss-cheese composition rule with an independence proviso: N layers each catching p percent of failures compose to high reliability if the layers are independent. L6 takes that proviso seriously. Real-world deployed systems rarely have truly independent layers; their layers share blind spots, share infrastructure, share adversaries who attack all of them with the same technique. The Swiss-cheese stack you actually have in production is one whose slices have correlated holes.
The framing that makes the failure mode visible is the complex-systems lineage. The chapter draws on Charles Perrow’s Normal Accidents (1984), which argued for nuclear plants, petrochemical plants, and air-traffic control that a system can be assembled from components that are individually correct and still produce behavior the designers did not predict and cannot easily prevent.
Four properties recur. Emergence: the system has properties no component does (a network represents concepts no neuron does; a market discovers prices no trader does). Nonlinearity and sensitivity to initial conditions: small input changes produce large output changes in analytically intractable ways. Feedback loops: outputs feed back as inputs, sometimes stabilizing (negative feedback toward setpoints) and sometimes amplifying (positive feedback producing boom-bust patterns). Tight coupling: the state of one part constrains others within timescales too short for human intervention; tight coupling plus interactive complexity is Perrow’s diagnostic for normal accidents, the class of accidents whose mechanism is statistically inevitable given the system structure.
Three historical illustrations the literature returns to. Three Mile Island (1979): no component failed in a way engineering had not designed for; the interaction between a stuck indicator, an inappropriate operator response, and coupled subsystem dynamics was the failure mode. Flash Crash (2010): each algorithmic trader did exactly what its specification said; the interaction produced a liquidity-draining feedback loop. 737 MAX MCAS (2018-2019): tight coupling between a single-sensor flight-control augmentation, a certification process that did not require pilot training on it, and operational scenarios where the system diverged from pilot mental models. In each case, component-level engineering could have correctly answered every component-level question and still produced a system whose failure mode was structurally inevitable.
AI deployments fit the pattern. They are tightly coupled to their environments (recommendation systems shape user preferences which become training data on a timescale shorter than policy can respond). Multi-agent AI deployments produce emergence at the population level (markets crash, traffic deadlocks, web content collapses into self-referential loops). Emergent capabilities at scale are themselves a complex-systems phenomenon. Model monoculture is a fourth pattern worth naming: when many deployed systems share the same base model, correlated failure modes that are invisible at the individual-model level become visible at the population level; the risk lives at a layer no individual product team can address.
L5’s Swiss-cheese rule breaks when layers share blind spots, share infrastructure, or face an adversary who can defeat all layers with one technique. The operational fix is not more layers but more genuine independence (different teams, methods, signals, timescales). The Swiss-cheese stack you can defend is the one where you can articulate why each slice’s holes are uncorrelated with every other slice’s holes.
L6 closes Phase 2. L3 named the failure surface, L4 named the substrate, L5 brought the engineering toolkit, L6 inverts L5 by showing where the toolkit’s assumptions break. Phase 3 opens at L7 with ethics; L8 takes multi-agent dynamics that L6 previewed and works them at full depth; L9 brings governance as the outside-any-individual-deployment layer.
The L6 capability is the four-step move: name four complex-systems properties in a deployment; distinguish normal accidents from preventable engineering failures; recognize Swiss-cheese-independence failures; propose system-structure design changes (not component fixes) that address complex-systems risk. Practice has the delivery-logistics worked example, a Perrow-flavored decomposition of three historical incidents, and a design-proposal exercise.