In the latest quarterly review from the Bank for International Settlements, researchers backtest the “stress tests” employed by regulators and supervisors to gauge the solidity of banks during financial crises.

After studying tests conducted around 43 banking crises in 30 countries since 1974—including those immediately preceding the latest crash—the research finds that the “overwhelming majority [of countries] concluded that their banking systems were robust even in the face of very severe adverse scenarios.” In fact, nearly 70% of the “very severe” macroeconomic scenarios envisioned by the tests proved milder than subsequent events.

Economic growth preceding the crisis is an important factor in determining the accuracy of assumptions. High growth in the run-up to a crisis tends to result in overly mild stress-test scenarios, for example. And as Nassim Nicholas Taleb was probably thrilled to read, stress testers often find it hard to imagine conditions “beyond the realm of anything that has been experienced.”

Stress testing, as traditionally practiced, is likely to “lull users into a false sense of security” by continuing to underestimate the dangers of extreme events, the research concludes. Put another way, the Alea blog answers the hypothetical question in the paper’s title—Macro stress tests and crises: what can we learn?—with a simple rejoinder: “That they don’t work.”

Advertisements