Don’t Bet on It
THE SIGNAL AND THE NOISE:
Why So Many Predictions Fail—but Some Don’t.
By Nate Silver. Penguin Press. 534 pp. $27.95
Nate Silver is the new toast of the punditocracy. The author of The New York Times’ widely followed FiveThirtyEight blog, he correctly predicted the winner of the presidential vote in all 50 states, besting his already impressive record of 49 correct calls in the 2008 election. But what if Silver had been wrong? The last person to be surprised probably would have been Silver himself. That is what makes The Signal and the Noise, in which he surveys methods of prediction in everything from Texas hold ’em to global climate change, such a useful and important book.
Throughout human history, people have ascribed special and sometimes sacred qualities to those who seem able to see the future, from the Oracle at Delphi in ancient Greece to Nassim Taleb, the PhD-holding derivatives trader who famously warned right before the recent financial crisis that unforeseen “black swan” events occur more often than we think. Before Silver, Warren Buffett was the prophet of the hour.
In modern times, the human preoccupation with the future has become a near obsession. Our lives revolve around questions about what will happen tomorrow: When will terrorists strike America next? How severe will the effects of climate change be? Will I outlive my retirement savings? We are bombarded by honest but flawed forecasts as well as ones designed chiefly to scare or lull us into certain courses of action. Silver’s fame is deserved, but picking election winners is a relatively trivial business. What makes him special is that he is probably the first celebrity prophet not only to help us think more intelligently about the future but to caution that there is very little that we can know about it with certainty.
That has something to do with his life story. After getting an undergraduate degree in economics at the University of Chicago, Silver worked in the early 2000s as a low-level international tax consultant before turning his talents to baseball (his childhood passion) and developing PECOTA (Player Empirical Comparison and Optimization Test Algorithm), a statistical database designed to predict players’ performance. His interest was sparked by the success of the Oakland Athletics under the statistically minded general manager Billy Beane—a tale later chronicled by journalist Michael Lewis in Moneyball (2003). PECOTA was successful—Silver later sold it to another firm that sells forecasts to professional baseball teams and others—but it was also frequently wrong. Looking back over six years at the end of 2011, Silver found that his system had identified a good number of minor league players who went on to success in the big leagues, but it had also been outperformed by a competing system emphasizing scouting reports. It is a lesson that he repeats throughout the book: Data crunching is not enough; there is no substitute for experience and judgment.
Even as he was working on PECOTA, Silver slid into the world of professional gambling, quickly making his way to the World Series of Poker. Play at that level requires a prodigious ability to calculate probabilities on the fly, along with other skills, and Silver prospered for a time. After a string of losses, however, he abandoned poker in 2007, concluding that the competition had stiffened and he had not improved his own methods. “My years in the game taught me a great deal about the role that chance plays in our lives and the delusions it can produce when we seek to understand the world and predict its course,” he writes.
That is not the kind of autobiographical lesson we are likely to hear from the soothsayers on the political talk shows and stock market Web sites. Silver’s key point is that we need to think in terms of probabilities rather than comforting certainties. His FiveThirtyEight prognostications (the blog’s name derives from the number of electoral votes in a presidential election) are always expressed in these terms: At various times during the last election, Silver put Barack Obama’s chances of winning as low as 59 percent and as high (on the eve of the vote) as 91 percent. That made clear that the prediction could be wrong—sometimes even a full house can lose. Remember that the next time you hear some bold forecast about the coming of American energy independence or the size of the federal government’s budget deficit in 10 years.
Silver’s methods involve taking huge amounts of data—in the case of the presidential election, mostly state-level political polls—and averaging and adjusting them for demographic and other factors using techniques based on the principles of the 18th-century English statistician Thomas Bayes. Bayesian probability theory requires us to make our best guess about the future and then continually revise it as we get new information.
Most of The Signal and the Noise is not concerned with Silver’s work but with other fields, from poker and chess to economic and climate prediction, in which Bayesian principles can be applied. Climate modelers come off pretty well in this light, though Silver says their predictions are still surrounded by high levels of uncertainty. Social scientists take perhaps the biggest drubbing. Late in 2007, for example, economists surveyed by The Wall Street Journal said there was only a 38 percent chance of recession in the next year. In fact, as data would later reveal, the economy had already slid into a downturn.
One surprise winner in the prediction sweepstakes is weather forecasting, which has measurably improved in recent decades. In the 1980s, the National Hurricane Center couldn’t predict within 100 miles where a storm would make landfall more than a day in advance, but now, because researchers have been able to build sophisticated models of how storm systems behave, it can do so three days out, giving people in threatened areas precious time to evacuate. When the National Weather Service says there is a 20 percent chance of rain, it actually does rain 20 percent of the time.
Good predictions have two main sources. There are lots of data that provide plenty of feedback, which allows forecasters to constantly check and adjust their models against reality. And there is a reasonably solid understanding of how the underlying weather system works. Economists do a dreadful job because they have mountains of data but, lacking a good model of how the economy works, are hard put to understand what it means. They have difficulty separating the “signal” from the “noise.” As Silver puts it, “The signal is the truth. The noise is what distracts us from the truth.” Finding the signal, however, is not merely a matter of creating better techniques. There were clues that could have alerted us to the 9/11 terrorist attacks before they occurred, for example, but defense and intelligence officials had already dismissed the possibility of attacks on such a scale by suicide bombers. They simply couldn’t imagine such a thing.
“We just aren’t that good at prediction,” Silver writes. It is an unexpected and important message to hear from a certified celebrity seer. Prediction is a necessary but peril-ridden art. We need to be less confident in our own ability to see around corners, he argues, and more skeptical of others’ claims. Those are among the many lessons that give this book a very high signal-to- noise ratio.