Every so often I come across a book that should be read by: (a) all economists; (b) all students; (c) everybody involved in politics and policy; and (d) everybody else with an interest in the world. Nate Silver’s The Signal and The Noise: The Art And Science of Prediction – which I’ve finally read shamefully long after publication – is one of those books. It should in fact be read by all macroeconomists who publish forecasts at least annually, as a condition of their continuing membership of the profession. If I were teaching, it would emphatically be a required item on the course reading list.
It is a wonderfully clear and engaging explanation of the challenges of making predictions in fields ranging from politics and elections to weather and earthquakes to economics and poker. Apart from a couple of sections on American sports, which might as well have been written in a foreign language, the examples illustrate how to, and how not to, make forecasts. You’ll be wiser for reading it, not to mention able to put Bayesian inference into practice. Silver makes a compelling case for adopting the Bayesian approach, rather than the standard (‘frequentist’) statistics descended from R.A.Fischer and universally taught to economists in their econometrics courses. The emerging new economics curricula should at least include Bayesian statistics in the modules covering empirical methods. As Silver writes:
“Essentially the frequentist approach toward statistics seeks to wash its hands of the reason that predictions most often go wrong: human error. It views uncertainty as something intrinsic to the experiment rather than something intrinsic to our ability to understand the real world.”
In other words, it is not true that collecting more and more data – although usually useful to a forecaster – will eliminate your uncertainty about the real world. The signal-noise problem is epistemologically unavoidable. What’s more the frequentist approach involves assumptions about the distribution of the population; we know about the (in-)validity of the normal curve assumption, and anyway, “What ‘sample population’ was the September 11 attack drawn from?”
The chapter on macroeconomic forecasting is a pretty devastating critique of economists who do that kind of thing. There is a demand for macro forecasts, and I’d rather economists supply them than anybody else. But we shouldn’t pretend they’re going to be accurate. Almost all forecasters, even if they publish standard errors, will give the impression of precision – is growth going to be 0.5% or 0.6%? – but it is inaccurate precision. Silver calculates that over the period 1993-2010, GDP growth fell outside the 90% confidence intervals of macro forecasts for the US economy a third of the time, and a half the time if you look back to 1968.
Macroeconomic data are very noisy, especially early estimates of GDP: in the US the margin of error on the initial quarterly estimate of GDP is plus or minus 4.3%. The initial estimate for the final quarter of 2008 was a decline of 3.8% – later revised to minus 9 per cent. Silver makes the comparison between economic forecasts and weather forecasts, similarly difficult problems. However, weather forecasting has improved over the decades, thanks to a better understanding of the causal links and a greater degree of disaggregation of data, made possible by more powerful computers. Economists have neither the improved understanding – on the contrary, important causal links notably finance were ignored until recently – not seemingly the appetite for better data (as I’ve pointed out before).
The book also makes the point that others (like Paul Ormerod) have emphasised, that the economy is a complex non-linear system so there is a lot of unavoidable uncertainty about forecasts more than a short period ahead. It also notes that although we know about the Lucas Critique and Goodhart’s Law – both pointing out that policy affects behaviour – economic forecasters typically ignore it in practice. Silver also underlines the rarely-resisted temptation to overfit the data – and microeconomists are just as guilty as macroeconomists here. The temptation is strong because an over-fitted model will seem to ‘explain’ more than a ‘true’ model when the data are noisy, so the usual tests for good fit will look better. Deirdre McCloskey and Stephen Ziliak have been pointing out the siren allure of ‘statistical significance’ for ages – it has almost nothing to do with economic meaning – and perhaps The Signal and the Noise will help broadcast the message further.
Finally, I learned a lot from the book. The chapter on how to approach the question of CO2 emissions and climate change is a model of clear thinking. My favourite new fact: members of Congress – with access to lots of company information via lobbyists and the ability to influence companies’ fortunes by legislation – see a profit on their investments that beats the market averages by 5 to 10 per cent a year, “a remarkable rate that would make even Bernie Madoff blush,” as Silver observes.
Anyway, if you haven’t yet read this, go and do so now. The new UK paperback also has a wonderful cover image!
Update: Dan Davies (@dsquareddigest) has gently rebuked me for the paragraph about Bayesian versus frequentist statistics. Via Twitter, he said: “Silver has a really annoying misintepretation of Bayesian vs frequentist which is now becoming commonplace… the paragraph you quote is really confused – NS is a good practical statistician but all over the place on theory & methodology. The (otherwise excellent) book gains nothing from taking a very strong position in someone else’s philosophical debate.” Needlesss to say, I know less than Dan about this debate. This doesn’t change my mind that econ students should be taught the Bayesian approach too, nor the conclusion that the book clearly explains how to do it in practice.
