Secret statistics

Is there a word for ‘serendipity’ that doesn’t have the overtone of being a positive thing? In a horrible coincidence, over the past day or two I’ve been reading the sections of The Theory That Would Not Die by Sharon Bertsch McGrayne concerning using Bayes’ Theorem to help trace missing objects at sea, albeit submarines and nuclear missiles gone astray rather than airplane debris. The theme of the book is the way Bayesian statistics survived two centuries of dismissive treatment by the academic statistics establishment because the techniques are just so useful in an acute situation. The description in today’s newspapers of the way Inmarsat and other investigators combined different sources of information to piece together the path of the vanished plane could slot in to the book very neatly.

The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from

Although an econometrician by training, and trained in the early 1980s so almost wholly in frequentist techniques, I’ve never really understood the ‘either-or’ aspect of the debate about Bayesian techniques. Why would you expect the same technique to be useful in every situation? Why would you throw away information? I was a bit disappointed in The Theory That Would Not Die for two reasons. One is that it actually doesn’t do a very good job of explaining what the alternative approaches actually are – the explanation of Bayes Rule and how to apply it is left until Appendix B. Maybe an editor said it would be too scary for readers to have any actual statistics in the main text.

The second reason is that I ended up not really clear why Bayesian methods were infra dig for so long – the book describes one thing after another conspiring against public acclaim for ‘Bayesianism’ without giving a synthesis. The closest it comes to a theory is that the code-breakers from Bletchley Park on and the military were such heavy users of Bayesian methods that statisticians actually did use them but never talked about them.

Having said that, the book’s a very enjoyable read. There are lots of episodes and characters new to me, and it’s very well written. The problem-solving sections – finding those nuclear warheads gone astray – are gripping. Bayesian statistics seem particularly appropriate to economics, which has relatively little scope for repeated experiments and much historical, context -specific, non-repeatable evidence. This book made me think I ought to find something far more practical to read – recommendations welcome.