The largeness of small errors

I enjoyed Oskar Morgenstern’s trenchant observations about the (in)accuracy of economic statistics in On The Accuracy of Economic Observations. Here are a few more examples:

“The idea that as complex a phenomenon as the change in a ‘price level’, itself a heroic theoretical abstraction, could at present be measured to such a degree of accuracy [a tenth of one percent] is simply absurd.”

“It behooves us to pause in order to see what even a 5 percent difference in national income means. Taking the US and assuming a Gross National Product of about 550 billion dollars, this error equals + or – 30 billion dollars. This is more than twice the best annual sales of General Motors…. It is far more than the total annual production of the entire electronics industry in the United States.”

(Updating and relocating this exercise, a 5% error in the £1.7 trillion GDP of the UK would be almost the same size as the entire UK motor industry including the supply chain, more than the total profits of the financial services sector, or about the same as households spend in total on food and drink.)

The errors don’t get the attention they deserve, Morgenstern writes: “Instead, in Great Britain as in the United States and elsewhere, national income statistics are still being taken at their face value and interpreted as if their accuracy compared favourably with that of the measurement of the speed of light.” And he points out that arithmetically, when you are looking at growth rates of figures each measured with some error, even proportionately small errors in the levels turn into large errors in the rate of change. He gives an arithmetical example, of a ‘true’ growth rate of 1.8% being measured as somewhere between -7.9% and +12.5% for measurement errors of up to 5% in the two levels.

It’s interesting that every economist and statistician would acknowledge the errors problem and yet virtually all ignore it. We’ve invested so much that to admit great uncertainty would undermine the totemic value of the figures and the ritual pronouncements about them. At a talk I did at IFN in Stockholm yesterday about GDP, one of the respondents, Karolina Ekholm, State Secretary at the Ministry of Finance, said it made her uneasy that key policy decisions such as cutting government spending depended so much on the output gap – the difference between two imaginary and uncertain numbers. Of course we have to try to measure, and how marvellous it would be if the official statisticians got some extra resources to improve the accuracy in the raw data collection, and yet I think she’s right to be uneasy.

Next on my reading pile: The Politics of Large Numbers: A History of Statistical Reasoning by Alain Desrosières and The Rise of Statistical Thinking 1820-1900 Theodore Porter.

 

EmailLinkedInShare/Save

Why haven’t economic statistics improved?

Party animal that I am, I’ve been spending my spare time in Stockholm this week (where I’ve been doing a couple of events for the Institutet for Naringslivsforskning) reading Oskar Morgenstern’s On The Accuracy of Economic Observations. The 1963 edition of this 1950 book was just sent to me by a friend who had read a draft paper of mine that will be out quite soon, which refers to the book in a quotation from a recent paper by Charles Manski.

As so often when you start researching something, it turns out there’s nothing completely new. Morgenstern wrote trenchantly about some of the things that have been bothering me about the way economists use economic statistics. For example: “A significant difference between the use of data in the natural and social sciences is that in the former the producer of observations is usually also their user.” Economic data on the other hand are collected by many hands, often consist of time series, and are processed by statisticians. Economists rarely think hard enough about their numbers – a problem made all the worse by the easy availability of statistics to download from handy websites and pour into a software package that will churn through regressions and generate test statistics.

Morgenstern also notes the strong incentives many ‘creators’ of economic data have to give misleading responses to survey questions. What’s your income? What price do you charge for this service, oh oligopoly provider? What is the level of your GDP, oh Greek government? “‘Strategic’ considerations play havoc with reliability.” Even if all respondents are well-intentioned, the sampling error, general mistakes in data collection and processing and so on mean we place a ludicrous amount of confidence in any and all economic statistics. “Three or four digits is probably the maximum accuracy of primary data that ever needs to be considered in the vast majority of economic arguments,” he wrote.

Gratifyingly, he likes the technique of spectral analysis of time series, which I did a lot of in my PhD days, because they arrange the data in frequency bands. “There can hardly be any doubt that the powerful new techniques of spectral analysis will put the study of economic fluctuations on a new basis,” the book says. Alas, it didn’t catch on except among a small number of time series anoraks. The chapter I’ve just finished concludes that economics still needs to learn to live with the fundamental importance of errors in the data, and include them in its theories – just as modern physics has. Alas, this too has not caught on among economists. All of which raise the question – why not? Why haven’t economic statistics improved, it seems, since 1960?

My favourite chart at the moment is the Bank of England’s GDP forecast in the Inflation Report. February’s showed a 90% chance that UK annual GDP growth was between 1% and 5% at the start of 2015, taking into account simply past experience of revisions of the data, and not any of the additional potential sources of error. This is the difference between the standard of living doubling in 70 years and 14 years, or in other words vast. Something to bear in mind for anyone making any last minute decisions about how to vote today on the basis of parties’ economic claims.

It’s the power, stupid

What’s not to like about a book with the title The Utopia of Rules: On Technology, Stupidity, and the Secret Joys of Bureacracy. This is a book of essays by Occup Wall Street activist and LSE anthropologist David Graeber, author of the tome Debt: The First 5000 Years. Well, various things. Just like his previous book, I enjoyed the read, strongly disagreed with some points and strongly agreed with others. That makes for quite good entertainment.

The book is a series of essays, ostensibly about bureaucracy, but really about the iniquities of neoliberal capitalism, which is not at all an arena of individual liberty but on the contrary a highly circumscribed system of exploitation. ‘Bureaucracy’ does service as a catch-all term for the set of rules through which the oligarchy in cahoots with the government restrict the options of the great mass of people.

There is certainly something in this portrayal of a society of rich people and large organisations able to act as they please while the great majority experience a highly regulated life. Think about wanting to open a small cafe or a two-person plumbing business, and the extent of certification and inspections that are needed. Or opening a bank account. Graeber writes that we spend hours entangled in paperwork to apply for licences or pay taxes or open accounts, and: “The paperwork we do exists in this sort of in between zone – ostensibly private, but in fact entirely shaped by a government that provides the legal framework, underpins the rules with its courts and all the elaborate mechanisms of enforcement that come with them but – crucially – works closely with the private concerns to ensure the results will guarantee a certain rate of provate profit.” All the more so in these days of outsourcing of state functions such as determining eligibility for benefits. There has been, he says, a gradual fusion of public and private power into “a single entity, rife with rules and regulations.”

So I think there is something in this. Matt Taibbi was onto the same point in Griftopia when he argued that all the people who seem crazy to support Tea Party policies actually do have an arbitrary and unfriendly government controlling their lives.

Where I strongly disagree with Graeber is in his analysis of how to respond. The good part of his argument is about experimenting with other kinds of association and collective organisation, not government but not for profit, perhaps idealistic but worth trying. However, he also writes: “Do not underestimate the importance of sheer physical violence,” drawing on the experience of protests like the 1999 Battle for Seattle riots against the WTO.

He also has a long excursion into technology, arguing that the pace of change peaked in the 1950s or 60s, and that current technological change is not delivering things people might actually want such as flying jet packs. The high hopes of the 1960s as manifested in Star Trek, say, have not materialized. No beaming up, no holodecks etc. Instead, technology has been diverted to military causes. Apart from the fact that the military have always influenced technological research, this is a fact-free essay. I’d want to see some kind of evidence, because if you look at things like either price declines (or how many hours of work are needed to purchase a unit of computer power, or light), or the character of post-1960s discoveries  (medical advances, compelling technologies such as mobile telephony) then there seems to me to be decent evidence for no slowdown. Why can’t computers think yet? Well, give them time. Economic historians like Paul David and David Landes have pointed out how long and variable and unpredictable are the lags between discoveries and their economic and social impact. The dreams and fears about electricity probably peaked more than 60 years before its use had become widespread.

Still, disagreements don’t really matter with a book like this. It’s enjoyable to read something thought provoking with lots of interesting material. I enjoyed the Star Trek riff (“The Federation is Leninism brought to its future absolute cosmic success… a happy conjuncture of material abundance and ideological conformity”) and the potted history of the German Postal system of the 19th century and its influence on future ideal societies, and his review of the third Batman movie and its political interpretation. This book is also less than half the length of Debt, Bureaucracy having a shorter history I suppose. Although actually the one thing this isn’t really about is bureaucracy, but instead another Graeber take on power.

What should rule the world after GDP?

The Little Big Number: how GDP came to rule the world and what to do about it by Dirk Philipsen is out next month. As a GDP-afficionado, I was eager to read it and found it an enjoyable read and generally thought-provoking, although not agreeing with the author on all points.

My main disagreement was more about tone and exaggeration in what is a rather emotional book. There are for example assertions like: “It is safe to say our ancestors, for some 200,000 years prior to the agricultural revolution, engaged in labour only to the very extent to which it helped them survive.” Really? No cave paintings, ancient jewellry, religion? Was Stonehenge essential for survival? Or, because of our “fixation with the accumulation of things”, trying to capture the reality of late 18th century life “by saying that people were poor would represent a fundamental misread.” So were they not less well-nourished than us with more illnesses and shorter lives and many children dying in infancy? Did women (and even men) not spend hours in domestic drudgery? I don’t hesitate to call people in the 18th century poor on this basis; it’s nothing to do with a passion for accumulating cars or handbags. I don’t want more than one washing machine but wouldn’t be without the one – just like Hans Rosling’s mother.

Many readers will like the polemical tone of The Little Big Number; it’s obvious I didn’t. Apart from that, this is an interesting book looking at the history of GDP, its inadequacies as a measure of social welfare, and the environmental consequences of seeking continuing economic growth. It covers some of the same ground as my own GDP: A Brief but Affectionate History, with additional detail. The book identifies the turn to growth rather than levels of national income as a policy aim in the 1950s. Philipsen attributes this to American optimism as the victor in World War 2. I wonder if it isn’t more related to the dawning Cold War, and the need to demonstrate over and over again that the American system was superior to the Soviet one? Geoff Tily (pdf) pinpoints an OECD document of 1961 as the first official reference to targetting growth, so quite a while after the end of the Second World War. (And as ever for anyone who hasn’t read it, I highly recommend Francis Spufford’s Red Plenty.)

The Soviet bloc used Net Material Product as their definition of ‘the economy’. The statistics were highly suspect, even those calculated as a best effort by the CIA; it wasn’t really until the mid-1980s, with glasnost, that the true, sustained weakness of growth in the planned economies became evident. There is a nice anecdote in the book: “Once Soviet archives opened to historical research in the years after 1991, we learned that American GDP figures of the Soviet national economy had been far more accurate than estimates provided by the Soviet Union’s own economic planners, who found it near-impossible to come up with reliable data for their centralized planned economy. What did Soviet planners do? They spied on American economists calculating Soviet GDP, and then incorporated what they learned from their American colleagues into their own planning.” Of course, the Americans were spying on the Soviets to get the basic data. Who knows what the figures meant? But the queues and shortages and poor quality of goods were all real.

The second half of the book looks at the ‘Beyond GDP’ debate, although oddly asserting that nobody paid much attention to the issues between Robert Kennedy’s assassination and Congressional hearings in 2001. This is a little US-centric; the global environmental movement kept the candle burning for alternatives all through that period. Philipsen concurs with the kinds of indicator like the Global Progress Index that show progress coming to a complete halt in the 1970s. This always seems absurd to me: even if that was a real turning point in terms of costs to the environment, which gets a heavy weight in the alternative index, there has been a lot of welfare-enhancing innovation and straightforward growth since the 1970s. It’s not just the invention of tamoxifen or the internet, but the fact that more westerners live in houses with phones, indoor toilets and central heating. Sure, there’s a trade-off with the environment but is that really no progress? Nor is Philipsen interested in the issues about defining either market output or social welfare for the growing category of digital goods that are often free and have strong public good characteristics.

As for what to do about it, the book advocates ditching GDP completely, and having a national dialogue about economic goals based on the principles of sustainability, equity, democratic accountability and economic viability. It isn’t clear how this prescription fits with the several ‘dashboard’ initiatives under way now, which are described here. The dashboard approach is attractive, as is public consultation. However, it isn’t yet clear which dashboard is best or what should go in it – it’s easy to end up with a laundry list of good things, and no analytical framework for assessing outcomes or trade-offs. So the real need now is for the hard grind of the kind that Kuznets, Stone and Meade and their many colleagues sustained through the 1930s and 40s in creating the national accounts to make a GDP-plus set of social accounts practical.

I still think dropping GDP altogether would be a mistake – hence ‘affectionate’. How would a government run fiscal policy or a central bank monetary policy without a nominal GDP figure and some of the national accounts detail? The national accounts statistics as a whole also contain a lot of the material that could furnish a meaningful dashboard, so again it would be a waste of an intellectual asset to ditch all of that.

However, the answer to the underlying question, are we going to move ‘beyond GDP’ is: yes.

Lies, damned lies, statistics, and GDP

On the train to Manchester this morning I finished a terrific book I should really have read long ago. I’m very glad I finally have. It’s Morten Jerven’s Poor Numbers: how we are misled by African development statistics and what to do about it. The title made me think it was only relevant to African statistics, when in fact anybody interested in GDP and national accounts should read it.

The book is short and non-technical, but includes a number of important arguments and examples. Here are the conclusions I take from it:

1. Statistics are the ‘facts’ “states collect to get knowledge about their own economic or social conditions.” Having reliable statistics is a marker of an effective state – “the ability to collect information and taxes are closely related” – and the statistics chosen reflect the power structures and political priorities of states. African states are not effective, their statistics are not reliable. (But this also made me reflect that there is a lot happening in the developed economies for which we have no statistics – and no ability of the state to understand or influence change.)

2. African GDP statistics in the key online databases used by economists – the World Bank, the Penn World Tables, the Maddison database – are inconsistent because of different interpretations of the underlyaing national data, different base years, different price indices. The sources even rank African countries differently in terms of GDP per capita. Econometric work will get different results depending which is used.” Jerven argues that economists need to have a much more detailed understanding of both the data they download and the specifics of individual countries’ circumstances to be able to interpret the numbers.

3. The underlying national level data are unreliable because of a lack of resources and statistical capacity. Surveys are rarely carried out, there is much guesswork, base year changes happen too infrequently, there is political influence.

4. 2 and 3 together mean little reliance can be placed on the standard cross-country regressions using the standard data sets. “These problems undermine any general conclusions drawn about what stimulates or hinders economic development in Africa.’

5. The standard national accounts concepts don’t apply well to developing economies with a large informal sector. The distinction between production and consumption or working and not-working is not as clear. (And may be becoming less clear in developed economies too, as technology blurs these boundaries and working patterns change.)

The book argues that the standard outline of African growth – a dismal 1970s, a better outcome post- structural adjustment remedies, and a recent acceleration in growth is largely ‘illusory’. The recent uplift in particular comes from the World Bank/IMF splicing recent rebased GDP figures onto an earlier series, as Jerven describes it. He argues that more data needs to be collected, in regular surveys, to enable both good statistics and an effective state knowing what is happening in the economy and to its tax base. He also argues strongly for greater transparency by national statistical offices but especially by the international agencies such as the World Bank and IMF, whose say-so determines the methods used to create the statistics and the world’s interpretation of what is happening in each economy.

“Accounting for the national economy is fundamental for government accountability. Without reliable macro data, political transparency is hard to imagine. …. Numbers are too important to be ignored and the problems surrounding the production and dissemination of numbers too serious to be dismissed.”

So don’t make my initial mistake of thinking this is a bit of a specialist book. It’s a fascinating and important read.