At a governors’ meeting this morning, at the primary school where I’m the chair of governors, we discussed the school’s results in the Key Stage 2 SATs (for those who don’t know, it’s the national test children are required to take at the end of their primary school career in England, aged 10-11). The results were excellent. Our philosophy is that we aim for ever-greater attainment so we look for year-on-year improvement, every year.
However, at this time of year I always reflect on the inability of the entire educational, policy and political establishment to understand that the variability observed in a sample will always be larger, the smaller the sample. Ours is currently a small school with under 30 pupils in the final year, and as few as 24 or 25 might be entered for a test subject depending on their circumstances. Each child is worth several percentage points in the results table. Stuff happens, and one child might do better or worse on test day. There is no real meaning to be read into quite large year-on-year changes in the scores for a small school – and more meaning to be read into them the bigger the school. So I’m delighted by the big increases in our results this year (and one would never want to use small sample size as an excuse for a decline without really careful probing); but I take more comfort from the upward trend over three years, and even more from the other data that we look at as governors, and from observing lessons and looking at children’s work in school.
The bigger the sample, the more likely it is that extremes at one and and the other will cancel each other out when you calculate the average. This inverse relationship between variability in the sample and sample size is well-explained in Chapter 1 of Howard Wainer’s excellent book, Picturing the Uncertain World: How to Understand, Communicate and Control Uncertainty through Graphical Display. He gives brilliant examples of people drawing incorrect or at least unproven conclusions from their failure to take account of this relationship. They include the movement in the US supporting smaller schools (as he puts it, billions of dollars are being spent on increasing variance), interpretations of cancer “clusters”, supposed differences in intelligence or attainment between the sexes – there are countless examples.
I don’t think basic statistical literacy is included in the new curriculum for English primary schools – a shame when there’s evidence everywhere of its absence.
Picturing the Uncertain World: How to Understand, Communicate, and Control Uncertainty through Graphical Display