Quality Education / Holding Numbers Lightly

Use measurement to raise questions, never to close them

The best thing a figure can do is send somebody to go and look.

There is a specific discipline that separates institutions using measurement well from those merely reporting it. When a figure moves unexpectedly, the well run institution treats it as an alarm rather than a finding. Somebody goes and looks at the teaching. They read the work students produced, sit in on sessions, talk to the people running the course, and try to work out what the figure is reacting to. Often it is something mundane, a timetable change or a shift in cohort. Sometimes it is real. Either way the knowledge came from looking, and the figure only supplied the address.

The alternative is to treat the figure as the conclusion, write the explanation from the desk, and file it. This is faster and produces a tidier report, and it teaches the organisation nothing at all. Over time it produces staff who are fluent in explaining numbers and increasingly disconnected from the activity the numbers refer to, which is a specific and recognisable kind of institutional decay. It shows up first in the vocabulary, when discussions of teaching are conducted entirely in the language of indicators and nobody in the meeting has recently read a piece of student work. By the time that is normal, the reporting has become a closed system talking to itself, and the classroom has stopped supplying it with anything.

Framing measurement as a question generator also changes how it feels to the people measured. A number that triggers an enquiry is survivable, and can even be welcome, because the enquiry might vindicate you. A number that constitutes the verdict is something to be managed rather than understood, and the moment people start managing figures rather than understanding them, the figures have stopped being informative. The effect on candour is just as large. People will volunteer that a course is not working if the admission triggers help, and will defend it to the last if the admission triggers a penalty, and no amount of encouragement to be open will overcome what the system visibly does to people who are. What measurement is for, in the end, is largely decided by how the organisation behaves in the week after a bad result.