Quality Education / The Thing Being Measured

We keep counting education, and it keeps refusing to sit still

Everyone agrees that some education is better than other education. Almost nobody agrees on how we would know. This issue is about that gap, and about what happens inside institutions when we pretend the gap is not there.

Start with the smallest honest observation available. A person walks into a room not knowing something, and some time later walks out of a course, a workshop or a degree changed in ways that neither they nor anyone else can fully describe. Something happened. The question that has occupied educators, funders and administrators for as long as formal education has existed is whether that something can be caught, weighed and compared against the something that happened to a different person in a different room. Every serious attempt to answer that question runs into the same wall: the thing we care about is not directly observable, and everything that is directly observable is only loosely related to it.

This is not a complaint about bureaucracy. It is a description of a genuine epistemic difficulty, and one that education shares with medicine, with parenting and with almost any field where the outcome of interest is a change inside a person. Where education differs is in the length of the delay. A course may pay off in the week after it ends, or in a decision made a decade later, or in a habit of mind so absorbed that the person no longer remembers acquiring it. By the time the effect is visible, the causal chain has been thoroughly contaminated by everything else that person did, met and survived in the meantime.

So the sector does what any sensible field does when the thing it wants is out of reach. It measures something else. It measures the things that leave a trace: who enrolled, who stayed, who finished, who said they were happy, who found work, who earned what. These traces are real. They are collected honestly, at real cost, by people trying to do a decent job. The problem is not that the traces are fake. The problem is that each one answers a narrower question than the one being asked, and the gap between the narrow question and the broad one is where nearly all the trouble lives.

Consider what happens when the narrow answer becomes consequential. Once a figure decides funding, reputation, ranking or renewal, everybody who is judged by it starts, quite rationally, attending to it. Not deceitfully. Attending to it. They ask what moves the number and they do more of that, and the sector eventually notices that the number has risen while the underlying thing may or may not have moved at all. This effect is so well documented across so many fields that it should be treated as a law of administration rather than a scandal. It is what measurement does when it is loaded with consequence, and it is entirely predictable from the incentives.

None of this argues for giving up. An institution that measures nothing is not humble, it is unaccountable, and the students who are poorly served by it have no way to say so and no evidence to say it with. Refusing to count is a decision to leave the field to whoever shouts loudest, and the people who shout loudest have rarely been the ones with the least power. The honest position is harder than either enthusiasm or refusal, and it is the position this issue argues for throughout.

That position is roughly this. Use several imperfect measures rather than one clean one, because the disagreements between them are more informative than any of them alone. Treat every figure as the beginning of an enquiry rather than the end of one, and go and look at the teaching when the figure surprises you. Ask people about their education long after they have stopped being flattered or aggrieved by it. And accept the conclusion that no amount of counting will remove: somewhere in the process, a person with expertise has to look at the work and form a judgement, and that judgement is the load bearing part.