The measurement problem

We keep counting education, and it keeps refusing to sit still

Everyone agrees that some education is better than other education. Almost nobody agrees on how we would know. This issue is about that gap, and about what happens inside institutions when we pretend the gap is not there.

Start with the smallest honest observation available. A person walks into a room not knowing something, and some time later walks out of a course, a workshop or a degree changed in ways that neither they nor anyone else can fully describe. Something happened. The question that has occupied educators, funders and administrators for as long as formal education has existed is whether that something can be caught, weighed and compared against the something that happened to a different person in a different room. Every serious attempt to answer that question runs into the same wall: the thing we care about is not directly observable, and everything that is directly observable is only loosely related to it.

This is not a complaint about bureaucracy. It is a description of a genuine epistemic difficulty, and one that education shares with medicine, with parenting and with almost any field where the outcome of interest is a change inside a person. Where education differs is in the length of the delay. A course may pay off in the week after it ends, or in a decision made a decade later, or in a habit of mind so absorbed that the person no longer remembers acquiring it. By the time the effect is visible, the causal chain has been thoroughly contaminated by everything else that person did, met and survived in the meantime.

The Thing Being Measured

The effect arrives long after the invoice does

Education pays out on a schedule nobody designed and nobody can wait for.

Learning is distributed across a life, not stored in a course

Whatever a graduate can do, the course is only part of the reason.

Why the sector settles for proxies, and why each one is contested

A proxy is an honest admission that the real thing is out of reach.

What The Instruments Read

Completion counts finishers, and finishing is not learning

The most quoted figure in the sector answers a question about persistence.

Satisfaction measures expectation, comfort and the moment it was asked

Asking people how it went is worth doing, and worth reading carefully.

Employment and earnings read the labour market as loudly as the teaching

Where graduates end up says a great deal about the economy they entered.

Pressure And The Pointer

Any measure that allocates money will start to shape behaviour

This is the most reliable finding in the whole field, and the least acted upon.

The completion figure that improves by changing who is admitted

The clearest illustration of the problem involves no rule breaking at all.

What gets quietly abandoned when only some things are counted

The unmeasured parts of a course have no advocate in the budget meeting.

Holding Numbers Lightly

Many imperfect measures beat one clean number

Triangulation is not indecision, it is the appropriate response to uncertainty.

Use measurement to raise questions, never to close them

The best thing a figure can do is send somebody to go and look.

Professional judgement cannot be measured away

At the end of every quality system there is a person deciding whether the work is good.

About Quality Education

Quality Education exists because the question of whether education is any good is harder than either its defenders or its critics usually admit, and because the difficulty deserves better than the two available scripts. One script says that quality is obvious to anyone in the room and resists all attempts to capture it. The other says that anything real can be counted and that resistance to counting is resistance to accountability. Both are comfortable positions and both are wrong in ways that cost students something. This publication occupies the harder middle, where measurement is necessary, deeply imperfect, and worth doing with an unusual degree of care about what each figure is actually claiming.

The editorial position is that measurement should be used honestly and held lightly. Honestly means describing what a figure captures rather than what everybody would like it to capture, and refusing the small dishonesty of letting a narrow indicator quietly stand for a broad quality. Lightly means treating results as provisional, expecting them to be wrong in interesting ways, and going to look at the actual teaching whenever they surprise. Neither commitment is a call to stop counting. An institution that measures nothing has simply made itself unaccountable and left the people it serves without any means of showing that they were served badly.

What follows from that position is a set of habits rather than a system. Prefer several imperfect measures to one clean one. Ask learners long after they have left as well as at the moment of departure. Assume that any consequential figure is already shaping behaviour, and design around that assumption instead of hoping otherwise. Keep asking what is being neglected because it is not recorded anywhere. And accept the part that no reform removes: somebody with expertise has to look at the work, form a view, and be prepared to defend it. Quality Education is written for people who find that conclusion realistic rather than disappointing.