Quality Education / Holding Numbers Lightly

Professional judgement cannot be measured away

At the end of every quality system there is a person deciding whether the work is good.

The recurring hope behind measurement reform is that a sufficiently well designed system will not require anyone to exercise discretion. Build the right instrument, the thinking goes, and quality will be determined rather than decided. It has not happened anywhere and there is good reason to think it cannot. Judging whether a piece of student work demonstrates real capability, whether a course is stretching people or merely tiring them, whether an explanation illuminated or confused, requires someone who knows the field to look at the specifics and decide. No aggregation of proxies performs that act.

This is not an argument for trusting whoever happens to be in the chair. Judgement can be lazy, biased, self serving or simply wrong, and the point of a good system is to structure it: multiple judges, work made visible to outsiders, reasons recorded so they can be challenged, and disagreements surfaced rather than smoothed. Structured judgement is very different from unexamined judgement, and the difference is where the real quality work sits. A judgement that has to be written down, shown to a peer who did not make it, and defended when challenged is a very different instrument from one formed privately and announced. Structure does not replace discretion. It makes discretion answerable, which is all that could reasonably be asked of it.

So the conclusion of this issue is deliberately unsatisfying. Keep measuring, because the alternative leaves the least powerful without evidence. Keep several measures, because their disagreements are the useful part. Describe honestly what each one actually captures. Ask people long after they have left. And accept that when the counting is done, someone still has to look at the work and say whether it is any good, and that the quality of that judgement is the quality of the education. Every improvement worth making runs through the people making those judgements, which is why building their capability is the most reliable investment available.