Quality Education / Holding Numbers Lightly

Many imperfect measures beat one clean number

Triangulation is not indecision, it is the appropriate response to uncertainty.

The appeal of a single figure is that it settles arguments. That is also its danger, because the arguments it settles are often ones that deserved to continue. A basket of measures behaves differently. It rarely produces a clean verdict, and it does something more useful instead: it disagrees with itself. When completion is strong and later reports from former students are lukewarm, when learners speak well of a course whose work samples look thin, the contradiction is telling you where to look. A single measure never contradicts itself and therefore never points anywhere except at its own value.

Keeping several measures also makes distortion harder. Improving one figure by adjusting who is admitted or what is offered will usually push another figure the wrong way, and the mismatch is visible to anyone reading the set together. No basket is immune, and a determined focus on the whole set can still drift, but the effort required is much greater and the tracks it leaves are much more obvious to anyone who bothers to read the readings side by side rather than one at a time. This is the practical case for redundancy in a quality system. Overlapping and partly conflicting evidence is harder to game precisely because it is harder to summarise, and the awkwardness that administrators complain about is the same property that makes it robust.

The cost is that nobody can be ranked cleanly, and institutions and funders both find that frustrating. It is worth being direct about the trade. A number clean enough to rank with is almost always a number narrow enough to mislead with. Choosing the messier set is choosing to know something imprecise about the right thing rather than something precise about the wrong one. Most institutions, asked to state the choice that plainly, would make the same decision. Very few systems are actually built that way, because ranking is what funders and prospective students want, and the demand for a league position is far more persistent than the evidence that would justify one.