Jerry Muller has a piece in the Wall Street Journal that should be required reading for foundation staff and ed reformers who are obsessed with metrics. Here is a snippet:
Metric fixation consists of a set of interconnected beliefs. The first is that it is possible and desirable to replace judgment with numerical indicators of comparative performance based on standardized data. The second is that making such metrics public (transparency) assures that institutions are actually carrying out their purposes (accountability). Finally, there is the belief that people are best motivated by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay for performance) or reputational (rankings).
But not everything that is important is measurable, and much that is measurable is unimportant. Most organizations have multiple purposes, and that which is measured and rewarded tends to become the focus of attention, at the expense of other essential goals. Similarly, many jobs have multiple facets, and measuring only a few of them creates incentives to neglect the rest. Almost inevitably, people become adept at manipulating performance indicators. They fudge the data. They deal only with cases that will improve performance indicators. In extreme cases, they fabricate the evidence.
It’s not that measurement is useless or intrinsically pernicious. The challenge is to specify when performance metrics are genuinely useful—that is, how to have metrics without the malady of metric fixation….
Tools of measurement are most useful for internal analysis by practitioners rather than for external evaluation by the public, which may fail to understand their limits. Such measurement can be used to inform practitioners of their performance relative to their peers, offering recognition to those who have excelled and offering assistance to those who have fallen behind. To the extent that they are used to determine continuing employment and pay, they will be subject to gaming the statistics or outright fraud….
Just because performance measures often have some negative outcomes doesn’t mean that they should be abandoned. They may still be worth using, despite their anticipatable problems. It’s a matter of trade-offs, and that too is a matter of judgment.
With measurement as with everything else, recognizing limits is often the beginning of wisdom. Not all problems are soluble, and even fewer are soluble by metrics. It’s not true, as too many people now believe, that everything can be improved by measurement, or that everything that can be measured can be improved.