Education studies based on the professional judgment of experts is phony science and is usually nothing more than an exercise in political manipulation. Unfortunately, the recent “study” released by Fordham assigning grades to state standards and the national standards proposed to replace them is an example of this kind of research.
As Rick Hanushek has carefully demonstrated in the context of education spending adequacy lawsuits, the “professional judgment” or “expert panel” method is completely unreliable: He writes:
Indeed, no indication is generally given of the selection criteria for panelists. Were they chosen because they came from particularly innovative or high quality districts? Were they chosen because of previously expressed views on programs or resources? Or were they just the subset of a larger invited
group representing those willing to attend a weekend session in exchange for some added pay?
The consultants performing the study seldom know any of the education personnel in the state, so they obviously need to solicit nominations – frequently from the organization commissioning the study. But, since these organizations generally have a direct interest in the outcomes of the study, it seems unlikely that they will produce a random selection of educators to serve on the professional judgment panels. The nature of the selection process ensures that the judgments of any panel cannot be replicated (a fundamental concern of any truly scientific inquiry).
Why would we trust expert panels any more when it comes to educational standards than education spending. The same basic problems exist. The experts do not necessarily represent all or the best views on the matter and may simply be selected by the researchers for their predisposition to support the researcher’s favored conclusion. In other words, we don’t learn anything from these analyses. It is simply a way of disguising and making more impressive the opinion of the researchers for the purpose of political manipulation.
Anyone interested in serious education research should shun professional judgment studies, whether for spending adequacy or for education standards.
Posted by Jay P. Greene 
Jay P. Greene's Blog 