Expert Panels are Phony Science

Education studies based on the professional judgment of experts is phony science and is usually nothing more than an exercise in political manipulation.  Unfortunately, the recent “study” released by Fordham assigning grades to state standards and the national standards proposed to replace them is an example of this kind of research.

As Rick Hanushek has carefully demonstrated in the context of education spending adequacy lawsuits, the “professional judgment” or “expert panel” method is completely unreliable:  He writes:

Indeed, no indication is generally given of the selection criteria for panelists. Were they chosen because they came from particularly innovative or high quality districts? Were they chosen because of previously expressed views on programs or resources? Or were they just the subset of a larger invited  

group representing those willing to attend a weekend session in exchange for some added pay?

The consultants performing the study seldom know any of the education personnel in the state, so they obviously need to solicit nominations – frequently from the organization commissioning the study. But, since these organizations generally have a direct interest in the outcomes of the study, it seems unlikely that they will produce a random selection of educators to serve on the professional judgment panels. The nature of the selection process ensures that the judgments of any panel cannot be replicated (a fundamental concern of any truly scientific inquiry).

Why would we trust expert panels any more when it comes to educational standards than education spending.  The same basic problems exist.  The experts do not necessarily represent all or the best views on the matter and may simply be selected by the researchers for their predisposition to support the researcher’s favored conclusion.  In other words, we don’t learn anything from these analyses.  It is simply a way of disguising and making more impressive the opinion of the researchers for the purpose of political manipulation.

Anyone interested in serious education research should shun professional judgment studies, whether for spending adequacy or for education standards.

About these ads

5 Responses to Expert Panels are Phony Science

  1. robin says:

    Given the backgrounds, published writings, and expressed interests of the nominees and new members of the IES Board, this is only going to get worse. They control the research funding in the first place.

  2. [...] Greene’s post, “Expert panels are phony science” Education studies based on the professional judgment of experts is phony science and is usually [...]

  3. Daniel Earley says:

    Of all the creative ways of pitching an echo chamber and its ‘findings’ to the public, these panels may at least deserve some kind of people’s choice award. After all, they’re much more civil than even the most objectively spun, carefully scripted kangaroo courts and show-trials anywhere in the world, and far less theatrical to boot. Kudos!

    Then again, to nominate this methodology for the award we might be tempted to assemble a panel of experts. Hmmm… dilemma.

  4. [...] P. Greene’s observations on the problems of relying upon expert panels (read the whole post here): The experts do not necessarily represent all or the best views on the matter and may simply be [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,623 other followers