Indeed, no indication is generally given of the selection criteria for panelists. Were they chosen because they came from particularly innovative or high quality districts? Were they chosen because of previously expressed views on programs or resources? Or were they just the subset of a larger invited
group representing those willing to attend a weekend session in exchange for some added pay?
The consultants performing the study seldom know any of the education personnel in the state, so they obviously need to solicit nominations – frequently from the organization commissioning the study. But, since these organizations generally have a direct interest in the outcomes of the study, it seems unlikely that they will produce a random selection of educators to serve on the professional judgment panels. The nature of the selection process ensures that the judgments of any panel cannot be replicated (a fundamental concern of any truly scientific inquiry).
Why would we trust expert panels any more when it comes to educational standards than education spending. The same basic problems exist. The experts do not necessarily represent all or the best views on the matter and may simply be selected by the researchers for their predisposition to support the researcher’s favored conclusion. In other words, we don’t learn anything from these analyses. It is simply a way of disguising and making more impressive the opinion of the researchers for the purpose of political manipulation.
Anyone interested in serious education research should shun professional judgment studies, whether for spending adequacy or for education standards.
This entry was posted on Wednesday, July 21st, 2010 at 6:43 am and is filed under school accountability. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
Given the backgrounds, published writings, and expressed interests of the nominees and new members of the IES Board, this is only going to get worse. They control the research funding in the first place.
[…] Greene’s post, “Expert panels are phony science” Education studies based on the professional judgment of experts is phony science and is usually […]
Of all the creative ways of pitching an echo chamber and its ‘findings’ to the public, these panels may at least deserve some kind of people’s choice award. After all, they’re much more civil than even the most objectively spun, carefully scripted kangaroo courts and show-trials anywhere in the world, and far less theatrical to boot. Kudos!
Then again, to nominate this methodology for the award we might be tempted to assemble a panel of experts. Hmmm… dilemma.
[…] P. Greene’s observations on the problems of relying upon expert panels (read the whole post here): The experts do not necessarily represent all or the best views on the matter and may simply be […]
[…] will cite is basically advocacy research funded by the Gates Foundation, which consists of unscientific panels of experts paid by Gates who agree that Gates’ big project is a good idea. Not even joking […]
Jay P. Greene's Blog 
Given the backgrounds, published writings, and expressed interests of the nominees and new members of the IES Board, this is only going to get worse. They control the research funding in the first place.
[…] Greene’s post, “Expert panels are phony science” Education studies based on the professional judgment of experts is phony science and is usually […]
Of all the creative ways of pitching an echo chamber and its ‘findings’ to the public, these panels may at least deserve some kind of people’s choice award. After all, they’re much more civil than even the most objectively spun, carefully scripted kangaroo courts and show-trials anywhere in the world, and far less theatrical to boot. Kudos!
Then again, to nominate this methodology for the award we might be tempted to assemble a panel of experts. Hmmm… dilemma.
[…] P. Greene’s observations on the problems of relying upon expert panels (read the whole post here): The experts do not necessarily represent all or the best views on the matter and may simply be […]
[…] Expert Panels are Phony Science […]
[…] will cite is basically advocacy research funded by the Gates Foundation, which consists of unscientific panels of experts paid by Gates who agree that Gates’ big project is a good idea. Not even joking […]