What Doesn’t Work Clearinghouse

October 4, 2010

The U.S. Department of Education’s “What Works Clearinghouse” (WWC) is supposed to adjudicate the scientific validity of competing education research claims so that policymakers, reporters, practitioners, and others don’t have to strain their brains to do it themselves.  It would be much smarter for folks to exert the mental energy themselves rather than trust a government-operated truth committee to sort things out for them.

WWC makes mistakes, is subject to political manipulation, and applies arbitrary standards.  In short, what WWC says is not The Truth.  WWC is not necessarily less reliable than any other source that claims to adjudicate The Truth for you.  Everyone may make mistakes, distort results, and apply arbitrary standards.  The problem is that WWC has the official endorsement of the U.S. Department of Education, so many people fail to take their findings with the same grains of salt that they would to the findings of any other self-appointed truth committee.  And with the possibility that government money may be conditioned on WWC endorsement, WWC’s shortcomings are potentially more dangerous.

I could provide numerous examples of WWC’s mistakes, political manipulation, and arbitrariness, but for the brevity of a blog post let me illustrate my point with just a few.

First, WWC was sloppy and lazy in its recent finding that the Milwaukee voucher evaluation, led by my colleagues Pat Wolf and John Witte, failed to meet “WWC evidence standards” because “the authors do not provide evidence that the subsamples of voucher recipients and public school comparison students analyzed in this study were initially equivalent in math and reading achievement.” WWC justifies their conclusion with a helpful footnote that explains: “At the time of publication, the WWC had contacted the corresponding author for additional information regarding the equivalence of the analysis samples at baseline and no response had been received.”

But if WWC had actually bothered to read the Milwaukee reports they would have found the evidence of equivalence they were looking for.  The Milwaukee voucher evaluation that Pat and John are leading has a matched-sample research design.  In fact, the research team produced an entire report whose purpose was to demonstrate that the matching had worked and produced comparable samples. In addition, in the 3rd Year report the researchers devoted an entire section (see appendix B) to documenting the continuing equivalence of the matched samples despite some attrition of students over time.

Rather than reading the reports and examining the evidence on the comparability of the matched samples, WWC decided that the best way to determine whether the research met their standards for sample equivalence was to email John Witte and ask him.  I guess it’s all that hard work that justifies the multi-million dollar contract Mathematica receives from the U.S. Department of Education to run WWC.

As it turns out, Witte was traveling when WWC sent him the email.  When he returned he deleted their request along with a bunch of other emails without examining it closely.  But WWC took Witte’s non-response as confirmation that there was no evidence demonstrating the equivalence of the matched samples.  WWC couldn’t be bothered to contact any of the several co-authors.  They just went for their negative conclusion without further reading, thought, or effort.

I can’t prove it (and I’m sure my thought-process would not meet WWC standards), but I’ll bet that if the subject of the study was not vouchers, WWC would have been sure to read the reports closely and make extra efforts to contact co-authors before dismissing the research as failing to meet their standards.  But voucher researchers have grown accustomed to double-standards when others assess their research.  It’s just amazingly ironic to see the federally-sponsored entity charged with maintaining consistent and high standards fall so easily into their own double-standard.

Another example — I served on a WWC panel regarding school turnarounds a few years ago.  We were charged with assessing the research on how to successfully turnaround a failing school.  We quickly discovered that there was no research that met WWC’s standards on that question.  I suggested that we simply report that there is no rigorous evidence on this topic.  The staff rejected that suggestion, emphasizing that the Department of Education needed to have some evidence on effective turnaround strategies.

I have no idea why the political needs of the Department should have affected the truth committee in assessing the research, but it did.  We were told to look at non-rigorous research, including case-studies, anecdotes, and our own experience to do our best in identifying promising strategies.  It was strange — there were very tight criteria for what met WWC standards, but there were effectively no standards when it came to less rigorous research.  We just had to use our professional judgment.

We ended up endorsing some turnaround strategies (I can’t even remember what they were) but we did so based on virtually no evidence.  And this was all fine as long as we said that the conclusions were not based on research that met WWC standards.  I still don’t know what would have been wrong with simply saying that research doesn’t have much to tell us about effective turnaround strategies, but I guess that’s not the way truth committees work.  Truth committees have to provide the truth even when it is false.

The heart of the problem is that science has never depended on government-run truth committees to make progress.  It is simply not possible for the government to adjudicate the truth on disputed topics because the temptation to manipulate the answer or simply to make sloppy and lazy mistakes is all too great.  This is not a problem that is particular to the Obama Administration or to Mathematica.  My second example was from the Bush Administration when WWC was run by AIR.

The hard reality is that you can never fully rely on any authority to adjudicate the truth for you.  Yes, conflicting claims can be confusing.  Yes, it would be wonderfully convenient if someone just sorted it all out for us.  But once we give someone else the power to decide the truth on our behalf, we are prey to whatever distortions or mistakes they may make.  And since self-interest introduces distortions and the tendency to make mistakes, the government is a particularly untrustworthy entity to rely upon when it comes to government policy.

Science has always made progress by people sorting through the mess of competing, often technical, claims.  When official truth committees have intervened, it has almost always hindered scientific progress.  Remember that  it was the official truth committee that determined that Galileo was wrong.  Truth committees have taken positions on evolution, global warming, and a host of other controversial topics.  It simply doesn’t help.

We have no alternative to sorting through the evidence and trying to figure these things out ourselves.  We may rely upon the expertise of others in helping us sort out competing claims, but we should always do so with caution, since those experts may be mistaken or even deceptive.  But when the government starts weighing in as an expert, it speaks with far too much authority and can be much more coercive.  A What Works Clearinghouse simply doesn’t work.

8 Comments | methodology, research reports, vouchers | Tagged: , , , , | Permalink
Posted by Jay P. Greene

The Federally Tilted Playing Field

March 3, 2010

(Guest post by Greg Forster)

In the spirited debate Jay has started on whether the feds should fund research analysis, here’s an angle nobody seems to have picked up – where the money comes from.

Most of Jay’s critics are pointing out that all research has to be funded by somebody, and lots of privately funded research is clearly biased, too – some of it as much as federal research, or more so.

In his replies, Jay has focused on what he thinks is public ignorance of the bias introduced in federally funded research. People assume that research funded by curricular development companies is biased, but they don’t assume that about government research. I’ll agree that that’s a problem to some extent, but I don’t think it’s the biggest problem.

The biggest problem is that government funding isn’t like private funding. Private funders control business interests, but government controls public policy. That provides a ton more potential for corrupting influence (even unconsciously). Which is stronger, the temptation to kiss up to a man who can make you rich or the temptation to kiss up to a man who can make you rich or destroy your whole life, or just about any outcome in between?

Add the fact that government can collect funds coercively from the entire economy and has far greater agency problems in its bureaucracy. This means government largesse is likely to be much more lavish than private largesse and is much less likely to come under any form of scrutiny, whether for cost/benefit purposes or accountability.

With private funders, everyone is equally free to fund the research they want done, and everyone is equally free to judge the results, including taking things with a grain (or a boulder) of salt if you don’t trust the source. Federal research funding tilts the playing field.

Leave a Comment » | methodology, politics | Tagged: | Permalink
Posted by Greg Forster

Feds And Research Shouldn’t Mix

March 2, 2010

 

As head of a department that has received and may wish to continue receiving federal research funds, it is completely contrary to my self-interest to say this:  the federal government should not be in the business of conducting or funding education policy research.  The federal government should facilitate research by greatly expanding the availability of individual student data sets stripped of identifying information.  But the federal government is particularly badly positioned to conduct or fund analyses based on those data.

The reasons for keeping the federal government out of education policy research should be obvious to everyone not blinded by the desire to keep eating at the trough.  First, the federal government develops and advocates for particular education policies, so it has a conflict of interest in evaluating those policies.  Even when those evaluations are outsourced to supposedly independent evaluators, they are never truly independent.  The evaluation business is a repeat-play game, so everyone understands that they cannot alienate powerful political forces too much without risking future evaluation dollars.  The safe thing to conclude in those circumstances is that the evidence is unclear about the effectiveness of a policy but future research is needed, which, not surprisingly, is what many federally funded evaluations find.

Unfortunately, political influence in education policy research is often more direct and explicit than the implicit distortions of a repeat-play game.  Every federally funded evaluation with which I am familiar has been subject to at least some, subtle political influence. 

I can’t mention most without breaking confidences, but I can briefly describe my own experience with a What Works Clearinghouse (WWC) panel on which I served (which was managed by a different firm than the one that currently manages WWC).  On that panel we were supposed to identify what was known from the research literature on how to turn around failing schools.  As we quickly discovered, there was virtually nothing known from rigorous research on how to successfully turn around failing schools.  I suggested that we should simply report that as our finding — nothing is known.  But we were told that the Department of Education wouldn’t like that  and we had to say something about how to turn around schools.  I asked on what basis we would draw those conclusions and was told that we should rely on our “professional judgment” informed by personal experience and non-rigorous research.  So, we dutifully produced a report that was much more of a political document than a scientific one.  We didn’t know anything from science about how to turn around schools, but we crafted a political answer to satisfy political needs.

In addition to being politically influenced, federally funded research is almost always overly expensive.  The cost of federal education policy research is many-fold more expensive than that research has to be.  There are several federal evaluations where the cost of the evaluation rivals the annual cost of the program being evaluated.

Beyond being politically distorted and cost-inefficient, a whole lot of federally funded research is really awful.  In particular, I am thinking of the work of the federally funded regional research labs.  For every useful study or review they release, there must be hundreds of drek.  The regional labs are so bad that the Department of Education has been trying to eliminate them from their budget for years.  But members of Congress want the pork, so they keep the regional labs alive.

Being politically distorted, cost-inefficient, and often of low quality is not a good combination.  Let’s get the feds out of the research business.  They can still play a critically important role of providing data sets to the research community, but they should not be funding evaluations or research summaries.  We need the feds to help with data because privacy laws are too great of a barrier for individual researchers.  But once basic data is available, the cost of analyzing the data should be quite low — just the time of the researchers and some computer equipment, perhaps supplemented with additional field data collection.  And if there is no “official” evaluation or “official” summary of the research literature, the research community is free to examine the evidence and draw its own conclusions.  Yes, there will be disagreement and messiness, but the world is uncertain and messy.  Freedom is uncertain and messy.  The solution is not to privilege over-priced, often lousy, politically driven federally funded work.

(edited for typos)

18 Comments | methodology, politics | Tagged: | Permalink
Posted by Jay P. Greene

Head Start Basically Has No Effect

January 13, 2010

As I described last week, the Department of Health and Human Services has been sitting on an evaluation of the Head Start government run pre-school program.  Well, today the study was released (and it’s not even a Friday!). (Update: HHS moved the link to the study to here.)

As the leaks suggested, the study found virtually no lasting effects to participation in Head Start.  The study used a gold-standard, random assignment design and had a very large nationally representative sample.  This was a well done study (even if it mysteriously took more than 3 years after data collection was complete to release the results).

For students who were randomly assigned to Head Start or not at the age of 4, the researchers collected 19 measures of cognitive impacts at the end of kindergarten and 22 measures when those students finished 1st grade.  Of those 41 measures only 1 was significant and positive.  The remaining 40 showed no statistically significant difference.  The one significant effect was for receptive vocabulary, which showed no significant advantage for Head Start students after kindergarten but somehow re-emerged at the end of 1st grade.

The study used the more relaxed p< .1 standard for statistical significance, so we could have seen about 4 significant differences by chance alone and only saw 1.  That positive effect had an effect size of .09, which is relatively modest.

For students randomly assigned to Head Start or not at the age of 3, the researchers also collected 41 measures of lasting cognitive effects.  This time they found 2 statistically significant positive effects and 1 statistically significant negative effect.  For the students who began at age 3 they showed a .08 effect size benefit from Head Start in oral comprehension after first grade and a .26 effect size benefit in spanish vocabulary after kindergarten but a .19 effect size decline in math ability at the end of kindergarten.  Again, 38 of the 41 measures of lasting effects showed no difference and the few significant effects (which could be produced by chance) showed mixed results.

I think it is safe to say from this very rigorous evaluation that Head Start had no lasting effect on the academic preparation of students.

The study also measured lasting effects on student behavior and emotion as well as the skills of parents.  Again, the effects were largely null and the few significant differences were in mixed directions.  The few positive effects from these categories were from parent reports and the few negative tended to come from teacher reports.

The long and short of it is that the government has a giant and enormously expensive pre-school program that has made basically no difference for the students who participate in it.  And folks are proposing that we expand government pre-school to include all students.  Those same folks have some bridges they’d like to sell.

(edited for clarity)

7 Comments | methodology | Tagged: , | Permalink
Posted by Jay P. Greene

Just About Everything is Endogenous

September 30, 2009

A common technique in analyses of education policies (and popularized in the book, Freakonomics) has suffered a setback recently.  The technique attempts to correct for endogeneity, which occurs when your dependent variable is causing one of your independent variables rather than simply the other way around.

It’s probably best to explain this with an example.  Let’s say you want to know how the number of police officers in a city affects the crime rate.  In this example the dependent variable is the crime rate and the independent variable is the number of police officers.  That is, you are trying to explain how the size of the police force causes crime rates to be high or low.

The trouble is that the causal arrow also goes in the other direction.  The crime rate affects the size of the police force because cities with a lot of crime may decide to hire a lot of police officers.  So, the number of police officers is endogenous to the crime rate.  

That endogeneity could produce some odd results if we didn’t do anything to correct it.  We might find that the number of police officers causes crime rates to be higher when it might really be the case that the size of the police force reduces crime but high crime rates cause larger police forces.

This kind of problem comes up quite often in econometric analyses in general and in particular in evaluations of education policies.  So, it was a great a thing that University of Chicago economist James Heckman developed a technique for unravelling these circular relationships and correcting for endogeneity bias.  Basically, the technique uses some exogenous variable to predict the independent variable without bias.

Again, it’s probably easiest to explain with an example.  If we can find something that predicts the number of police officers that has nothing to do with the crime rate, then we can come up with an unbiased estimated of the number of police officers.  We can then use that unbiased estimate of how many police officers there would be (independent of the crime rate) to predict the crime rate.  In theory the technique works great.  Heckman won the Nobel Prize in economics for developing it.

The tricky part is coming up with a truly exogenous instrument (something that predicts the independent variable but has no relationship with the dependent variable).  The only obviously exogenous instrument is chance itself.  An example of that kind of instrument can be found in analyses of the effect of using a voucher on the student achievement of students who actually attend a private school when the vouchers are awarded by lottery.  Those analyses use whether a student won the lottery or not to predict whether a student attended a private school and then used that unbiased estimate of whether a student attended a private school to predict the effect of private schooling on student achievement. 

Whether a student won the lottery is purely a matter of chance and so is completely unrelated to student achievement, but it is predictive of whether a student attends a private school.  It is a perfectly exogenous instrument.

The problem is that other than lotteries, it isn’t always clear that the instruments used are truly exogenous.  Even if we can’t think of how things may be related, they may well be.

A perfect example of this — and it is one that raises questions about how exogenous all instruments other than lotteries truly are — was recently described in the Wall Street Journal having to do with date of birth.  The date during the year when babies are born has long been thought to be essentially random and has been used as an exogenous instrument in a variety of important analyses, including a seminal paper in 1991 by Josh Angrsit and Alan Krueger on the effects of educational attainment on later life outcomes. 

Since states have compulsory education laws require that students stay in school until a certain age, babies born earlier in the year reach that age at a lower grade and can drop out having attained less education.  By comparing those born earlier in the year to those born later, which they believed should have nothing to do with later life outcomes, they were able to make claims about how staying in school longer affected income, etc…

But new work by Kasey Buckles and Daniel Hungerman at the University of Notre Dame suggests that the month and day of birth is not really exogenous to life outcomes.  As it turns out, babies born in January are more likely to be born to unwed, less educated, and low income mothers than babies born later in the year.  The difference is not huge, but it is significant.  And since this variable is not exogenous, perhaps some or all of the effect of attainment Angrist and Krueger observed is related to this relationship between date of birth and SES, not truly attributable to attainment.

And if birth order is not random when we all assumed it was, what other instruments in these analyses are also not truly exogenous but we just don’t know how yet?  It’s a potentially serious problem for these analyses.

6 Comments | methodology | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Schoolhouses, Courthouses, and Statehouses

August 9, 2009

The new book from Rick Hanushek and Alfred Lindseth, Schoolhouses, Courthouses, and Statehouses, is a remarkably comprehensive and accessible review of K-12 education reform strategies.  It’s a must-read for education policymakers, advocates, and students — at both the graduate and undergraduate levels.  Even experienced researchers will find this to be an essential reference, given its broad sweep and extensive citations.

The book basically makes four arguments.  First it establishes how important K-12 educational achievement really is to economic success and how far we are lagging our economic competitors in this area.  Second, it demonstrates the dominance and utter failure of input-oriented reform strategies, including across-the-board spending increases and class-size reductions.  Third, it describes how the court system has perpetuated failed input-reform strategies after having bought intellectually dishonest methods of calculating how much spending schools really need.  And fourth, it makes the case for reform strategies that involve “performance-based funding,” including merit pay, accountability systems, and choice.

None of these arguments is original to this book.  But to the extent that others have made these arguments, they have drawn heavily on Rick Hanushek’s research.  In this book you get to hear it directly from the source and you get to hear it all so persuasively and completely.

If I have any complaint about the book it is that they are too restrained in their criticisms of the methods by which adequate school spending has been determined and the “researchers” who have developed and profited from those methods.  These fraudulent analyses have justified court decisions ordering billions of dollars to be taken from taxpayers and blown ineffectively in schools.  And the quacks promoting these methods have made millions of dollars in consulting fees in the process.

Those methods include the “professional judgment approach,” which essentially consists of gathering a group of educators and asking them how much money they think they would need to provide an “adequate” education,  Naturally, they need flying saucers, ponies, and a laser tag arena to ensure an adequate education. 

Another method is the “evidence-based approach,” which selectively reads the research literature to identify what it claims are effective educational practices.  It then sums the cost of those practices while paying no attention to how many are really necessary for an adequate education or whether any of them are really cost-effective.

There is also the “successful schools approach,” which looks at how much money a typical successful school spends and calls for all schools to spend at least that much.  This of course ignores the fact that many successful schools spend less than the typical amount and are still successful.  One would have thought it impossible for them to be successful with less money than that deemed necessary to succeed. 

And lastly, there is the “cost-function approach.”  This approach takes the conventional finding that higher spending, controlling for other factors, has little to no relationship with student achievement, and then turns that finding on its head.  It does this by switching  the dependent variable from student achievement to cost.  The question then becomes: how much each unit of achievement contributes to school costs.  Switching the dependent variable does nothing to change the lack of relationship between spending and achievement.  If you hide behind enough statistical mumbo-jumbo you can hope that the courts won’t notice that there is still virtually no relationship between spending and achievement controlling for other factors.

The Hanushek and Lindseth book lays all of this out (see especially chapter 7), but they are remarkably restrained in denouncing these approaches and the people who cynically profit from them.  I don’t think we should be so restrained.  The promoters of this snake oil are often university professors with sterling national reputations.  They’ve cashed in those reputations to market obviously flawed methods.  We shouldn’t let them do this without paying a significant price in their reputation.

The University of Southern California’s Larry Picus, and the University of Wisconsin’s Allan Odden, are both past presidents of the well-respected American Education Finance Association.  They shouldn’t be able to sell the “evidence-based approach” to 5 states for somewhere around $3 million without people pointing and laughing when they show up at conferences.

I know that Rick Hanushek and Alfred Lindseth are too professional and scholarly to call these folks frauds, but I’m not sure what else one could honestly call them.  Rick comes close in his Education Next article on these school funding adequacy consultants, entitled, “The Confidence Men.”  But in this book,perhaps with the tempered emotions of his co-author,  he adopts a more restrained tone.  Perhaps this is all for the best because the book maintains the kind of scholarly temperament that strengthens its persuasiveness to those who would be more skeptical. 

This has been a great year for education reform books.  Schoolhouses, Courthouses, and Statehouses joins Terry Moe and John Chubb’s Liberating Learning, released earlier this summer, as members of the canon of essential education reform works.

4 Comments | methodology, school accountability, school funding | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Why Random Assignment is Important

July 2, 2009

Bill Evers has an excellent post over on his Ed Policy blog about how unreliable observational studies can be and how important it is to test claims with random-assignment research designs. 

Observational studies (sometimes called epidemiological or quasi-experimental studies) do not randomly assign subjects to treatment or control conditions or use a technique that approximates random-assignment (like regression discontinuity).  Instead they simply compare people who have self-selected or otherwise been assigned to receive a treatment to people who haven’t received that treatment, controlling statistically for observed differences between the two groups.  The problem is that unobserved factors may really be causing any differences between the two groups, not the treatment.  This is especially a problem when these unobserved factors are strongly related to whatever led to some people getting the treatment and others not. 

The solution to this problem is random assignment.  If subjects are assigned by lottery to receive a treatment or not, then the only difference between the two groups, on average, is whether they received the treatment.  The two groups should otherwise be identical because only chance distinguishes them.  Any differences between the two groups over time can be attributed to the treatment with high confidence.

If you don’t believe that research design makes a big difference, consider this table that Bill Evers provides on how much results change in the field of nutrition when random assignment (or clinical) studies are done to check on claims made by observational studies:

If we want to avoid the educational equivalent of quack medicine, we really need more random-assignment studies and we need to give the random-assignment studies we already have significantly greater weight when forming policy conclusions.

As I’ve written before, we have 10 random-assignment studies on the effects of vouchers on students who participate in those programs. Six of those ten studies show significant academic benefits for the average student receiving a vouchers and three studies show significant academic benefits for at least one major sub-group of students.  One study finds no significant effects.  

I believe that there are more random-assignment studies on vouchers than on any other educational policy and there are certainly more studies with positive results.  The depth of positive, rigorous studies on voucher participant effects is worth keeping in mind each time some new observational or (even descriptive) study comes out on school choice, including the most recent report from Florida.  Our opinion shouldn’t be based entirely on the latest study, especially if it lacks the rigorous design of several earlier studies.

3 Comments | methodology, vouchers | Tagged: , | Permalink
Posted by Jay P. Greene

The Professional Judgment Un-Dead

March 25, 2009

It’s time we drive a stake through the heart of “professional judgment” methodologies in education.  Unfortunately, the method has come back from the grave in the most recent Fordham report on regulating vouchers in which an expert panel was asked about the best regulatory framework for voucher programs.

The methodology was previously known for its use in school funding adequacy lawsuits.  In those cases a group of educators and experts was gathered to determine the amount of spending that is required to produce an adequate education.  Not surprisingly, their professional judgment was always that we need to spend billions and billions (use Carl Sagan voice) more than we spend now.  In the most famous use of the professional judgment method, an expert panel convinced the state courts to order the addition of $15 billion to the New York City school system — that’s an extra $15,000 per student.

And advocates for school construction have relied on professional judgment methodologies to argue that we need $127 billion in additional spending to get school facilities in adequate shape.  And who could forget the JPGB professional judgment study that determined that this blog needs a spaceship, pony, martinis, cigars, and junkets to Vegas to do an adequate job?

Of course, the main problem with the professional judgment method is that it more closely resembles a political rather than a scientific process.  Asking involved parties to recommend solutions may inspire haggling, coalition-building, and grandstanding, but it doesn’t produce truth.  If we really wanted to know the best regulatory framework, shouldn’t we empirically examine the relationship between regulation and outcomes that we desire? 

Rather than engage in the hard work of collecting or examining empirical evidence, it seems to be popular among beltway organizations to gather panels of experts and ask them what they think.  Even worse, the answers depend heavily on which experts are asked and what the questions are. 

For example, do high stakes pressure schools to sacrifice the learning of certain academic subjects to improve results in others with high stakes attached?  The Center for Education Policy employed a variant of the professional judgment method by surveying school district officials to ask them if this was happening.  They found that 62% of districts reported an increase in high-stakes subjects and 44% reported a decrease in other subjects, so CEP concluded that high-stakes was narrowing the curriculum.  But the GAO surveyed teachers and found that 90% reported that there had not been a change in time spent on the low stakes subject of art.  About 4% reported an increase in focus on art and 7% reported a decrease.  So the GAO, also employing the professional judgment method, gets a very different answer than CEP.  Obviously, which experts you ask and what you ask them make an enormous difference.

Besides, if we really wanted to know about whether high stakes narrow the curriculum, shouldn’t we try to measure the outcome directly rather than ask people what they think?  Marcus Winters and I did this by studying whether high stakes in Florida negatively impinged on achievement in the low-stakes subject of science.  We found no negative effect on science achievement from raising the stakes on math and reading.  Schools that were under pressure to improve math and reading results also improved their science results.

Even if you aren’t convinced by our study, it is clear that this is a better way to get at policy questions than by using the professional judgment method.  Stop organizing committees of selected “experts” and start analyzing actual outcomes.

5 Comments | methodology, school accountability, vouchers | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Looking Abroad for Hope

November 5, 2008

hope

HT despair.com. Looking for a Christmas idea to suit the new reality? Why not a despair.com gift certificate – “For the person who has everything, but still isn’t happy.”

(Guest post by Greg Forster)

Looking around for something to give me hope this morning, I find the best place to turn (for today, at least) is outside the U.S. Specifically, I turn to the recently released study in Education Next by Martin West and Ludger Woessmann finding that around the world, private school enrollment is associated with improved educational outcomes in both public and private schools, as well as lower costs.

Well-informed education wonks will say, “duh.” A large body of empirical research has long since shown, consistently, that competition improves both public school and private school outcomes here in the U.S., while lowering costs. And the U.S. has long been far, far behind the rest of the world in its largely idiosyncratic, and entirely irrational, belief that there’s somthing magical about a government school monopoly.

And private school enrollment is an imperfect proxy for competition. It’s OK to use it when it’s the best you’ve got. I’ve overseen production of some studies at the Friedman Foundation that used it this way, and I wouldn’t have done that if I didn’t think the method were acceptable. However, that said, it should be remembered that some “private schools” are more private than others. In many countries, private school curricula are controlled – sometimes almost totally so – by government. And the barriers to entry for private schools that aren’t part of a government-favored “private” school system can be extraordinary.

That said, this is yet another piece of important evidence pointing to the value of competition in education, recently affirmed (in the context of charter schools, but still) by Barack Obama. Who I understand is about to resign his Senate seat – I guess all those scandals and embarrasing Chicago machine connections the MSM kept refusing to cover finally caught up with him.

Leave a Comment » | competitive effects, methodology, money myth, politics, research reports, vouchers | Tagged: , , , , | Permalink
Posted by Greg Forster

Educating Journalists about Education Science

July 16, 2008

(Guest post by Greg Forster)

Don’t worry, this post is definitely not a continuation of the recent big dustup about 1) whether it’s naughty for scholars to provide journalists with accurate information about their work; and 2) whether it’s naughty for anonymous bloggers to argue that scholars’ motives are relelvant to their credibility, but bloggers’ motives aren’t relevant to theirs (which reminds me of Pat Moynihan’s quip about the Supreme Court cases, since overturned, holding that government can’t subsidize private school books but can subsidize classroom equipment such as maps; Moynihan asked, “What about atlases?” – books of maps? What about scholars who are bloggers? Or bloggers who write about scholarly studies? Once you start legitimizing ad hominem arguments, where do you stop?).

But I would like to expand on a comment that Eduwonk made during said dustup, which deserves more attention and has significance well beyond the issues that were at stake in that squabble. The comment got lost in the exchange because it was somewhat tangential to the main points of contention.

He wrote:

Not infrequently newspapers get snookered on research and most consumers of this information lack the technical skills to evaluate much of the work for themselves.   As education research has become more quantitative — a good thing — it’s also become less accessible and there is, I’d argue, more an asymmetry to the information market out there than a fully functioning marketplace of ideas right now.  In terms of remedies there is no substitute for smart consumption of information and research, but we’re not there yet as a field.

We are living in the first golden age of education research, brought on by the advent of systematic data collection, which every other field of human endeavor began undertaking a long time ago but which education is only getting around to now because it has been shielded from pressure to improve thanks to its protected government monopoly. Given the explosion of new information that’s becoming available, educating journalists about quantitative research is a huge problem. Jay is right that there is a marketplace of ideas. There really can’t help but be one; the idea some people seem to have that we can forbid people who own information from spreading it around as much as they want is silly. But just because there’s a market doesn’t mean there’s a perfect market, and Eduwonk is right that markets require informed consumers to function well. The current state of methodological ignorance among journalists does hinder the market of ideas from functioning as well as it should. (I’ll bet Jay would agree.)

As it happens, the same subject came up this morning in a completely different context, as my co-workers and I struggled to figure out the best way to present the findings of an empirical study we’re coming out with so that journalists will be able to follow them. And I wasn’t there, but I hear this topic also came up at a bloggers’ panel at the recent conference of the Education Writers’ Association.

Here at the Friedman Foundation, this has been a topic of great importance to us for some time, since exposing the bad and even bogus research that’s used to justify the status quo is one of our perennial challenges. We took a stab at composing a journalist’s guide to research methods. It went over well when we first distributed it (at last year’s EWA, if memory serves). But it’s necessarily very basic stuff.

Eduwonk is also right about journalists having been snookered by lousy research, and I think that has had both good and bad effects. The good news is that I’ve noticed a clear trend toward greater care in reporting the results of studies (not at propaganda factories like the New York Times, of course, but at serious newspapers). In particular, we’re seeing journalists talk about studies in the context of previous studies that have looked at the same question. Of course, we have a long way to go. But we’re on the way up.

On the bad side, however, I have also noticed a greater reluctance to cover studies at all. Part of that is no doubt due to the increase in volume. I’m young, but even I can remember the heady days of 2003 when any serious empirical study on the effects of a controversial education policy (vouchers, charters, high-stakes testing) would get at least some coverage. Now it’s different, and (to echo Eduwonk) that’s a good thing. But I think it’s extremely unlikely that this is the only factor at work. Junk science has poisoned the well for serious research. No doubt that was part of its intended purpose (although of course the motives of those who produce it have no relevance to its scientific merts or lack thereof).

My hope is that journalists will soon realize they’re getting left behind if they don’t learn how to cover the research accurately. Their job is to go where the news is. If the news is in quantitative research – and that is in fact where a lot of it is – they’ll have to learn how to get there.

Also, the changing media landscape will help. The old idea that journalists must be neutral stenographers with Olympian detachment from all the issues they cover is an artifact of the mid-20th-century role of the media as oligarchic gatekeeper, and is rapidly dying out. As “news” increasingly includes coverage by people who are actively engaged in a field, even as advocates, we can expect the news to be increasingly provided by people with greater amounts of specialized knowledge. (By the way, the old idea of the scholar as detached Olympian stenographer is equally an artifact of vanished circumstances, and will probably be the next thing to go; see the Our Challenge to You statement on the inside cover of any empirical study published by the Friedman Foundation for our views on the relationship between advocacy and scholarship.)

An optimistic view, yes – but since my optimism on other subjects has been triumphantly vindicated over the past year, even when the conventional wisdom said to head for the hills, I think I’ll let it ride.

Leave a Comment » | journalism, methodology | Tagged: , , | Permalink
Posted by Greg Forster

« Previous Entries
Next Entries »