False Claims of Cherry Picking are the Pits

August 20, 2008

Leo Casey over at Edwize is urging me to join the “United Cherry Pickers” union because he thinks I’ve cherry picked the evidence on vouchers in a previous post.  This sounds like a great deal if my dues, like those from AFT and NEA members, can contribute to paying for skyboxes for Leo and his buddies at the Democratic National Convention to make-up for the convention’s shortfall of $10 million.  Where do I sign up?

Making a charge of cherry picking is easy.  Substantiating it requires, well, uhm, evidence.  Evidence isn’t exactly Leo Casey’s strong-suit.

I said that there have been 10 analyses of random assignment voucher experiments.  I said that 9 of those 10 analyses show significant, positive effects (at least for some subgroups).  If I am cherry picking, which random assignment analyses am I leaving out? 

Leo Casey then asserts: “Serious research conducted by respected scholars without an ideological axe to grind has consistently found every major voucher experiment in the United States wanting. John Witte’s and Cecilia Rouse’s definitive analyses of the Milwaukee voucher program and the Indiana University studies of the Cleveland voucher program have shown no meaningful educational performance advantage for students in those two high profile, large scale voucher programs.”

Neither Witte nor the IU studies analyzed random-assignment experiments, making it harder to have confidence in their results, which is why I focus on the 10 analyses using the gold-standard approach. 

Rouse’s study did examine a random-assignment experiment, but Casey mischaracterizes her findings.  She writes: “I find that students in the Milwaukee Parental Choice Program had faster math score gains than, but similar reading score gains to, the comparison groups. The results appear robust to data imputations and sample attrition, although these deficiencies of the data should be kept in mind when interpreting the results.”   Remember, Casey falsely claims that she finds “no meaningful educational performance advantage for students.”

Casey also mischaracterizes my citation of Belfield and Levin’s findings: “[He even cites research that is not on the subject of vouchers: Hank Levin will be most surprised to learn that his research ‘supports’ vouchers.]” 

Since I actually bothered to quote Belfied and Levin’s findings about the effects of expanding choice and competition, I don’t think Hank Levin will be the least bit surprised to read what he wrote.  I’ll repeat the quotation here so that no one is shocked: “A sizable majority of these studies report beneficial effects of competition across all outcomes… The above evidence shows reasonably consistent evidence of a link between competition (choice) and education quality. Increased competition and higher educational quality are positively correlated.”

If Leo Casey is going to make the charge of cherry picking and improperly citing evidence, he has to deliver proof of those charges.  To the contrary, the facts indicate that Casey is the one cherry picking and improperly citing research.

Is there a union for playing fast and loose with the truth?  Maybe Leo Casey should join it.  Oh, I forgot.  He’s already a member of the AFT.

(Links added)

8 Comments | vouchers | Tagged: , , , , | Permalink
Posted by Jay P. Greene

The Vitamin C of Education

August 20, 2008

Earlier this week I made my Modest Proposal for B.B. (Broader, Bolder or is it Buying Bananas?).  I noted that Randi Weingarten denounced vouchers as a waste of time despite considerable evidence supporting it, while she embraced the B.B. idea of community schools despite there being absolutely no evidence to support the claim that public schools could improve achievement by expanding their mission to include a host of social services.

Given the lack of evidence for B.B. I generously : ) offered to support a series of large pilot studies of the community schools approach, if Weingarten, Leo Casey, and the B.B. crowd would agree to a similar series of large pilot voucher programs as a way of learning more about both reform strategies.  No word yet but perhaps their internet is broken (just try unplugging it and plugging it back in).

Shital Shah from the Coalition for Community Schools, however, sent me a nice note with a link to a report claiming to contain the evidence supporting their approach.  After reviewing the report I still see virtually no evidence to give us confidence that public schools can increase student achievement by offering everything from legal assistance to health care.

In Appendix B the report lists 21 studies of the community school approach.  Seven of them have no student achievement outcomes.  Seven examine student test scores but only make pre/post comparisons without any control group.  And another seven have comparison groups but none employ random assignment, regression discontinuity, or another rigorous research design.  Four of those seven just compare achievement at schools using the B.B. approach to city or statewide averages.  And of the seven studies with some kind of control group, two find null effects, another finds null effects in math but not reading and even then only among schools with “high implementation” of the approach.  The quality (and quantity) of the evidence supporting community schools is no greater than what we could find to support the healing power of crystals. 

I understand why Randi Weingarten or Leo Casey would be pushing the educational equivalent of crystal healing.  Their job is to advocate for the interests of their union, not to make fair and reasonable assessments of research claims.  If schools expand their mission to include providing health care and other social services just think of all of the dues-paying nurses and social workers they could add to their rolls.

The greater mystery is why normally tough-minded and rigorous researchers, like Jim Heckman and Diane Ravitch, would sign on to this approach entirely lacking empirical support.  Heckman won the Nobel Prize for Economics for crying out loud.  But then again Linus Pauling won the Nobel Prize for Chemistry and later became a public advocate for mega doses of vitamin C to cure cancer, another intervention completely unsupported by rigorous evidence.

I’ll repeat that I am not against trying the B.B. community school approach with large pilot programs that are carefully studied.  I just can’t see why normally smart people would fully endorse untested approaches while ignoring other interventions, like expanding choice and competition in education, which have considerably more supporting evidence.

(edited for typos)

Leave a Comment » | myth of helplessness, vouchers | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Broader, Bolder = Bloated Behemoth

August 13, 2008

 

Over at D-Ed Reckoning Ken DeRosa reviews the “evidence” that the AFT’s Leo Casey presents on the effectiveness of the Broader, Bolder approach being pushed by the union-backed Economic Policy Institute (with the support of some impressive people who you would think would know better). 

The issue is not whether kids would benefit from better health care or social services, or even whether receiving those benefits might contribute to higher achievement.  The issue is whether public schools are capable of expanding their mission to effectively provide these additional services, and whether those schools can translate the provision of additional services into higher achievement.

The Broader, Bolder folks provide a list of “background papers” to support their cause.  But those papers are very far in the background in that only a handful of the more than 100 studies cited actually assess the effects of providing students with additional services.  And even fewer look at the effects of public schools providing those services.  Before we endorse a bold new plan for education wouldn’t we want at least a few  evaluations of pilot programs in which public schools actually provided the full set of services being advocated?  I can’t find one such evaluation in the list of 100+ studies provided.

But don’t worry, Leo Casey has stepped into the breach with the solid research we need.  Here’s DeRosa’s commentary bracketing Casey’s, uhm, evidence:

“Leo must have had a few of his underlings pouring over the ERIC databases non-stop finding the requested evidence. Here is Leo’s evidence. I am leaving in all the internal citations and footnotes.

Classroom teachers recognize immediately the educational value of providing a comprehensive array of services to students living in poverty. They have seen the effects of undiagnosed and untreated eye problems on a student’s ability to learn how to read, and of untreated ear infections on a student’s ability to hear what is being said in the classroom. They know that the lack of proper medical care heightens the severity of childhood illnesses and makes them last longer, leading to more absences from school for students who need every day of school they can get. They have seen asthma reach epidemic proportions among students living in poverty, and they know that the lack of preventive and prophylactic medical care leads to more frequent attacks of a more severe nature, and more absences from school. They understand that screening for lead poisoning happens least among children in poverty, even though their living conditions make them the most likely victims, with all of the negative effects on cognitive functions. They know that the stresses of life in poverty make mental health and social work services for students and their families all that more important, and yet they are least likely to receive them. They see how the transience that marks poverty disrupts the education of students again and again, as the families of students are constantly on the move. In short, teachers know that the students living in poverty lack the health and social services routinely available to middle class and upper class students, despite the fact that they need them even more. And they know that the absence of these services has a detrimental impact on the education, as well as the general well-being, of students living in poverty.

I emphasized Leo’s evidentiary citations since they do not conform to the generally accepted norm.”

(edited for typos)

7 Comments | myth of helplessness | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

Next Entries »