David Figlio’s latest report on Florida’s Corporate Tax Credit (CTC) Scholarship program was released yesterday. I can’t find the report online but Ron Matus of the St. Pete Times sent it to me and you can read about it in his article.
(UPDATE: Here’s a link to the study.)
I agree with almost everything said in the article. I even agree with Mark Pudlow, the spokesperson for the teacher union when he said: “There is no quick fix for struggling students.”
The problem is that the standard for success when it comes to school choice is that it has to produce a quick fix or critics deem it a failure and declare: “we really ought to reconsider why we’re doing it.” No one demands that every other education policy produce huge gains in a single year or they should be “reconsidered.” Yes, promoters of policies may make unrealistic promises to get them adopted, but the standard for success should be long-term progress, not promises made by politicians.
So let’s slow the rush to judgment and review what we really know about the CTC program. First, Figlio finds that 92.7% of all CTC students in private schools provided a usable standardized test to the evaluation. This shows widespread compliance with the legal requirements for those students to be tested to satisfy political concerns for accountability.
Second, Figlio finds that the CTC program has largely targeted students who are significantly more disadvantaged than students remaining in Florida’s public schools — even significantly more disadvantaged than public school students receiving subsidized or free lunch. So, concerns that the program would cream off the best students appear unfounded.
Third, and most importantly, Figlio’s report does not make any claims about whether students benefited academically from participating in the CTC program. He simply provides descriptive information on the academic achievement of CTC students as well as subsidized lunch students in Florida public schools. But we know that CTC students are even more disadvantaged than those public school students and Figlio makes no attempt in this report to control fully for those disadvantages.
Figlio makes these points explicitly and repeatedly in the report: “it is important to recognize that they are not causal estimates of the effect of program participation on student outcomes. Causal comparisons require more complete modeling of the selection decisions into the scholarship program and fuller data from a baseline than is afforded using the 2006-07 school year test score collection. More compelling causal estimates of program participation will be possible following the collection of the 2008-09 school year’s test score data. The comparisons in this subsection should be interpreted as purely descriptive in nature.”
Unfortunately, most people never pay attention to these warnings and rush ahead as if descriptive information is causal. Folks wrongly conclude that if CTC students make year to year test score gains that are about the same as subsidized lunch public school students, then they must not be benefiting from the program. Nothing in Figlio’s report supports that conclusion.
To know whether CTC students are benefiting we would have to know how they would be doing had they remained in public schools. The best way to judge that is with a random-assignment study where students admitted to the CTC program by lottery are compared with students who lose the lottery and remain in public schools. Unfortunately, that research design is not possible because there was no lottery. The next best thing would be to use a research design that approximated random-assignment (like a regression discontinuity) or a rigorous quasi-experimental design that controlled for all observed differences between the two groups. But Figlio didn’t do that in this report. He just provided descriptive statistics while promising a more rigorous research design next year.
Of course, we might wonder why Figlio bothered reporting this descriptive information without a more rigorous analysis. I suspect that he was required to produce a report each year by the legislature, so he complied even though he didn’t have the information he needed for a causal analysis.
And the descriptive information is useful. It suggests that choice was no miracle cure since the raw differences between CTC and public students in academic progress were not huge. Again, miracle cure is the wrong standard for judging a program’s success.
The CTC program may well have attracted students who had been on a downward trajectory before they switched to a private school. And the CTC program may well helped those students level-off and may, over time, enable them to make significantly greater progress than they would have made had they remained in public schools. This is what we’ve seen from rigorous evaluations of other choice programs, including the most recent evaluation of the DC voucher program. But these things require careful research designs and time to show themselves. Let’s give David Figlio more time to use a better research design so that we can actually say something about the academic effects of the CTC program.
(edited for typos)
Posted by Greg Forster 
Jay P. Greene's Blog 