David Figlio’s latest report on Florida’s Corporate Tax Credit (CTC) Scholarship program was released yesterday. I can’t find the report online but Ron Matus of the St. Pete Times sent it to me and you can read about it in his article.
(UPDATE: Here’s a link to the study.)
I agree with almost everything said in the article. I even agree with Mark Pudlow, the spokesperson for the teacher union when he said: “There is no quick fix for struggling students.”
The problem is that the standard for success when it comes to school choice is that it has to produce a quick fix or critics deem it a failure and declare: “we really ought to reconsider why we’re doing it.” No one demands that every other education policy produce huge gains in a single year or they should be “reconsidered.” Yes, promoters of policies may make unrealistic promises to get them adopted, but the standard for success should be long-term progress, not promises made by politicians.
So let’s slow the rush to judgment and review what we really know about the CTC program. First, Figlio finds that 92.7% of all CTC students in private schools provided a usable standardized test to the evaluation. This shows widespread compliance with the legal requirements for those students to be tested to satisfy political concerns for accountability.
Second, Figlio finds that the CTC program has largely targeted students who are significantly more disadvantaged than students remaining in Florida’s public schools — even significantly more disadvantaged than public school students receiving subsidized or free lunch. So, concerns that the program would cream off the best students appear unfounded.
Third, and most importantly, Figlio’s report does not make any claims about whether students benefited academically from participating in the CTC program. He simply provides descriptive information on the academic achievement of CTC students as well as subsidized lunch students in Florida public schools. But we know that CTC students are even more disadvantaged than those public school students and Figlio makes no attempt in this report to control fully for those disadvantages.
Figlio makes these points explicitly and repeatedly in the report: “it is important to recognize that they are not causal estimates of the effect of program participation on student outcomes. Causal comparisons require more complete modeling of the selection decisions into the scholarship program and fuller data from a baseline than is afforded using the 2006-07 school year test score collection. More compelling causal estimates of program participation will be possible following the collection of the 2008-09 school year’s test score data. The comparisons in this subsection should be interpreted as purely descriptive in nature.”
Unfortunately, most people never pay attention to these warnings and rush ahead as if descriptive information is causal. Folks wrongly conclude that if CTC students make year to year test score gains that are about the same as subsidized lunch public school students, then they must not be benefiting from the program. Nothing in Figlio’s report supports that conclusion.
To know whether CTC students are benefiting we would have to know how they would be doing had they remained in public schools. The best way to judge that is with a random-assignment study where students admitted to the CTC program by lottery are compared with students who lose the lottery and remain in public schools. Unfortunately, that research design is not possible because there was no lottery. The next best thing would be to use a research design that approximated random-assignment (like a regression discontinuity) or a rigorous quasi-experimental design that controlled for all observed differences between the two groups. But Figlio didn’t do that in this report. He just provided descriptive statistics while promising a more rigorous research design next year.
Of course, we might wonder why Figlio bothered reporting this descriptive information without a more rigorous analysis. I suspect that he was required to produce a report each year by the legislature, so he complied even though he didn’t have the information he needed for a causal analysis.
And the descriptive information is useful. It suggests that choice was no miracle cure since the raw differences between CTC and public students in academic progress were not huge. Again, miracle cure is the wrong standard for judging a program’s success.
The CTC program may well have attracted students who had been on a downward trajectory before they switched to a private school. And the CTC program may well helped those students level-off and may, over time, enable them to make significantly greater progress than they would have made had they remained in public schools. This is what we’ve seen from rigorous evaluations of other choice programs, including the most recent evaluation of the DC voucher program. But these things require careful research designs and time to show themselves. Let’s give David Figlio more time to use a better research design so that we can actually say something about the academic effects of the CTC program.
(edited for typos)
Jay P. Greene's Blog 
Now I’m going to have a screeching nasal Candian male witch voice stuck in my head all day “We’ve taken care of everything…”
One thing the study does reveal is that voucher programs, like charters, do whatever the opposite of “cherry-picking” might be. Road-apple picking?
Since one of the primary complaints of supporters of the educational status quo is the lousy quality of the students and parents this ought to be seen an all around win. The lousy students and their, presumably, lousy parents are foisted off on private schools and at the bargain price of only $3,995/year, a bit more then half of what it would cost if they were being their lousy student selves in a district school.
Shouldn’t district schools, relieved of this baggage show an up-tic in their school scores?
Perhaps the first step of ridding constraints on student learning is the end of NCLB and not vouchers.
Allen,
Perhaps you don’t live in Florida. I do. I suppose it doesn’t matter becaise math will always prevail and when you rid the sample of low scorers, the mean will always rise. For example, in Florida, we have a grade three retention policy which filters poor scorers from the grade 4 sample. Yet, Mr. Ladner , who frequently forgets to note this filtration,likes to boast about Florida’s grade 4 scores when they constitute a filtered sample. He then tries to compare a filtered sample to unfiltered ones and is amazed by the superiority of the filtered sample. I read Florida’s tenth graders from this year have the same percentage scoring proficient as before full exposure to our flawed accountability system.
As for your question about an up tic in scores being relieved of what you term “baggage”, ask Mr’ Ladner to quote the decrease in F schools. In Florida, there are provisons which allow one to transfer out of an F school. Perhaps these low scorers can be absorbed into a school which can accomodate them and still maintain their state school grade.
The study does not show that school choice attracts low performers. It does not allow for any conclusions at all about what kinds of students do and do not select into school choice. Figlio doesn’t have the kind of data that would allow him to examine that question.
Allen, what exactly in the study gives you the impression that it shows school choice attracts lower performers?
Charter schools probably do attract low performers, because in most states they’re designed to select for those populations. This is one of the reasons charters have such limited potential for real reform – the unions do oppose charters, but they mostly tolerate them without putting up too much of a fight as long as charters are only taking away the students who are hardest to teach. If charters ever threaten to become a broad-based substitute for district schools in the general population, the unions will turn on them with a vengeance.
It’s not the study that indicates charters attract lower performing kids but common sense. Also, I’m not above needling charter opponents on one their favorite misrepresentations, the charge of “cherry-picking”. Pricking their complacency just seems to be an inherently proper thing to do.
Beyond self-indulgence though is common sense.
If you assume parents don’t randomly enroll their kids in charters then they must have a reason. There just aren’t that many reasons for a parent to enroll their child in a charter.
Either their child is doing well in the district school but the parent believes the charter to be sufficiently superior to make the transfer worth the inevitable problems or their child is doing poorly and a charter looks like a better bet then more of the same which is what they’ll get in a district school.
Of the two propositions I’d say the latter is much more likely to be the case. Road-apple picking. Or rather, to make the metaphor a bit more accurate and probably more tedious, road-apple proffering since charters aren’t magnet schools and thus can’t select applicants.
Also, as you point out, charters in most states are aimed at lower-performing populations which stacks the deck in favor of road-apples and not cherries. But I don’t think that the fact that charters are aimed at the hardest to teach and poorest students has much to do with the unions’ relatively tame response to charters to date. Will teachers throw out the current union officialdom in favor of union representatives that’ll make sure charters don’t make off with the easier-to-teach students? I rather doubt it.
The point at which charters become a real danger to the unions is when the possibility exists that charters might replace districts. Unions depend on districts to hold down the cost of bargaining and organizing. If charters replace a district, depending on state law, the union might be faced with the prospect of organizing dozens, probably hundreds, of individual schools each with its own managment. I rather doubt that all those charters would sit on their hands while the unions went about arranging state law to suit their convenience so here’s another fight.
One of the face cards in any such political dustup is poor, black parents who are desperate for the their children’s futures. Steve Barr seems to have come to understand the political potency of such a constituency with his organizing of Parent Revolution but I doubt that’ll be the last such organization. Also, Steve Barr’s not the only politically-astute player who understands the importance of that particular constituency. Democrats for Education Reform isn’t all the well known but they must be feeling pretty good with a president and Secretary of Education who both seem to have nothing but nice things to say about charters.
And since I’ve gone on again at too great a length, I’ll finish by pointing out yet another pro-charter constituency that isn’t yet really aware of its status: mayors.
Charters obviate the necessity for a central office staff and with it the central office staff budget. Mayors may look on those budget dollars covetously and so be tempted to engineer a political deal that aggressively dissolves the school district, replacing the district schools with charters, while seizing the now unnecessary tax revenue. In any such coup they’d be very likely to find support among the various city unions despite teacher’s union unhappiness – what are the fatcats of the school administration to the city’s firemen and cops? – and the current and past charter parents.
Given the amount of money that disappears into the black hole of the central office there might be money enough for a tax cut on top of an increase in city services so it might even be possible to rope anti-tax crusaders into the coalition.
[…] to some it may seem an unlikely pairing, Greene acknowledges agreement with the teachers union spokesman that “There is no quick fix for struggling students”. […]
Allen,
I live in Florida. Schools with low percentages of proficiency offer their student body a chance to opt out for a school with better stats. I think there may be a time limit on years with low proficiency percentages but you may need to research it on your own. I would correlate low static achievement measures with low SES, as research has heavily shown, and guess you are correct in assuming that many voucher students are poor. From my observation, and in no way research, when you oppen the category to charter or specialty programs, you would see more diversity in the population.