Jay P. Greene's Blog

The National Center for the Study of Privatization in Education at Columbia University’s Teachers College has posted a new study on charter schools by Kevin Booker (Mathematica), Tim Sass (Florida State), Brian Gill (Mathematica), and Ron Zimmer (Rand).  The researchers look at whether attending a charter high school in Chicago and Florida increases the likelihood that students would graduate high school and go on to college.  The short answer is that it does.

The paper’s abstract states:

“We find that charter high schools in Florida and in Chicago have substantial positive effects on both high school completion and college attendance. Controlling for observed student characteristics and test scores, univariate probit estimates indicate that among students who attended a charter middle school, those who went on to attend a charter high school were 7 to 15 percentage points more likely to earn a standard diploma than students who transitioned to a traditional public high school. Similarly, those attending a charter high school were 8 to 10 percentage points more likely to attend college. Using the proximity of charters and other types of high schools as exogenous instruments for charter high school attendance, we find even stronger effects in bivariate probit models of charter attendance and educational attainment. While large, our estimates are in line with previous studies of the impact of Catholic high schools on educational attainment.”

But I can already hear doubters wondering how you could compare students in charter schools to other students when the kinds of students who self-select into charter schools could be very different from those who do not. 

But never fear.  These researchers are pretty bright and they worried about this problem as well.  So they came up with three novel strategies to address the possibility of selection bias.  First they try the usual (and not entirely persuasive) technique of controlling statistically for any observed differences between the charter and non-charter students, including race/ethnicity, gender, disability status, family income, and — most importantly — 8th grade student test scores. 

But what about the unobserved (and uncontrollable) qualities of students who choose charters?  Well, their second technique to address potential selection bias is that they compare students who were all in charter schools in 8th grade.  The treatment group went on to a charter high school while the control group went to a traditional public high school.  Since both groups began as charter-choosers, the unobserved qualities of people who choose charters should be present in both groups.  As the authors describe it, “If there are unmeasured student/family characteristics that lead to the selection of charter high schools, these unmeasured characteristics ought to also lead to the choice of a charter school at the middle school level.”

But wait, they did one more thing that really nails the potential problem of selection bias.  They took advantage of the fact that not all students who attend charter middle schools live within a reasonable distance of charter high schools (especially in Florida) to create an “exogeneous” instrument for predicting whether students would attend a charter high school.  That is, they could obtain an unbiased estimate of attending charter high school based on geographic distances and then use that unbiased estimate of charter attendance to obtain an unbiased estimate of the effect of attending a charter high school on graduation and college-attendance.  If you don’t trust me that this technique works to correct for selection bias, you can trust the Nobel prize in economics, which was awarded to James Heckman at the University of Chicago for having developed this technique.

This study comes on the heels of positive results from Caroline Hoxby’s random-assignment evaluation of charter schools in New York City.  Random-assignment corrects for potential selection bias because the students accepted into the charter schools by lottery.  Only chance distinguishes the students in the treatment group (charters) from those in the control group (traditional public).  Hoxby’s analysis finds:

“What is the main result or the bottom line for the grade 3-8 tests? New York City’s charter schools raise their third through eighth graders’ math scores by 0.09 standard deviations for every year they spend in the school. Remember, these gains are in addition to whatever gains the students would have been expected to make in the traditional public schools, had they been lotteried-out. This result is statistically significant with a high level of confidence. (The p-value, shown in parentheses, is less than 0.001.) That means that we are very confident, more than 99% confident, that the effects of New York City’s charter schools on math achievement are not zero or negative…. New York City’s Charter Schools raise their third through eighth graders’ reading scores by 0.04 standard deviations for every year they spend in the school. Remember, these gains are in addition to whatever gains the students would have been expected to make in the traditional public schools, had they been lotteried-out. This result is statistically significant with a high level of confidence. (The p-value, shown in parentheses, is 0.016.) That means that we are very confident (98% confident) that the effects of New York City’s Charter Schools on reading achievement are not zero or negative…. What is a standard deviation? A standard deviation or “effect size” is a conventional way of expressing test scores that works for all tests. If students’ scores rise by one standard deviation, it is a large change in achievement. On most tests it corresponds to more than a grade’s worth of learning and more than a performance level.”

So we now have some very well-designed studies to address selection concerns and they are finding significant benefits from attending charter schools.

This entry was posted on Thursday, November 13th, 2008 at 5:07 pm and is filed under charter schools. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.