(Guest Post by Collin Hitt)
KIPP schools demand a certain kind of student – a student who is willing to put in long hours and put up with very strict rules. KIPP has been shown to substantially increase student test scores. But critics argue that the culture at KIPP has major effects on recruitment and retainment. KIPP schools attract better students and are more likely to weed out low performing students, the argument goes. If this is true, KIPP students who persist in school are more likely to have a high-achieving peer group – and the effects of simply being in a peer group are really what explain any positive effects at school. A new study from Mathematica destroys this critique.
At its core, the critique of KIPP is a restatement of larger questions facing the charter school sector. Do charter schools cream the best students from nearby schools? And, compared to surrounding schools, are the lowest performing students at charter schools most likely to leave? Two rigorous studies reviewed here at JPGB answer an unequivocal “no.” But KIPP is a crucial case. The average charter school might take all the students it can find and do anything to keep those students. But surely if anybody engineers the makeup of their student body, it would be a school like KIPP, right?
So, does KIPP cream the best students (or at least better-than-average students) from nearby schools? The following chart shows, clearly not.
Entering KIPP students perform the same or worse than students in surrounding schools. But does KIPP then take exceptional efforts to push low performers back into surrounding schools. Again, clearly not.
Students transfer out of KIPP schools at the same rate as surrounding schools. And the students who transfer perform the same on standardized tests. So the only manner that KIPP may in some way create a measurably different peer group is through the quality of students in later grades who replace the KIPP students who transfer out. In this respect, the students who later transfer into KIPP are higher performers on average than students who transfer into district schools, according to the Mathematica study. But this, of all the ways to create a higher-performing peer group, is the least likely to have any meaningful impact on the performance of students who enter KIPP early on. The high performing peer group wouldn’t even be formed until students’ time at KIPP was almost over.
With their typical class, the Mathematica authors give their critics a charitable hearing, in fact constructing the strongest possible case for the peer-effect hypothesis. So, do peer effects explain KIPP’s impact on test scores? From the Mathematica study itself:
“One way to estimate the possible size of peer effects at KIPP is to combine our findings with other research on how peers’ prior scores affect student achievement. Unfortunately, published estimates of the effect of peer ability on student achievement range widely, from close to zero to nearly half a standard deviation impact for each standard deviation of difference in peer achievement. Even if the largest estimates of peer effects are correct, however, the improvement in peers’ prior test scores would appear to benefit KIPP students’ achievement only by about 0.07 to 0.09 standard deviations after four years at KIPP. KIPP’s cumulative impacts in middle school are three times that size, so even the largest estimates of the size of peer effects suggest that they are unlikely to explain more than one-third of the cumulative KIPP impact.
“Moreover, the best available evidence shows that KIPP produces large impacts on students in their first year at a KIPP school—before late-entering students could possibly have any effect. Consequently, the true peer effect resulting from late entrants is likely to be substantially below the back-of-the-envelope estimate of 0.07 to 0.09 standard deviations.”
The peer-group critique of KIPP essentially says this: anybody could get KIPP’s results if they had KIPP’s students. This simply isn’t true. KIPP is getting better results because of the work being done by teachers and staff. Rather than wonder, if only other schools could have students like KIPP’s, perhaps we should wonder why other schools don’t have adults like KIPP’s. (And, for that matter, why don’t other think tanks have scholars like Mathematica’s?)