(Guest Post by Matthew Ladner)
So I’ve still been digging into the AZ NAEP data. When something seems too good to be true, it is best to assume it isn’t true. I decided to investigate the possibility that something goofy was going on with the free and reduced lunch variable. Not all Arizona charters choose not to participate in the program, and the eligibility criteria for the program has changed over time.
Parental education may be a good stand-in for what may be a suspect income variable. At the 8th grade level, the NAEP data slicer has an a variable for parental education. The below figure presents the 8th grade math scores for students with college graduate parents. In order to account for possible differences in special program participation, the figure is only for general education students with college graduate parents (the ranking results don’t change much if you look at all students). I will again stress that these comparisons do not substitute for a proper random assignment study-only that they tell us more than an examination of aggregate scores for all students.
Watch out New England…Arizona charter schools are coming to get you!
For you incurable skeptics, the below figure presents the same comparison using 8th grade reading, and bear in mind that each NAEP test involves a different sample of students.
We can also look at these numbers by race/ethnicity. NAEP provides subset numbers for Anglos and Hispanics attending charter schools in Arizona. Here is the NAEP 8th grade reading test for Hispanic students:
And here it is for Anglo students:
Okay but what if those Arizona charter schools are chock full of Anglo kids whose parents graduated college? Now the National Alliance of Public Charter Schools reports that Arizona charter schools have a majority-minority student body, so this is not the case- but what if a large portion of the Anglo kids attending charter schools have parents with college degrees? Ok, well, let’s compare Anglo kids whose parents graduated from college:
Did I mention the part where Arizona charter schools did this with $8,041 per kid in public funding? Better results at a lower cost is what America is going to need very soon- and well here it is. Massachusetts NAEP scores taste like chicken btw, only gamier, could use a little salt.
Jay P. Greene's Blog 
Hi Matt,
Hey, as a Massachusetts guy, I hope that AZ charters or anyone else comes to topple “us” — b/c I think the excellence story is overstated….
But – I do have a (friendly) question…..
The CREDO data that (charter-friendly) Marty West examined from 2006 to 2012: “On average, AZ charter schools at every grade level have been modestly less effective than traditional public schools in raising student achievement.”
I realize you probably have posted your views on this in the past, but I don’t recall what they are, so:
What’s your view?
That the CREDO gain data tells the wrong story, and NAEP absolute scores are more useful/relevant for Boooom purposes?
Or perhaps that the 2006-2012 story is not the same as the 2015 story – i.e., things are looking up for AZ charters recently?
Or something else?
Mike-
Good question. I have not examined the Credo studies, but Paul Peterson noted a few years ago that their methodology fails to capture a couple of potentially large sources of bias. Kids take an academic hit when they transfer between schools (hopefully temporary) and new schools in their shakedown cruise period tend not to have hit their academic stride.
http://www.wsj.com/articles/SB10001424052748703909804575123470465841424
If you have a rapidly growing sector with lots of new schools full of kids that just transferred into the schools…let’s just say that the snapshot may contain an optical illusion.
Also what we could be seeing here is the maturation of the AZ charter school sector- still new schools opening, but having a smaller impact on the sector wide score than the past. Low performers have continued to close over time as well.
Also Arizona’s school grading system jibes better with the 2015 NAEP than with the Credo study. During the 2013-14 school year 40 percent of Arizona charter schools earned an “A” grade compared to only 28% of district schools. Arizona charters were also relatively underrepresented at the low-end, with 7% “D” grades compared to 9% among district schools. Arizona’s school grades reflect both proficiency and academic growth.
Jay has been writing a great series of cautionary posts about overly fixating on test scores, so looking at some other measures would be a good idea as well. The AZ Board of Regents tracks higher education graduation rates by public high school. Seven out of the top 10 general enrollment schools in AZ are charter schools:
Click to access College-Completion-Rate.pdf
In short we should use a skeptical eye towards all of these sources, including but not limited to the Credo report. In my view the weight of the evidence (NAEP, school grades, college success) > Credo report.
“Criteria” is plural. Other than that, BOOOOOOOOM!