Mamma Mia- Here we go again!
(Guest Post by Matthew Ladner)
NAEP has announced that it will release new math and reading results on April 10. I am going to do something truly foolish and dare to make a few predictions.
Making predictions is foolish because the reality is that our understanding of NAEP trends is imperfect at best. In any case, I have a theory about what has been driving NAEP gains here in the Cactus Patch, and I’m willing to make a few calls in advance on the 2017 NAEP based upon that theory.
Prediction 1: Arizona continues to improve.
Arizona’s improvement process is multi-faceted but the elephant in the room in my book is a hyper-active open enrollment market, the nation’s largest and most geographically inclusive charter sector and private choice programs. Arizona lead the nation in academic gains even during a period in which it had the largest cuts in spending due to the Great Recession. If we can do this during a period of funding cuts, we ought to manage it during a period of funding recovery given the broad consistency of policy. Note however that the state’s A-F letter grades has been turned off during the entirety of the 2015 to 2017 period, the demographics of students have continued to move further into majority-minority status, etc. Put me down for Arizona improving anyway based on the AZMerit improving in both 2016 and then again in 2017.
Prediction 2: North Carolina climbs.
Recent NAEP trends have not been great in NC. Notice in the Reardon tables that North Carolina starts with a (good) greenish tint on the top map-showing good early scores-but turns purple on the second map (bad) on cohort NAEP gains during the 2011 to 2015 period.
I suspect North Carolina will do better in 2017 based upon the information in this news report- basically that while statewide enrollment is growing, school district enrollment has been relatively flat due to the growth of choice options. North Carolina lifted a statewide cap on charter schools in 2011, created voucher programs for low-income and special needs children. I’m not sure whether they’ve done enough to start the open-enrollment virtuous cycle, but I think they may have done enough to shake things up a bit.
Third prediction: Indiana improves.
Give Indy a good stare in the Reardon chart, and it looks a bit like the green/purple undesirable combo between the two maps. It looks to me though like the districts are getting into the choice act (until 2007 you had to pay tuition in order to attend an out of zone school) and so I’m willing to buy some Hoosier stock.
Anyone else willing to dare a prediction should spell out their theory/evidence in the comment section.
Jay P. Greene's Blog 
I generally like that you’re making predictions.
Question – can you characterize the “meaningfulness” of the NAEP gain sizes? Perhaps as SD or something?
Mike-
So I have figured out how to do a significance test on the new NAEP data explorer, and have poked around a bit. It looks like it is rare for a two year difference in scores to hit statistical significance at the state level, with it being much easier for smaller trends to pass significance tests at the national level. So for instance a national two point decline was statistically significant at the national level in 2015, but three point gains at the state level did not make standard significance tests.
State level results can and do however get significant over longer time periods. Usually this takes the form of multiple incremental gains that add up to statistical significance over time. I will be happy if we see a lot of this in AZ, IN and NC.
Good question- I knew how to do some statistical significance tests on the old data explorer, but they changed the interface.
2013 was the first and only time our charter school was selected to take the NAEP. I suspect we did well for our demographic as that happened to be one of the more talented classes I’ve taught (they’re all young adults/first year of college now). I’d prefer students take this test consistently vs. the hodge-podge of state tests; if we did that it would pretty much set a nation-wide math curriculum. If states voted to make the NAEP their state assessment…wait…that would put a lot of testing folks out of business….smh.