I Say the Future is Ours if You Can Count

April 13, 2018

(Guest Post by Matthew Ladner)

Yesterday we went over how states have done very little to improve in 8th grade math since 2009, but how many state charter sectors rocked it. Let’s see about reading, starting with the states:

There are more declines in scores than statistically significant gains in that first chart-some states more lost than others, but a lost decade nationally. Next let’s look at statewide charter sectors-these are the states with charter sectors large enough to make the sample in both 2009 and 2017 in 8th grade reading:

Once again, just as with math, Excel had to move the growth axis scale for several state charter sectors. You will appreciate this better when the following chart combines statewide averages and state charter sector averages:

So in this chart ideally you would like to see high scores and high gains, but there is no shame in just very high scores.

Can you dig it?

Can you dig it?!?

CAN YOU DIG IT?!?!?!?!

Charters CeleNAEP Good Times During “Lost Decade”

April 11, 2018

(Guest Post by Matthew Ladner)

The news overall is grim. You want to be in the top right quadrant of this chart. Some states thankfully did land there-including the state with the largest student population- but many only barely due to small reading gains. If you need the dot size to push you in, it doesn’t count-better luck next NAEP.

You don’t want to land in the top left or in the bottom right, and you most of all don’t want to see your state in the bottom left quadrant (declines in both subjects). Mike Petrilli used the phrase “lost decade” to describe the results. Some states seem far more lost than others, but it is hard to find fault with that assessment overall.

The two main reform strategies employed since the 1990s have involved test-based accountability and increased parental choice. During the era covered by the top chart, the test-based folks swung for the fences by creating a federal incentive for states to adopt a preferred set of academic standards and to pass statewide teacher evaluation systems based upon the scores on those tests. Gigantic investments of political and financial capital supported these policy changes, but it is hard to characterize the results as much more than disappointing.

Now some of you will be thinking around about now “oh yeah but we’ve expanded choice during this period as well!” That is true, and while we have numerous studies establishing positive competitive effects on district schools from choice programs, few states have choice programs going at a scale to place a large amount of pressure on district enrolments. NAEP does however allow us to track state charter sector gains over time. Sixteen state charter sectors had scores for 8th grade math and reading in both 2009 and 2017, allowing the following calculation:

Excel had to change the scale of the axis for the above chart. You may not have noticed. Putting state averages and state charter sector averages into the same chart will help:

Suddenly those statewide gains in Arizona, California and Mississippi (i.e. the good ones) from the first chart don’t seem so impressive eh? I’m thinking out loud here and inviting you along for the ride. Gains aren’t everything, so the next iteration will include achievement and gains by subject area, but for the huge gainer sectors (spoiler alert) they didn’t get that way with low 2017 scores. I could go on about standard errors being bigger for charter sectors and whatnot, but who are you going to believe a boring statistics lecture or your own lying eyes? If someone can explain why random error would systematically dramatically favor charter sectors, I’m all ears and the comment section eagerly awaits your thoughtful challenge.

In fact there is a white lie in the above chart-some of the sectors and states that look meh in this chart immediately above had very high scores in both 2009 and 2017, and while not ideal there is no crime in holding your mud with high scores. Last year my friend Robert Pondiscio convinced me that combining achievement and gains to provide a clearer picture, so here goes for 8th grade math:

In this chart you either want to have large gains, or high scores, or preferably both of these things. I’m for instance not inclined to criticize Idaho charter schools for modest gains given that they outscore Massachusetts and all despite spending about half the amount per pupil.

Well…yup it is officially time:


Arizona Charters Crushed the Ball Again But They Have Competition Out West

April 10, 2018

(Guest Post by Matthew Ladner)

Arizona charter schools continued to display impressive scores and gains on the 2017 NAEP. Note as always that only a random assignment study properly conducted could hope to isolate the role of school quality in all of this, and that such studies are not only unavailable but also are impractical for a statewide system of schools- not all of which are oversubscribed (a precondition for a random assignment study). Thus the role of average school quality in driving the above results remains a holy mystery- likely to be hotly debated, impossible to be resolved in this world. I and others will be digging into Arizona charter subgroup scores in the days ahead.

Now, behold the nothing burger that has been net American academic progress since 2009 (first set of columns). On the math and reading tests 10 points approximately equals a grade level worth of average academic progress (the science exams have a different scaling). Net American progress equals nothing, nothing, next to nothing a tiny bit of something in 8th grade reading.

The middle blue columns are the statewide numbers for Arizona. As you can see they consistently surpass the American nothing-burger. The final set of gains are the gains for Arizona charter school students.

Some of Arizona’s neighbors also have charter sectors that crushed the ball- starting again with Colorado. I’m happy to say that Nevada’s sector made the minimum reporting requirements for NAEP this year, and the results look good. Give me some time to dig, but a monster story may be California charters:


Gains/Scores by State for Anglo Students

April 10, 2018

(Guest Post by Matthew Ladner)

Sorry-nodded off before posting the final piece of the ethnic trilogy. On to other subjects…

2017 NAEP-Scores and Trends for Black Students

April 10, 2018

(Guest Post by Matthew Ladner)

A few months ago I speculated that a statewide Black population would catch up to a statewide Anglo NAEP score. It got close in 8th grade math with Arizona’s Black students scoring 272 and the lowest scoring statewide Anglo score coming in at 274. Here’s a chart on 8th grade scores and gains in math since 2003 (Black students did not make the minimum size for NAEP to report scores in several states):

Indiana and Arizona can thumb wrestle later for the championship.


NAEP- First Report from the Front

April 10, 2018

(Guest Post by Matthew Ladner)

So the data went live at midnight EST. The national data is basically flat on all four tests. Most single cycle differences are not statistically significant, so I’m looking at longer periods of time, but the 2015 to 2017 champ looks to be Florida. More details on this later but I’m resisting the temptation to bring Sally back pending further analysis…but only barely!

The short run (2015 to 2017) state level data doesn’t look terribly exciting. I’m still digesting the longer run state level data, but up top there’s a chart to tide you over. NCLB required all states to take math and reading NAEP starting in 2003, so here is the full period (2003 to 2017) gains for 8th grade math and reading by state. These are numerical gains subtracting 2003 from 2017 scores by state.

The 2003 to 2017 period is a natural one to examine, but so too is the post 2009 period, for a few different reasons. The 2009 NAEP happened both in the early days of the Obama administration and during the outset of the Great Recession. NAEP redid the 4th and 8th grade science exams that year. The inclusion of “non-tested” subjects is healthy in my view given concerns over curriculum narrowing. According to my bleary eyes two states have made statistically significant gains in all six exams for the entire available period- Arizona and Mississippi. For the 2009 to 2015 period it had been AZ alone. Arizona and Mississippi were also the only states to make statistically significant gains on all of the math and reading tests during the 2009 to 2017 period. Several states however had statistically significant gains in 3 out of 4 of the math and reading exams between 2009 and 2017-including California, Hawaii, Indiana and Wyoming. On the downside, I have not yet added up the number of significant statewide declines in scores during this period, but there are many of them.

Arizona had three flat aggregate statewide scores and a decline in 4th grade math between 2015 and 2017. At first blush Arizona and Colorado charter schools crushed the academic ball again in 2017, will dig further into details/subgroups but for now:

There is more good news in statewide charter sectors, haven’t touched the TUDA yet, stay tuned.

Will it Never Be Day?!? NAEP preview

April 9, 2018

(Guest Post by Matthew Ladner)

Shakespeare’s Henry V includes a scene in the tent of the French nobles in which the Dauphin prattles on an on about how many of the English he plans to slaughter on the following day in the battle of Agincourt. “Will it never be day!?!?!” the Dauphin exclaimed. Careful what you wish for…

NAEP’s 2017 results for Math and Reading for states and select large urban districts will become available tomorrow. I’ve got my basement primed with six laptops and a case of Monster energy drinks. Mrs. Ladner is planning to occasionally open the vault to hurl additional pizza boxes downstairs before quickly re-sealing civilization’s final defense against anarchy. A few notes:

Large score movements either up or down are rare so up is better than down but when examining a statewide average score. On the math and reading tests 10 points roughly equates to an average year worth of progress (e.g. if we did a random assignment exercise and had one group take the 4th grade math test as 4th graders and the other group as 5th graders we would expect the 5th graders to score about 10 points higher). Most gains or declines are of the incremental variety (1-3 points) but they can add up over time like in Arizona, or cancel themselves out like in New York:

Was the 2015 NAEP a disappointing blip or the start of a new trend?  A mere quarter century of national math improvement at both the 4th and 8th grade level came to an end in 2015. We’ll find out tomorrow.

If a trend, cry HAVOC and let slip the dogs of spin!  Put me down as skeptical on some sort of lagged impact of the great recession given things like:

This will be a big NAEP for the Common Core project. I have not seen any reason to doubt the Tom Loveless analysis that measured the impact as a tiny positive. A tiny positive however was not what was promised. Let’s see what happens next.

Discipline Reform. I’ve been reading the debate on the discipline reform efforts of the Obama administration. I have no idea whether discipline reform had an adverse impact on academic performance, but if we are looking to subject the notion to conjecture and refutation, I would take the most interest in the score trends of low-income students of color in early adopting jurisdictions, and I would take a greater interest in 8th grade scores/trends.

Computerized testing.  They piloted it in 2015, and then delayed the release of the 2017 NAEP to further study whether to make additional adjustments. Peggy Carr, the acting commissioner of the National Center for Education Statistics, told reporters last week “We have the full weight of scientific, psychometric evidence behind the release. We are comparing apples to apples. I think we can all be confident without a shadow of a doubt that we are going to be looking at true performance.” Perhaps understandably given the pressures involved some state chiefs took an interest in technical aspects of NAEP testing shortly after being briefed on their state scores, but the National Center for Education Statistics has a head start several years in the running on studying this issue, so I tend to believe them, unless Arizona scores tank, in which case…just kidding. 

Stay tuned to this jayblog channel and remember…



Beware of Mis-NAEPery but also NAEPilism

April 3, 2018

(Guest Post by Matthew Ladner)

The 2017 NAEP will be released next week, and a few notes seem in order. Over time, the term “mis-NAEPery” has slowly morphed into a catchall phrase to mean “I don’t like your conclusions.” Mis-NAEPery however has an actual meaning- or at least it should- which ought to be something along the lines of “confidently attributing NAEP trends to a particular policy.”

Arne Duncan for instance took to the pages of the Washington Post recently in order to lay claim to all positive NAEP trends since 1990 to his own tribe of reformer (center left):

Lately, a lot of people in Washington are saying that education reform hasn’t worked very well. Don’t believe it.

Since 1971, fourth-grade reading and math scores are up 13 points and 25 points, respectively. Eighth-grade reading and math scores are up eight points and 19 points, respectively. Every 10 points equates to about a year of learning, and much of the gains have been driven by students of color.

Duncan then proceeds to dismiss the possibility that student demographics had anything to do with this improvement, as the American student body has grown “It should be noted that the student population is relatively poorer and considerably more diverse than in 1971.” This is a contention however deserving dispute, given that the inflation adjusted (in constant 2011 dollars) income of the poorest fifth of Americans almost doubled between 1964 and 2011 once various transfers (food stamps, EITC etc.) have been taken into account. Any number of other things could also explain the positive trend, both policy and non-policy related, but never mind any of that, Mr. Duncan lays claim to all that is positive.

Duncan was not finished yet, however, as he was at pains to triangulate himself away from those nasty people who support more choice than just charter schools:

Some have taken the original idea of school choice — as laboratories of innovation that would help all schools improve — and used it to defund education, weaken unions and allow public dollars to fund private schools without accountability.

Well that sounds a bit like how a committed leftist would (unfairly) describe my pleasant patch of cactus. Arizona NAEP scores, could you please stand to acknowledge the cheers of the audience:

So the big problem in that chart are the blue columns. These charts stretch from the advent of the Obama years until the (until Tuesday) most recently available data. We won’t be getting new science data this year, so ignore the last two blue columns on the right. What we are looking at is changes in scores of 1 point in 4th grade math, -1 point in 8th grade math, 1 point in 4th grade reading and two points in 8th grade reading. There’s only one state that made statistically significant academic gains on all six NAEP tests during the Obama era, but it just so happens to be one of the ones adopting the policies uncharitably characterized by Duncan’s effort at triangulation.

There were some very large initiatives during these years- Common Core standards, teacher evaluation, etc. and we can’t be sure why the national numbers have been so flat, but let’s just say that a net gain of three scale points across four 500 scale point tests fails to make much of an impression. Supporters of the Common Core project for instance performed a bit of a Jedi mind trick around the 2015 NAEP by noting that scores were also meh in states that chose not to adopt, and that 2015 was early yet. Fair enough on the early bit, but the promise of an enormous investment of political capital in the project was not that adopting states would be equally meh, but rather that things would get better.

Where’s the BETTER?!?

Duncan’s misNAEPery however is of the garden variety- there has been far worse. Massachusetts for instance instituted a multi-faceted suite of policy reforms in 1993, and their NAEP scores increased from a bit better than nearby New Hampshire to two bits better than New Hampshire and tops in the country. So far as I can tell, there was approximately zero effort to establish micro-level evidence on any of the multiple reform efforts, or to disentangle to the extent policies were having a positive impact, which policies were doing what. That would be silly- everyone knows that standards and testing propelled MA to the top NAEP scores, and once everyone else does it we will surge towards education Nirvana Canadian PISA scores. Well, I refer the honourable gentleman to tiny blue columns in the chart I referenced some moments ago.

This is not to say that I am confident that testing and standards had nothing to do with MA’s high NAEP scores. I’m inclined to think they probably did, but some actual evidence would be nice before imposing this strategy on everyone. In Campbell and Stanley terms “Great Caesar’s Ghost! Look at those Massachusetts NAEP scores!” lacks evidence of both internal and external validity. In other words, we don’t know what caused MA NAEP scores, nor do we know who if anyone else might be able to pull it off, assuming policy had something to do with it.

So beware of mis-NAEPery my son- the jaws that bite, the claws that catch!  Also beware of NAEP nihilism. Taking off my social science cap, I will note that NAEP is an enormous and highly respected project and it is done expressly for the purpose of making comparisons. Yes we should exercise a high level of caution in so doing, and should check any preliminary conclusions reached against other sources of available evidence. The world is a complicated place with an almost infinite number of factors pushing achievement up or down at any point. There is a great deal of noise, and finding the signal is difficult. NAEP alone cannot establish a signal.

The fact that the premature conclusions drawn from the Massachusetts experience lacked evidence of internal and external validity did not mean that those conclusions were wrong but it did make them dangerous. Alas the world does not operate in a random assignment study. Policymakers must make decisions based upon the evidence at hand, NAEP and (hopefully) better than NAEP. The figure at the top of this post makes use of NAEP and there is a whole lot of top map green (early goodness) turning into bottom map purple (later badness) going on. This is a bad look assuming part of what you want out of your support of K-12 education is kids learning about math and reading in elementary and middle school. Let’s be careful, but let’s also see what happens next.


Somebody Stop Me-2017 NAEP predictions

March 2, 2018

Mamma Mia- Here we go again!

(Guest Post by Matthew Ladner)

NAEP has announced that it will release new math and reading results on April 10. I am going to do something truly foolish and dare to make a few predictions.

Making predictions is foolish because the reality is that our understanding of NAEP trends is imperfect at best. In any case, I have a theory about what has been driving NAEP gains here in the Cactus Patch, and I’m willing to make a few calls in advance on the 2017 NAEP based upon that theory.

Prediction 1: Arizona continues to improve.

Arizona’s improvement process is multi-faceted but the elephant in the room in my book is a hyper-active open enrollment market, the nation’s largest and most geographically inclusive charter sector and private choice programs. Arizona lead the nation in academic gains even during a period in which it had the largest cuts in spending due to the Great Recession. If we can do this during a period of funding cuts, we ought to manage it during a period of funding recovery given the broad consistency of policy. Note however that the state’s A-F letter grades has been turned off during the entirety of the 2015 to 2017 period, the demographics of students have continued to move further into majority-minority status, etc. Put me down for Arizona improving anyway based on the AZMerit improving in both 2016 and then again in 2017.

Prediction 2: North Carolina climbs.

Recent NAEP trends have not been great in NC. Notice in the Reardon tables that North Carolina starts with a (good) greenish tint on the top map-showing good early scores-but turns purple on the second map (bad) on cohort NAEP gains during the 2011 to 2015 period.

I suspect North Carolina will do better in 2017 based upon the information in this news report- basically that while statewide enrollment is growing, school district enrollment has been relatively flat due to the growth of choice options. North Carolina lifted a statewide cap on charter schools in 2011, created voucher programs for low-income and special needs children. I’m not sure whether they’ve done enough to start the open-enrollment virtuous cycle, but I think they may have done enough to shake things up a bit.

Third prediction: Indiana improves.

Give Indy a good stare in the Reardon chart, and it looks a bit like the green/purple undesirable combo between the two maps. It looks to me though like the districts are getting into the choice act (until 2007 you had to pay tuition in order to attend an out of zone school) and so I’m willing to buy some Hoosier stock.

Anyone else willing to dare a prediction should spell out their theory/evidence in the comment section.


Why Arizona Charter Leaders Should Feel Confident About the 2017 NAEP

March 2, 2017


(Guest Post by Matthew Ladner)

So the 2017 NAEP is underway as we speak. As the NAEP released all six 2015 exams a cornucopia of good news spilled out for Arizona as a whole and (especially) for Arizona charters. Statewide Arizona has been leading the nation in 4th to 8th grade cohort gains since 2009. That’s a pretty good measure of overall school quality for those grades, as the demographic profile of the cohort isn’t likely to change much for a cohort between (for instance) the time when they were 4th graders in 2011 to when they were 8th graders in 2015. Kids will come and go of course, but absent a DC level of gentrification the academic ability of those coming and going should not skew heavily in a particular direction. You also have a measured amount of sampling error in both the 4th and 8th grade measures, but these are quite modest.

Arizona’s charter schools rocked all 2015 NAEP exams in the fashion of a New England state. This is quite impressive given the very modest level of per-pupil spending in Arizona charters, the majority minority student population etc. Sampling error is a bigger issue with regards to charter schools, but as you can see in the figure above, the state’s AZMerit exam tells us a story very similar to NAEP, and does not involve sampling. AZMerit in short provides backup to the findings in NAEP.

The figure above should lend confidence to Arizona charter leaders that they are likely to rock the NAEP again in 2017. The figure shows 4th and 8th grade NAEP scores from 2015 (proficiency rates) and then the proficiency rates for districts and charters from 2016 on AZMerit. These tests are not scaled exactly the same so you would not expect them to match up exactly, but the similarity in the pattern provides confidence that 2015 was not a fluke. Some of the differences between Arizona charters and districts can be attributed to differences in student demographics, although not nearly so much as sometimes are imagined, but New Hampshire doesn’t have any excuses for losing out to Arizona charter schools.

On the final set of columns on the right, for perspective, Arizona charters scored just a smidge below the highest performing state (MA) while Arizona districts would be right around the national average. Drawing random samples of students leaves room for goofy variation in subgroup scores, but there isn’t any reason to believe such goofiness would skew the same way in six different samples unless one wants to believe the NAEP is stacking the deck for charter schools in Arizona for some nefarious reason. While they were at it, the same conspirators would have to have infiltrated the AZMerit as well.

Absent goofiness and based on what we see in the AZMerit, I’m expecting Arizona charter students to CeleNAEP Good Times again in 2017. The statewide trend will be of even greater importance. Let’s see what happens next.


%d bloggers like this: