Something Rotten in the State of NAEP?

(Guest Post by Matthew Ladner)

So if you measure the learning gains for children with disabilities on the four main NAEP exams for the entire period all 50 states and the DC have participated, you get the information in the above chart. Last week, the Bluegrass Institute’s Richard Innes alerted me in the comments and by email about fishy exclusion rates for children with disabilities and English Language Learners. I had only casually examined the exclusion rates, but having examined them more closely, I’m concerned.

The 2011 NAEP included standards for inclusion, which include 95% of all students selected for testing, including 85% of students with disabilities or classified as English Language Learners. One might possibly infer that some states were playing games and tricks with excluding such students in the past, and that simply listing the rates wasn’t doing the trick. This year, they listed expected standards and provided the gory details in an Appendix. On the conference call regarding the results, the NAEP team took pains to note this innovation.

So, as you can see, half of the states in the Top 10 gainers for children with disabilities just so happen to be states that violated the inclusion standards on one or more NAEP exam. Hmmm. Moreover, some of them didn’t just barely miss these standards, but instead chose to commit violence against them.

Maryland led the nation in gains among children with disabilities….or did they? Maryland’s inclusion rate for children with disabilities on the 4th grade reading test in 2011: 31%, which though completely pathetic actually beat the 30% rate for children with disabilities on the 8th grade reading test. The ELL rates were almost as bad.

The only other state to sink into the 30s? That would be second place Kentucky, which also excluded an enormous number of ELL students from NAEP examination. The math exams were better than the reading, but lo and behold- there is Maryland again falling below inclusion standards. Maryland failed to meet the 95% overall inclusion standard on 3 out of the 4 exams in 2011.

I have run the numbers for gains among children who are neither disabled nor ELL, and something real and positive is happening in Maryland: scores are up. It is however obvious that the NAEP created these standards for a reason, and have invited people to make up their own minds about whether to throw a skeptical flag in the air.

I’m throwing my flag. I don’t know if it explains all of the gains in Maryland and Kentucky, but it seems pretty obvious to me the results from those two states and perhaps others ought not to be considered comparable to the other states.

I’ve been told and I find it credible that these exclusions have only a small impact on the statewide numbers. Can we imagine however that very high exclusion rates for ELL students will not heavily bias the Hispanic number? Or that sky-high special ed exclusions won’t inflate a variety of subgroup scores? Or that excluding many of both of these subgroups won’t impact your Free and Reduced lunch eligible sample?

So given that the Congress mandated participation in NAEP as a part of NCLB, a mandate which all the federalist bones in my body find quite reasonable, perhaps it would be a jolly good idea for Congress to mandate minimum inclusion rates along with participation when reauthorization finally rolls around. Caesar’s wife must be above suspicion.


4 Responses to Something Rotten in the State of NAEP?

  1. Student of History says:


    The states and districts that are the most aggressive at implementing nonacademic curricula and practices are also the most informed on how to manipulate NAEP. They plan in advance to be able to point to NAEP for vindication. Most people do not know it only tests a small percentage and the ed writers who spread the gospel love to imply that NAEP makes a great affirmative defense until you point out the small percentage involved.

    It was a real favorite of Atlanta Public Schools and Beverly is still trying to claim there cannot have been widespread cheating because of NAEP. When I took a look at APS’ numbers before Hall resigned and cheating was just suspected, APS’ NAEP scores showed that it was representing less than 5% Hispanic. Not a testable cohort which was not true.

    That analysis also showed just how radically the reading test and what and how literacy was measured had been revised just to keep reading steady.

    People need to better appreciate just how little of what is being identified by either NAEP or PISA is a test of academic knowledge as traditionally understood. Both are called assessments for a reason and it’s not just pretension behind the word choice.

    Just as with the forthcoming SBAC and PARCC under Common Core. I forget who it was, maybe Shulman, who said if you insist on measuring what we are doing in schools, we will simply misrepresent the nature of what is being measured.

    Still going on but now it’s in hyperdrive.

  2. Greg Forster says:

    I’m throwing my flag.

    Wait. You have a flag? I don’t have a flag! How come I didn’t get a flag?

  3. Glad to see your flag getting tossed on the NAEP exclusion/inclusion issue. You are not alone.

    I tossed mine way back in 1999.

    In the October 5, 2011 Kentucky Board of Education meeting, Kentucky Commissioner of Education Terry Holliday admitted that the state’s NAEP scores are in question because of exclusion issues that are driven by Kentucky’s policy of allowing all tests, including reading tests, to be read to the commonwealth’s students. That sets up an automatic exclusion from NAEP, which will have none of this read-kids-the-reading-test nonsense.

    I continue to post information on this topic in the Bluegrass Policy Blog at for those who have interest.

    I also have a great article about other common pitfalls in NAEP interpretation here:

  4. allen says:

    Let’s keep in mind that there’s only cheating when there’s something worthwhile at stake. Educrats are now starting to feel the heat of the spotlight and some, plenty I’m afraid, are going to cheat.

    When you’ve never been measured on your results the first reflex is to try to maintain the previously salubrious state of affairs by means foul.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s