A Closer Look at DC NAEP Scores

January 12, 2012

(Guest Post by Matthew Ladner)

A few months ago, I provided a quick analysis of DCPS NAEP scores under Michelle Rhee. Having looked into the fine details, I believe that I underestimated the positive trend in DCPS reading scores during the 2007-2011 period.

NAEP has long dealt with a tricky issue with varying inclusion rates for special education and English language learners between jurisdictions. In 2011, the NAEP adopted inclusion rate standards for ELL and SD students, and notified readers of jurisdictions that violated those standards in an appendix.

Some states and jurisdictions had far more successful efforts to comply with these efforts than others. As you can see from the figure below, DC would have been far out of compliance with these standards (had they been in place) during the 1990s and (especially) in 2007. In 2007, DCPS had excluded nearly three times as many students as permissible under the 2011 standards.

So in 2007, DCPS officials excluded 14% of students from 4th Grade NAEP testing, and in 2011 that figure fell to 3% (the inclusion for all students standard in 2011 was 95%). In 2007, DCPS stood far out of compliance, but came well within compliance in 2011. This is all well and fine, other than the fact that it complicates our ability to assess the recent history of DC NAEP gains.

In order to get a clearer picture on this, I decided to run 4th Grade NAEP scores for students outside of ELL or special education programs. This should minimize the impact of inclusion policy changes. Examined in this fashion, you get the following results:

Recall that the unadjusted total scores for 4th grade reading jumped from 197 in 2007 to 202 in 2009 but dropped back a point to 201 in 2011. That is a four point gain in four years, which ranks in meh territory. Given Figure 1 above, I am not exactly inclined to trust those scores, and in fact out second table tells quite a different story: general education students in DC made a 10 point gain between 2007 and 2011 on 4th grade reading. Ten points approximately equals a grade level worth of progress, so it is fair to say that DCPS general education 4th graders were reading approximately as well as 2007 general education 5th graders. Ten points ranks as the largest reading gain in the nation during this period for these students. Mind you, a 209 score for non-Ell and non-special ed students is still terribly low. Only gains will get DC out of the cellar, however, and DC banked solid gains during this period.

If you combine 4th and 8th grade reading gains for general education students, and only look at Free and Reduced lunch eligible students for a bit of socio-economic apples to apples, here is what you find:

DC students had the largest general education 4th grade reading gains in the country, and tie for first in the combined 4th and 8th grade reading gains. The District of Columbia, in short, made very substantial reading gains during the 2007-2011 period.

21 Comments | NAEP | Tagged: , , | Permalink
Posted by matthewladner

Something Rotten in the State of NAEP?

November 10, 2011

(Guest Post by Matthew Ladner)

So if you measure the learning gains for children with disabilities on the four main NAEP exams for the entire period all 50 states and the DC have participated, you get the information in the above chart. Last week, the Bluegrass Institute’s Richard Innes alerted me in the comments and by email about fishy exclusion rates for children with disabilities and English Language Learners. I had only casually examined the exclusion rates, but having examined them more closely, I’m concerned.

The 2011 NAEP included standards for inclusion, which include 95% of all students selected for testing, including 85% of students with disabilities or classified as English Language Learners. One might possibly infer that some states were playing games and tricks with excluding such students in the past, and that simply listing the rates wasn’t doing the trick. This year, they listed expected standards and provided the gory details in an Appendix. On the conference call regarding the results, the NAEP team took pains to note this innovation.

So, as you can see, half of the states in the Top 10 gainers for children with disabilities just so happen to be states that violated the inclusion standards on one or more NAEP exam. Hmmm. Moreover, some of them didn’t just barely miss these standards, but instead chose to commit violence against them.

Maryland led the nation in gains among children with disabilities….or did they? Maryland’s inclusion rate for children with disabilities on the 4th grade reading test in 2011: 31%, which though completely pathetic actually beat the 30% rate for children with disabilities on the 8th grade reading test. The ELL rates were almost as bad.

The only other state to sink into the 30s? That would be second place Kentucky, which also excluded an enormous number of ELL students from NAEP examination. The math exams were better than the reading, but lo and behold- there is Maryland again falling below inclusion standards. Maryland failed to meet the 95% overall inclusion standard on 3 out of the 4 exams in 2011.

I have run the numbers for gains among children who are neither disabled nor ELL, and something real and positive is happening in Maryland: scores are up. It is however obvious that the NAEP created these standards for a reason, and have invited people to make up their own minds about whether to throw a skeptical flag in the air.

I’m throwing my flag. I don’t know if it explains all of the gains in Maryland and Kentucky, but it seems pretty obvious to me the results from those two states and perhaps others ought not to be considered comparable to the other states.

I’ve been told and I find it credible that these exclusions have only a small impact on the statewide numbers. Can we imagine however that very high exclusion rates for ELL students will not heavily bias the Hispanic number? Or that sky-high special ed exclusions won’t inflate a variety of subgroup scores? Or that excluding many of both of these subgroups won’t impact your Free and Reduced lunch eligible sample?

So given that the Congress mandated participation in NAEP as a part of NCLB, a mandate which all the federalist bones in my body find quite reasonable, perhaps it would be a jolly good idea for Congress to mandate minimum inclusion rates along with participation when reauthorization finally rolls around. Caesar’s wife must be above suspicion.

4 Comments | research reports, school accountability | Tagged: , , , , , , | Permalink
Posted by matthewladner

%d bloggers like this: