(Guest Post by Matthew Ladner)
A few months ago, I provided a quick analysis of DCPS NAEP scores under Michelle Rhee. Having looked into the fine details, I believe that I underestimated the positive trend in DCPS reading scores during the 2007-2011 period.
NAEP has long dealt with a tricky issue with varying inclusion rates for special education and English language learners between jurisdictions. In 2011, the NAEP adopted inclusion rate standards for ELL and SD students, and notified readers of jurisdictions that violated those standards in an appendix.
Some states and jurisdictions had far more successful efforts to comply with these efforts than others. As you can see from the figure below, DC would have been far out of compliance with these standards (had they been in place) during the 1990s and (especially) in 2007. In 2007, DCPS had excluded nearly three times as many students as permissible under the 2011 standards.
So in 2007, DCPS officials excluded 14% of students from 4th Grade NAEP testing, and in 2011 that figure fell to 3% (the inclusion for all students standard in 2011 was 95%). In 2007, DCPS stood far out of compliance, but came well within compliance in 2011. This is all well and fine, other than the fact that it complicates our ability to assess the recent history of DC NAEP gains.
In order to get a clearer picture on this, I decided to run 4th Grade NAEP scores for students outside of ELL or special education programs. This should minimize the impact of inclusion policy changes. Examined in this fashion, you get the following results:
Recall that the unadjusted total scores for 4th grade reading jumped from 197 in 2007 to 202 in 2009 but dropped back a point to 201 in 2011. That is a four point gain in four years, which ranks in meh territory. Given Figure 1 above, I am not exactly inclined to trust those scores, and in fact out second table tells quite a different story: general education students in DC made a 10 point gain between 2007 and 2011 on 4th grade reading. Ten points approximately equals a grade level worth of progress, so it is fair to say that DCPS general education 4th graders were reading approximately as well as 2007 general education 5th graders. Ten points ranks as the largest reading gain in the nation during this period for these students. Mind you, a 209 score for non-Ell and non-special ed students is still terribly low. Only gains will get DC out of the cellar, however, and DC banked solid gains during this period.
If you combine 4th and 8th grade reading gains for general education students, and only look at Free and Reduced lunch eligible students for a bit of socio-economic apples to apples, here is what you find:
DC students had the largest general education 4th grade reading gains in the country, and tie for first in the combined 4th and 8th grade reading gains. The District of Columbia, in short, made very substantial reading gains during the 2007-2011 period.
Jay P. Greene's Blog 
How about the cheating scandal? http://www.usatoday.com/news/education/2011-03-28-1Aschooltesting28_CV_N.htm
here’s another article on the cheating at DCPS.
http://www.americanthinker.com/2012/01/unsolved_mystery_dc_public_schools_cheating_scandal.html
Jflycn-
NAEP is a low-stakes test that evaluates random samples of students in entire states (and in this case DC) but not individual schools and certainly not individual teachers. Incentives for the sort of cheating alleged in the link you provided don’t exist, and I haven’t seen any allegation of cheating on NAEP in DC.
Good point. I agree with you.
NAEP Scores
8th grade reading
2002 240
2003 239
2005 238
2007 241
2009 240
2011 237
NAEP 8th grade reading
year Higher income Lower Income
2003 248 232
2005 249 234
2007 253 234
2009 263 232
2011 259 228
Phillip-
Glad to see you are familar with the NAEP data explorer. If you will take a look at figure 1 above (or delve into the appendix of the 2011 NAEP reading report) you will see that the 2007 13% of students were excluded from the 2007 8th Grade reading NAEP, while only 3% were excluded from the 2011 exam. The samples, in short, were very different.
If you will go back to your data explorer and track the 8th grade reading progress for DC general education students from 2007 to 2011, you’ll find that such students scored seven points higher in 2011 than in 2007. If you check the national average for those same sorts of students, you’ll find the progress was three points. We wouldn’t expect the ELL and Special Ed students to show better scores in 2011 than in 2007 given that approximately four times as many students were excluded in 2007.
NAEP
DCPS
4th Grade
Reading
Year High Income Low income
2003 206 182
2005 215 183
2007 216 188
2009 230 193
2011 234 188
Year Overall
2002 191
2003 188
2005 191
2007 197
2009 203
2011 201
Year Black Hispanic White
2002 188 193 248
2003 184 187 254
2005 187 193 252
2007 192 206 258
2009 195 207 257
2011 191 204 255
Exactly- vanishingly few of the 14% of students excluded on the basis of ELL and SPED in 2007 were likely to be high income Anglos in DC. The high exclusion rates before 2011, especially the rate in 2007, probably played a substantial role in inflating test scores for low income students.
Look at your Hispanic numbers between 2005 and 2007- a 13 point gain in two years? I suspect that the doubling of the exclusion rate between 2005 and 2007 played a substantial role in driving that gain.
This is from the NAEP report:
Score Gaps for Student Groups
In 2011, Black students had an average score that was 64 points lower than White students. This performance gap was not significantly different from that in 2002 (60 points).
In 2011, Hispanic students had an average score that was 51 points lower than White students. This performance gap was not significantly different from that in 2002 (55 points).
In 2011, students who were eligible for free/reduced-price school lunch, an indicator of low family income, had an average score that was 46 points lower than students who were not eligible for free/reduced-price school lunch. This performance gap was wider than that in 2002 (25 points).
Phillip-
Your NAEP report references would be even more impressive if they were relevant to the topic at hand. Just as a reminder, that topic is not whether DCPS continues to have massive academic problems (it does) but rather whether or not it has been improving, and what data should be used to make such judgements.
Matt,
“A Closer Look at DCPS NAEP Scores”, one “positive trend is that:
In 2011, Black students had an average score that was 64 points lower than White students. This performance gap was not significantly different from that in 2002 (60 points).
What role might regression to the mean play in these achievements? Places like DC and RI had low scores to start with, so it is statistically probable that their scores would improve.
Also, looking at your second graph, we see the same upward trend in the NAEP, excepting from 2002-2003.
And, if I am correctly reading between your lines, the unstated supposition is that the general lackluster performance in DCPS under the stewardship of Michelle Rhee is due to the increase in the percentage of SD and ELL students taking the test
If you will click the first link, you will see that DC came in 4th place nationwide in combined gains on the four main NAEP exams for the 2007 to 2011 period for FRL kids even before any considerations of the substantial changes in exclusion rates.
In all of the debates back and forth over Rhee’s record, what effect does it have on DCPS that so many kids have migrated to charter schools during the 2000s? We’re always told that charter schools are getting the “most motivated” kids, and if that’s true in DC (the people making that claim never feel the need to provide evidence for it), wouldn’t that be a huge negative pressure on DCPS scores on NAEP?
Stuart-
You are correct, if the assertion were true. Since we’ve seen very large improvements in DCPS scores since charter schools were introduced, I’d dare to guess that there either isn’t much to the assertion and/or the competitive effects drown them out.
Of course there are plent of other things going on, but one would struggle to find evidence that charter schools have had a negative impact on the academic performance of DCPS.
[…] Ladner says, essentially, that increased numbers of illegally excluded ESL students brought heroic DC chancellor Rhee to grief – that is, when NAEP no longer allowed them to be excluded. Making their scores count made his heroine look bad, he concludes. […]
Good post there
[…] based on a few years of inconclusive data we can determine that a reform has “failed.” There is mixed evidence regarding the results of the changes made in D.C., but as Matt DiCarlo pointed out in a takedown of […]
[…] progress is finally being made in 8th grade reading by both low-income and black students. Perhaps Jay Greene is correct and gains were slow to register because, in 2007, the NAEP sample started to include […]
[…] schools in DC have gained ground on national tests over the past 15 years, but much of that gain is due to the changing demographic composition of DC’s student body. […]