(Guest Post by Matthew Ladner)
I came across an interesting study from NCES recently concerning the practice of grade retention that creates yet another hole in NEPC boat regarding their Florida theories. In fact, here is a link to a study from the ASU precursor to NEPC by Columbia Teacher College Professor Chatterji (one of the NEPC critics) from 2003 calling on Florida to “rethink sanction and retention policies in light of new and past research showing that retention does not improve student achievement.”
Now you can look at the below figure and ask yourself just who needs to reconsider what. The red line is FCAT 1 scores for Black students, the Green line is for Hispanic students, and the blue line is for all students.
The NEPC boat is already sitting on the floor of the ocean, but hey, why not drop a depth charge on it?
The main pet theory of the NEPC squad has been that Florida’s 4th grade NAEP scores have been profoundly warped by the state’s retention policy. This beats the daylights out of their Harry Potter theory, but there still is far less to it than meets the eye. Problems with this theory include a substantial improvement in 4th grade NAEP scores before the retention policy went into place, a substantial decline in retentions since the onset of the policy, and a substantial improvement in 3rd grade reading FCAT scores. Oh and the advent of mid-year promotions and a few other things which NEPC has been either unable or unwilling to address. The peak of any aging effect would have come in 2005 and declined substantially, and yet Florida’s scores continued to rise.
An implicit assumption of this theory was that Florida is doing far more K-3 retention than other states around the country. After seeing this NCES study, I am no longer certain this is the case, especially now that Florida retention has fallen so substantially. Let’s dig into the data and find out.
State level data on grade level retention is very difficult to come by outside of Florida. However, NCES included a question about retention in their parent survey. Low and behold, 10% of parents in the NCES survey report that their child has been retained for one or more grade in grades K-8, more than 20% of low-income parents.
NCES: Students retained in one or more grade, K-8
So first off, this is quite a bit higher than I would have suspected and the trend has been rising. Given the hostility that many College of Education Professors have towards grade retention, it seems apparent that many of the teachers and administrators that go through their programs are not buying what they are selling on retention.
Now that we have a measure of retention nationally, we should explore the question of how prevalent the practice is in Florida. The Florida Department of Education provides this handy chart for the statewide numbers for retention for students in grades K-12. The technical term to describe this chart is “falling off a cliff.”
That sounds like a lot, until you go over to the NCES Common Core Data (note to Jay, Greg and MWAB- not the academic standards, please call off the cruise missle strike :-) and learn that there were over 1.7 million students in the Florida K-8 system in 2009-10. When you do the math, it turns out that 3.9% of Florida K-8 students were retained during the 2009-2010 school year. What about the peak of Florida retention the year the 3rd grade retention policy took place in 2003-04? The total retention rate for that year was (waaaaait for it…..) 5.5%- a little more than half of the national rate that the NCES found in 2007.
We don’t have national data for K-3 retention, which is what we would need to do an ideal comparison, but the data we do have certainly establishes that there is a substantial amount of retention going on around the country, which will be having some impact on NAEP scores of states across the nation, not just Florida. Unless a state is doing far more than average, it retention is likely to be white noise overall- blips in the error term. Furthermore, it is not clear that Florida was doing more K-3 retention than the national average, even during the peak of the practice in 2003-04.
Mind you that I make no claim that retention is necessarily a good practice overall. I think there have been terrible retention practices, such as the practice of “redshirting” 9th graders in Texas back when the state gave a 10th grade exit exam. Redshirting was a widespread district level practice not mandated by state law and it was truly an awful policy basically designed to get students to drop out of school in 9th grade and thereby inflate the passing rate for the 10th grade exit exam.
There was nothing admirable about Texas redshirting. I would venture to guess that both a casual and a sophisticated analysis of data would have found it associated with higher drop out rates.
The Florida policy however is the opposite of the old Texas practice in that it is designed to set kids up to succeed rather than to fail. Not only have there been bad retention practices, there has also been a great deal of bad research done on retention that lacked the statistical rigor to establish causality. Do cancer drugs kill people, or is it the cancer? Most of the retention research doesn’t allow us to answer that sort of question.
Jay, Marcus Winters and the RAND Corp however have been applying sophisticated regression discontinuity designs to retention policies in Florida and New York City. They have found positive academic results. RAND found no self-esteem harm to students, and that NYC educators have generally positive views of the policy, to boot.
The question is not whether retention is “good” or “bad”- that all depends on how it is used. The evidence on the overall literacy effort in Florida-which includes retention as a centerpiece-is overwhelmingly positive.
Jay P. Greene's Blog 
You make one very big mistake in this analysis, I think – you’re comparing the nine-year stats for the nation with the one-year stats for Florida. If 5.5% of children are retained in a single year, that would imply almost half of children would be retained over the course of their time in elementary school(neglecting multiple retention), which is far higher than the 10% elsewhere. To this day, Florida retains far more than most other states, assuming your numbers are accurate.
Now, of course if retention produces results like we’ve seen in Florida, I’m all for it. But accurate analysis is its own reward.
Alsadius-
You are correct that the numbers are different because the national numbers are counting students that have ever been retained K-8, whereas the Florida numbers are aggregate snapshot figures. The national figure will underestimate overall retention because some of the students retained will have been retained more than once, and the Florida numbers will underestimate retention because some of the kids in the “snapshot” retention will have been retained in a previous year.
I am assuming that instances of children being retained in more than one grade are fairly rare in both instances. Florida keeps track of double retentions in K-3 and only a little over 1,000 students statewide had been retained twice. Some of the NCES survey students will have been retained more than once because the question is asks if they had been retained one or more grade, and we are not capturing the “or more.”
Overall it seems likely to me that the error is fairly small in both cases, and that it cancels out with both figures being a bit of an underestimate. The differences in the methodology could explain some of the difference, but Florida has room to give and the trends couldn’t be clearer that retention has been rising nationally but declining in Florida.
There are a few different effects here. Just to explicitly identify them and provide evidence on each, they are:
1) FL’s retention policy benefits students who are retained. (See http://www.uark.edu/ua/der/People/Greene/Revisiting_Grade_Retention.pdf and in a forthcoming piece in Education Finance and Policy we find that these benefits endure through 7th grade.)
2) FL’s retention policy motivates schools and students to improve so that students never have to be retained (See Figure 2 above with the decline in students scoring at Level 1 in 3rd grade. It’s not conclusive evidence but the positive trend suggests that the policy had this systemic effect.)
3) FL’s retention policy cannot account for the exceptional improvement in FL’s 4th grade NAEP scores. (In addition to the arguments Matt provides above, Marcus Winters has a forthcoming analysis of individual level data that shows Matt’s arguments are right on track.)
[…] retention policy. (For actual arguments in favor of mandatory retention, see Matthew Ladner here and here; for actual arguments against mandatory retention, see Scott McLeod here and […]
Matthew: Sure, you can’t get precise stats, because the other states don’t provide nearly as good of data as Florida does. But you can at least do the obvious fudge of multiplying the annual numbers by nine, which completely inverts this post’s point.
Jay: Agreed.
Alsadius-
Thanks for setting me straight-I see the problem in my thinking now. Although very few students repeat more than one grade, each year student have some risk of being retained apart from whatever happened in the previous year. Jay suggested calculating a synthetic cohort by summing the percentages of students held back in each grade K-8. This leads to a figure lower than multiplying the snapshot by nine, but higher than the national average.
The NCES study shows that nationwide the percentage retained for poor students is more than four times as high as for poor compared to non-poor students, and about twice as high for Black as opposed to White students. Hispanic students are also almost 40 percent more likely to be retained as Anglos nationwide. If Florida followed the national trend and didn’t have a 3rd grade retention policy, we would expect a higher rate, and of course it does have the 3rd grade retention policy.
Like you said earlier, when retention helps to get the results in Figure 2, why should anyone oppose it? As far as an impact on the NAEP, retention would have played the largest role in 2005 and then would have fallen off substantially along with retention rates.
Your Common Core comment reminded me of the apocryphal story of the little old lady who told a reporter in 1964 that she was going to vote for Johnson because she had heard Goldwater wanted to get rid of TV. The reporter informed her that Goldwater didn’t want to get rid of TV, he wanted to get rid of the TVA – the Tennessee Valley Authority. She rubbed her chin and then said, “Well, I’ll vote against him anyway. Wouldn’t want to take any chances.”
Does this mean I should build a bunker in my back yard? 🙂
Wouldn’t want to take any chances!
Good post on a complex topic.
I’d add:
Within high-poverty schools, there seems to be 2 ways retention decisions/policies play out:
1. “Legitimate” discussion about the policy. Ie, teachers and principals engage in some reasonable discussion over an appropriate bar, and the inherent pros/cons.
2. “Illegitimate” discussion about the policy. Your Texas example is one.
More commonly, a bad principal who overrules teachers and socially promotes kids to bolster faux grad rates. Sometimes combined with massively deceptive “credit recovery” that doesn’t pass any smell test. Sometimes combined with caving in to any parent who complains.
Another variation is a school with an effective leader who is undermined by teachers — who lower the bar out of misbegotten sympathy, undermining the notion of standards.