Grade Retention is Common Nationally but Effective in Florida

February 28, 2012

(Guest Post by Matthew Ladner)

I came across an interesting study from NCES recently concerning the practice of grade retention that creates yet another hole in NEPC boat regarding their Florida theories. In fact, here is a link to a study from the ASU precursor to NEPC by Columbia Teacher College Professor Chatterji (one of the NEPC critics) from 2003 calling on Florida to “rethink sanction and retention policies in light of new and past research showing that retention does not improve student achievement.” 

Now you can look at the below figure and ask yourself just who needs to reconsider what. The red line is FCAT 1 scores for Black students, the Green line is for Hispanic students, and the blue line is for all students.

The NEPC boat is already sitting on the floor of the ocean, but hey, why not drop a depth charge on it?

The main pet theory of the NEPC squad has been that Florida’s 4th grade NAEP scores have been profoundly warped by the state’s retention policy. This beats the daylights out of their Harry Potter theory, but there still is far less to it than meets the eye. Problems with this theory include a substantial improvement in 4th grade NAEP scores before the retention policy went into place, a substantial decline in retentions since the onset of the policy, and a substantial improvement in 3rd grade reading FCAT scores.  Oh and the advent of mid-year promotions and a few other things which NEPC has been either unable or unwilling to address. The peak of any aging effect would have come in 2005 and declined substantially, and yet Florida’s scores continued to rise.

An implicit assumption of this theory was that Florida is doing far more K-3 retention than other states around the country. After seeing this NCES study, I am no longer certain this is the case, especially now that Florida retention has fallen so substantially. Let’s dig into the data and find out.

State level data on grade level retention is very difficult to come by outside of Florida. However, NCES included a question about retention in their parent survey. Low and behold, 10% of parents in the NCES survey report that their child has been retained for one or more grade in grades K-8, more than 20% of low-income parents.

NCES: Students retained in one or more grade, K-8

So first off, this is quite a bit higher than I would have suspected and the trend has been rising. Given the hostility that many College of Education Professors have towards grade retention, it seems apparent that many of the teachers and administrators that go through their programs are not buying what they are selling on retention.

Now that we have a measure of retention nationally, we should explore the question of how prevalent the practice is in Florida. The Florida Department of Education provides this handy chart for the statewide numbers for retention for students in grades K-12. The technical term to describe this chart is “falling off a cliff.”

So if you rummage around in the spreadsheet provided by the Florida Department of Education on retention by grade level and add a few cells together, you can calculate that the total retention figure in Florida in 2009-2010 for Grades K-8 was 54,843.

That sounds like a lot, until you go over to the NCES Common Core Data (note to Jay, Greg and MWAB- not the academic standards, please call off the cruise missle strike :-) and learn that there were over 1.7 million students in the Florida K-8 system in 2009-10. When you do the math, it turns out that 3.9% of Florida K-8 students were retained during the 2009-2010 school year. What about the peak of Florida retention the year the 3rd grade retention policy took place in 2003-04? The total retention rate for that year was (waaaaait for it…..) 5.5%- a little more than half of the national rate that the NCES found in 2007.

We don’t have national data for K-3 retention, which is what we would need to do an ideal comparison, but the data we do have certainly establishes that there is a substantial amount of retention going on around the country, which will be having some impact on NAEP scores of states across the nation, not just Florida. Unless a state is doing far more than average, it retention is likely to be white noise overall- blips in the error term. Furthermore, it is not clear that Florida was doing more K-3 retention than the national average, even during the peak of the practice in 2003-04.

Mind you that I make no claim that retention is necessarily a good practice overall. I think there have been terrible retention practices, such as the practice of “redshirting” 9th graders in Texas back when the state gave a 10th grade exit exam. Redshirting was a widespread district level practice not mandated by state law and it was truly an awful policy basically designed to get students to drop out of school in 9th grade and thereby inflate the passing rate for the 10th grade exit exam.

There was nothing admirable about Texas redshirting. I would venture to guess that both a casual and a sophisticated analysis of data would have found it associated with higher drop out rates.

The Florida policy however is the opposite of the old Texas practice in that it is designed to set kids up to succeed rather than to fail. Not only have there been bad retention practices, there has also been a great deal of bad research done on retention that lacked the statistical rigor to establish causality. Do cancer drugs kill people, or is it the cancer? Most of the retention research doesn’t allow us to answer that sort of question.

Jay, Marcus Winters and the RAND Corp however have been applying sophisticated regression discontinuity designs to retention policies in Florida and New York City. They have found positive academic results. RAND found no self-esteem harm to students, and that NYC educators have generally positive views of the policy, to boot.

The question is not whether retention is “good” or “bad”- that all depends on how it is used. The evidence on the overall literacy effort in Florida-which includes retention as a centerpiece-is overwhelmingly positive.

10 Comments | social promotion | Tagged: , , , , , , , | Permalink
Posted by matthewladner

Social Promotion Fig Leaf

March 1, 2011

Matt Ladner and I have been testifying to state legislatures around the country about the effects of Florida’s policy to end social promotion in 3rd grade.  The policy default-retains all 3rd graders who score below a certain threshold on the state’s reading test.  There are several exemptions to being retained, but about 59% of low-achieving 3rd graders repeated the grade.

Research that Marcus Winters and I have published in the peer-reviewed journal, Education Finance and Policy, finds significant achievement benefits for students retained under the policy.  After two years the retained students outperformed their promoted counterparts by about .46 standard deviations, which is the equivalent of receiving about 6.6 additional months of reading instruction.  We compared students who barely performed above the test threshold on the 3rd grade test and were default-promoted to students who performed just below the test threshold.  This regression-discontinuity design approximates a random assignment experiment.

When we testify about this research we are now commonly being asked about a “study” from the Miami-Dade School District that claims to find the effect fades after two years.  Clearly the opponents of the policy (read: the unions) are arming folks with this to dispute our research findings.  When people oppose a policy that is supported by rigorous research it is important that they at least have a fig leaf of research to support their opposition.  The Miami-Dade report is that fig leaf.  The report concludes:

This study has replicated the procedures of theGreene and Winters  (2006)  paper  evaluat ingFlorida’s test-based promotion policy and hasderived very different judgements. Where theyconcluded that the retention policy led to significant improvements in reading for the retained students,this study finds no ultimate advantages. However,it would be a mistake to interpret this study as somekind of indictment of the Greene and Winters work. Their interpretation was valid for the way the datalooked after two years. The picture is quite different after four years

First, it is important to note that the “study” is actually a 4 page document produced by the internal research department of the Miami-Dade School District.  It has no descriptive statistics, no detailed description of the methodology, and virtually no literature review.  In short, it is extremely hard to judge the accuracy of a “study” that is little more than two graphs that have never been published, reviewed, or fully-described.

Second, the Miami-Dade internal report only claims to analyze data from the Miami-Dade School District, while our research is based on data from the entire state of Florida.  It is perfectly possible that Miami-Dade poorly implemented the policy by doing things like granting the exemptions inappropriately or failing to offer effective reading interventions for students who were retained.  Even if Miami-Dade did not have successful results with the program, the entire state did.

Third, it is inaccurate to say that the Miami-Dade “study” replicated our positive findings after two years but that those positive effects later disappeared.  Their graphs suggest that there was no positive effect of being retained in Miami-Dade 1 and 2 years after the retention decision, and then they show a positive effect in years 3 and 4, which disappears in year 5.  We found a small positive effect after one year that grew into a larger effect after two years.

Our results (even in the first two years) are completely different from those in the Miami-Dade report.  It is hard to say whether this is because they only looked at Miami-Dade while we looked at the entire state, or because they did not actually replicate our methodology.  Four pages and two graphs do not allow for a lot of nuanced analysis of the findings.

We are in the process of extending our analyses to include additional years, so we may have a better idea of whether the benefits we observed state-wide grow, shrink, or remain constant.  In the meantime, the unions have provided their research fig leaf to cover state legislators who oppose the policy regardless of what research finds.

2 Comments | social promotion | Tagged: , | Permalink
Posted by Jay P. Greene

Indiana Might Be the Next Florida

February 21, 2010

Matt has written numerous times on the remarkable progress that has been made in Florida, see for example here.  Forces are gathering in Indiana that suggests they may be next to try to full court press of Florida reforms.  The governor, the state superintendent, the Indianapolis newspaper,  and a bipartisan coalition of state legislators on the education committee seem poised to pursue some significant reforms.

First up on their agenda is passage of a bill to end the social promotion of 3rd graders who are unable to read at a basic level.  Patricia Levesque and I each have op-eds in the Indy Star on this topic , with a favorable introduction from the editor.

Check it out.

Leave a Comment » | social promotion | Tagged: , | Permalink
Posted by Jay P. Greene

The Irony of Social Promotion

January 9, 2009

In the current issue of the Economics of Education Review, Marcus Winters and I have an article about the use of exemptions to Florida’s test-based promotion policy.  Under Florida’s policy students need to perform above a certain level on the 3rd grade reading test to automatically be promoted to 4th grade.  If  students score below that level they can still be promoted if they are granted one of various exemptions.  Some of those exemptions are objectively measured, like scoring well on an alternative test or having certain special ed or English Language Learning classifications.  But other exemptions are more subjectively determined, like having a portfolio of work worthy of being promoted.

Marcus and I looked at who received those exemptions and whether being exempted was beneficial.  We found that African-American and Hispanic students were less likely to receive exemptions and get promoted, controlling for other factors.  That is, minority students with the same test scores and economic status were less likely to be exempted from retention if they fell below the testing threshold.  The test-based policy is not racially biased, since all students who lack the academic skills to pass the test may be retained.  The bias is introduced in who gets exempted from that test-based policy.

And the irony of it all is that failing to receive an exemption actually benefited those minority students academically.  That is, students who were denied the exemption and repeated third grade outperformed their promoted colleagues on achievement tests two years later.  The retained students had more academic skill at the end of 4th grade than their comparable promoted peers at the end of 5th grade — despite being exposed to one less grade of curriculum. 

Minority students denied the exemptions may have been the vicitms of discrimination, but they ended-up making greater academic progress as a result.  Receiving those exemptions wasn’t doing many of the other students any favors.

The St. Pete Times has an article on the study today and had a blog post recently.

2 Comments | social promotion | Tagged: , , , | Permalink
Posted by Jay P. Greene

Eduwonkette and Eduwonk Aren’t Edumarried?

July 8, 2008

The New York Sun had a nice profile yesterday of Eduwonkette.  Well, it’s not exactly a profile because Eduwonkette writes anonymously.  In the article some folks complain that her anonymity is a problem: “A co-director of the Education Sector think tank, Andrew Rotherham, suggested on his blog Eduwonk that Eduwonkette might be unfairly pretending to be unbiased because she has ‘skin in the game… It’s this issue of you got all this information to readers, without a vital piece of information for them to put it in context.'”

I think Andy’s mistaken on this. (Did they have some kind of edu-break-up?)  The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.  I wish we all spent a whole lot less time analyzing people’s motives and a whole lot more time on their evidence and arguments. 

The only major problem with anonymity is lack of responsibility for being wrong.  There is a reputational price for making bad arguments or getting the evidence wrong that Eduwonkette avoids paying professionally — although she does pay a reputational price to the name brand of Eduwonkette.

Speaking of being wrong, Eduwonkette knocks the study Marcus Winters, Julie Trivitt, and I released today through the Manhattan Institute.  She complains: “It may be an elegantly executed study, or it may be a terrible study. The trouble is that based on the embargoed version released to the press, on which many a news article will appear today, it’s impossible to tell. There is a technical appendix, but that wasn’t provided up front to the press with the glossy embargoed study. Though the embargo has been lifted now and the report is publicly available, the technical appendix is not.”

This isn’t correct.  Embargoed copies of the study were provided to reporters upon their request.  If they requested the technical report, they could get that.  Both were available well in advance to reporters so that they could take time to read it and circulate it to other experts before writing a story.  Both the study and the technical report were made publicly available today (although there seems to be a glitch with the link to the technical report that should be fixed within hours).  The technical report can be found here.

And while we are on the subject of Eduwonkette being wrong, her attacks on test-based promotion policies are overdone.  The Jacob and Lefgren paper does raise concerns, but there is more positive evidence from the experience in Florida.  As I wrote in a previous post: “In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.”

Our results may actually be consistent with what Jacob and Lefgren find.  We find academic benefits for students retained in third grade.  They find: “that grade retention leads to a modest increase in the probability of dropping out for older students, but has no significant effect on younger students.”  It could be that test-based promotion is more beneficial when done with younger students.  It could also be that the policy has positive effects on achievement with some cost to graduation. 

And particularly severe problems with the integrity of test results used for promotion decisions in Chicago may limit the ability to generalize from Chicago’s experience.  In Chicago it may have been easier to move retained students forward by cheating on the next test than actually teaching them the basic skills they need to succeed in the next grade.

Besides, I’m sure that Edwuonkette wouldn’t put too much stock in Jacob and Lefgren’s non-peer-reviewed paper released straight to the public.  I’m sure she would be consistent in her view that: “By the time the study’s main findings already have been widely disseminated, some sucker with expertise in regression discontinuity may find a mistake while combing through that appendix, one that could alter the results of the study. But the news cycle will have moved on by then. Good luck interesting a reporter in that story… So as much as I like to kvetch about peer review and the pain and suffering it inflicts, it makes educational research better. It catches many problems and errors before studies go prime time, even if it doesn’t always work perfectly.”  

Or do these standards only apply to studies whose findings she doesn’t like?   If Eduwonkette isn’t careful she might get a reputation.

6 Comments | social promotion | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

Anti-Social Promotion

June 30, 2008

Georgia joined several other major school systems, including Florida, Texas, New York City, and Chicago, when its legislature mandated that student promotion to the next grade be linked to performance on standardized tests.  All failing students in certain grades must re-take the test, according to state law.  Students failing twice can be promoted anyway if the parents, principal, and teachers review the student’s other work and agree that promotion would be beneficial, but it is clear that this should be the exception, not the rule. 

But according to a report in yesterday’s Atlanta Journal Constitution, “school districts are promoting the vast majority of [failing] students anyway… In 2007, for instance, 92 percent of the nearly 9,500 eighth-graders who couldn’t pass the math CRCT were promoted.”    In Clayton County 97 percent of students who failed the re-test to get promoted or simply didn’t take the re-test were promoted to the next grade.  When asked about why these students were promoted, the District issued a statement that said, “the philosophy of prior administrators was to promote students who failed and provide them remediation.”

Oh.  I see.  The law says that students unable to pass the state’s test ought to be retained but Clayton County school officials had a different philosophy.  Their philosophy was that they don’t have to follow the law. 

Districts weren’t simply exploiting the loophole of promoting students by consent of parents, teachers, and principals.  Districts were directly violating the law by promoting students who did not even take the re-test, which is clearly mandated for all students before promotion by alternative means can be considered.  As the AJC writes, “About one in five students missed the retest after failing a high-stakes CRCT in 2006 and 2007.  Eighty percent were promoted anyway.”

Non-compliance by public school officials with legal mandates is an enormous obstacle to education reform.  Even if one thinks that this particular mandate on social promotion is mis-guided, the frequency with which school officials feel free to flout legally mandated attempts at reform undermines the potential for any reform strategy.

In an earlier study I led on NCLB’s choice provisions, I found that the requirement that chronically failing schools inform parents that they had the option to transfer to other schools was undermined by the failure of schools to inform parents of that option.  (The study was published in the Georgetown Journal of Law and Public Policy, but an earlier free version of it can be found here.)

Problems with implementation have also been central themes in the discussions on this blog on Response to Intervention, Reading First, more Reading First, and more Reading First. Until we develop the political will to actually enforce the reforms we do adopt, we shouldn’t expect much from them.

In addition, the evidence from Florida suggests that limiting social promotion by linking testing to retention is actually a very beneficial reform.  In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.

But the more important point is that each school district, school, or teacher cannot decide whether to comply with the law or not.  These are public schools and the public, through their elected representatives, is entitled to set policies that govern how those schools will operate.

11 Comments | social promotion | Tagged: , , , | Permalink
Posted by Jay P. Greene

%d bloggers like this: