Social Promotion Fig Leaf

March 1, 2011

Matt Ladner and I have been testifying to state legislatures around the country about the effects of Florida’s policy to end social promotion in 3rd grade.  The policy default-retains all 3rd graders who score below a certain threshold on the state’s reading test.  There are several exemptions to being retained, but about 59% of low-achieving 3rd graders repeated the grade.

Research that Marcus Winters and I have published in the peer-reviewed journal, Education Finance and Policy, finds significant achievement benefits for students retained under the policy.  After two years the retained students outperformed their promoted counterparts by about .46 standard deviations, which is the equivalent of receiving about 6.6 additional months of reading instruction.  We compared students who barely performed above the test threshold on the 3rd grade test and were default-promoted to students who performed just below the test threshold.  This regression-discontinuity design approximates a random assignment experiment.

When we testify about this research we are now commonly being asked about a “study” from the Miami-Dade School District that claims to find the effect fades after two years.  Clearly the opponents of the policy (read: the unions) are arming folks with this to dispute our research findings.  When people oppose a policy that is supported by rigorous research it is important that they at least have a fig leaf of research to support their opposition.  The Miami-Dade report is that fig leaf.  The report concludes:

This study has replicated the procedures of theGreene and Winters  (2006)  paper  evaluat ingFlorida’s test-based promotion policy and hasderived very different judgements. Where theyconcluded that the retention policy led to significant improvements in reading for the retained students,this study finds no ultimate advantages. However,it would be a mistake to interpret this study as somekind of indictment of the Greene and Winters work. Their interpretation was valid for the way the datalooked after two years. The picture is quite different after four years

First, it is important to note that the “study” is actually a 4 page document produced by the internal research department of the Miami-Dade School District.  It has no descriptive statistics, no detailed description of the methodology, and virtually no literature review.  In short, it is extremely hard to judge the accuracy of a “study” that is little more than two graphs that have never been published, reviewed, or fully-described.

Second, the Miami-Dade internal report only claims to analyze data from the Miami-Dade School District, while our research is based on data from the entire state of Florida.  It is perfectly possible that Miami-Dade poorly implemented the policy by doing things like granting the exemptions inappropriately or failing to offer effective reading interventions for students who were retained.  Even if Miami-Dade did not have successful results with the program, the entire state did.

Third, it is inaccurate to say that the Miami-Dade “study” replicated our positive findings after two years but that those positive effects later disappeared.  Their graphs suggest that there was no positive effect of being retained in Miami-Dade 1 and 2 years after the retention decision, and then they show a positive effect in years 3 and 4, which disappears in year 5.  We found a small positive effect after one year that grew into a larger effect after two years.

Our results (even in the first two years) are completely different from those in the Miami-Dade report.  It is hard to say whether this is because they only looked at Miami-Dade while we looked at the entire state, or because they did not actually replicate our methodology.  Four pages and two graphs do not allow for a lot of nuanced analysis of the findings.

We are in the process of extending our analyses to include additional years, so we may have a better idea of whether the benefits we observed state-wide grow, shrink, or remain constant.  In the meantime, the unions have provided their research fig leaf to cover state legislators who oppose the policy regardless of what research finds.