social promotion « Jay P. Greene's Blog

Ed Week on 3rd Grade Retention

March 27, 2012

(Guest Post by Matthew Ladner)

Ed Week on the 3rd grade retention debate, including quotes from my Foundation for Excellence in Education colleague Jaryn Emhof and Marcus Winters on his forthcoming research on persistence of the retention effect. I am very proud of our lawmakers in Arizona and especially Governor Brewer and Senator Crandall for taking action to close loopholes in the Arizona law.

Arizona has a sad history of punting on tough reforms, having dummied down the state AIMS test by dropping the cut scores and having delayed the enactment of using the AIMS test as a high school exit exam several times. The 3rd grade retention law that passed in 2010 put the new standard in place for incoming kindergarteners the following year, giving an ease-in adjustment period for the districts. Last year at the Arizona School Boards Association conference, an Arizona Superintendent confided in me that “we’ve found the loophole in the retention law, and we are getting ready to use it.”

Fortunately, Governor Brewer’s team found the loophole as well, and are taking action to close it. This law is going to be a tremendous test of character for the Arizona education community of the sort we have failed in the past. Reading intervention should have the top priority for every dime of federal funding received for K-3 students. All of the Title programs can be used to support early reading intervention, remediation, and professional development. So long as we are really going to see the policy through, I support Governor Brewer’s call for additional state resources without reservation.

The only time Arizona officeholders garner attention seems to be when they do something controversial or downright nutty. Kudos for getting some things right!

1 Comment | social promotion | Tagged: 3rd grade retention, Ed Week, Jaryn Emhof, Marcus Winters | Permalink
Posted by matthewladner

Grade Retention is Common Nationally but Effective in Florida

February 28, 2012

(Guest Post by Matthew Ladner)

I came across an interesting study from NCES recently concerning the practice of grade retention that creates yet another hole in NEPC boat regarding their Florida theories. In fact, here is a link to a study from the ASU precursor to NEPC by Columbia Teacher College Professor Chatterji (one of the NEPC critics) from 2003 calling on Florida to “rethink sanction and retention policies in light of new and past research showing that retention does not improve student achievement.”

Now you can look at the below figure and ask yourself just who needs to reconsider what. The red line is FCAT 1 scores for Black students, the Green line is for Hispanic students, and the blue line is for all students.

The NEPC boat is already sitting on the floor of the ocean, but hey, why not drop a depth charge on it?

The main pet theory of the NEPC squad has been that Florida’s 4th grade NAEP scores have been profoundly warped by the state’s retention policy. This beats the daylights out of their Harry Potter theory, but there still is far less to it than meets the eye. Problems with this theory include a substantial improvement in 4th grade NAEP scores before the retention policy went into place, a substantial decline in retentions since the onset of the policy, and a substantial improvement in 3rd grade reading FCAT scores. Oh and the advent of mid-year promotions and a few other things which NEPC has been either unable or unwilling to address. The peak of any aging effect would have come in 2005 and declined substantially, and yet Florida’s scores continued to rise.

An implicit assumption of this theory was that Florida is doing far more K-3 retention than other states around the country. After seeing this NCES study, I am no longer certain this is the case, especially now that Florida retention has fallen so substantially. Let’s dig into the data and find out.

State level data on grade level retention is very difficult to come by outside of Florida. However, NCES included a question about retention in their parent survey. Low and behold, 10% of parents in the NCES survey report that their child has been retained for one or more grade in grades K-8, more than 20% of low-income parents.

NCES: Students retained in one or more grade, K-8

So first off, this is quite a bit higher than I would have suspected and the trend has been rising. Given the hostility that many College of Education Professors have towards grade retention, it seems apparent that many of the teachers and administrators that go through their programs are not buying what they are selling on retention.

Now that we have a measure of retention nationally, we should explore the question of how prevalent the practice is in Florida. The Florida Department of Education provides this handy chart for the statewide numbers for retention for students in grades K-12. The technical term to describe this chart is “falling off a cliff.”

So if you rummage around in the spreadsheet provided by the Florida Department of Education on retention by grade level and add a few cells together, you can calculate that the total retention figure in Florida in 2009-2010 for Grades K-8 was 54,843.

That sounds like a lot, until you go over to the NCES Common Core Data (note to Jay, Greg and MWAB- not the academic standards, please call off the cruise missle strike :-) and learn that there were over 1.7 million students in the Florida K-8 system in 2009-10. When you do the math, it turns out that 3.9% of Florida K-8 students were retained during the 2009-2010 school year. What about the peak of Florida retention the year the 3rd grade retention policy took place in 2003-04? The total retention rate for that year was (waaaaait for it…..) 5.5%- a little more than half of the national rate that the NCES found in 2007.

We don’t have national data for K-3 retention, which is what we would need to do an ideal comparison, but the data we do have certainly establishes that there is a substantial amount of retention going on around the country, which will be having some impact on NAEP scores of states across the nation, not just Florida. Unless a state is doing far more than average, it retention is likely to be white noise overall- blips in the error term. Furthermore, it is not clear that Florida was doing more K-3 retention than the national average, even during the peak of the practice in 2003-04.

Mind you that I make no claim that retention is necessarily a good practice overall. I think there have been terrible retention practices, such as the practice of “redshirting” 9th graders in Texas back when the state gave a 10th grade exit exam. Redshirting was a widespread district level practice not mandated by state law and it was truly an awful policy basically designed to get students to drop out of school in 9th grade and thereby inflate the passing rate for the 10th grade exit exam.

There was nothing admirable about Texas redshirting. I would venture to guess that both a casual and a sophisticated analysis of data would have found it associated with higher drop out rates.

The Florida policy however is the opposite of the old Texas practice in that it is designed to set kids up to succeed rather than to fail. Not only have there been bad retention practices, there has also been a great deal of bad research done on retention that lacked the statistical rigor to establish causality. Do cancer drugs kill people, or is it the cancer? Most of the retention research doesn’t allow us to answer that sort of question.

Jay, Marcus Winters and the RAND Corp however have been applying sophisticated regression discontinuity designs to retention policies in Florida and New York City. They have found positive academic results. RAND found no self-esteem harm to students, and that NYC educators have generally positive views of the policy, to boot.

The question is not whether retention is “good” or “bad”- that all depends on how it is used. The evidence on the overall literacy effort in Florida-which includes retention as a centerpiece-is overwhelmingly positive.

10 Comments | social promotion | Tagged: Florida retention policy, grade retention, Jay Greene, Madhabi Chatterji, Marcus Winters, NEPC, New York City student retention policy, social promotion | Permalink
Posted by matthewladner

2011 NAEP: Florida Finally Hits a Wall

November 3, 2011

(Guest Post by Matthew Ladner)

The Florida Age of Public School Improvement hit a wall in the 2011 NAEP. This should not be terribly surprising, as Florida’s improvement seemed certain to plateau in the absence of additional reforms.

Governor Jeb Bush relentlessly pursued a dual strategy- transparency with teeth from the top down, parental choice from the bottom up. Together these reforms drove improvement in the public schools for a number of years. Accountability measures included school grading (A-F) and earned promotion in the early grades. Parental choice measures included Opportunity Scholarships for children attending F rated schools, the nation’s first special needs voucher program (McKay Scholarships), the nation’s largest scholarship tax credit program (Step Up for Students), a decent charter school law and the nation’s most robust system of digital learning. Florida lawmakers also attempted to thoughtfully incentivize success.

Governor Bush took office in 1999 and left office in 2007. It would be nice if these efforts could indefinitely push progress forward, but there have been plenty of bumps and problems along the way. In 2006, the Florida Supreme Court rendered a logic-free ruling abolishing Opportunity Scholarships (failing school vouchers) for private schools, and followed that up by ruling against a state authorizer for charter schools. Tax-credits, McKay and digital learning continued to incrementally advance, but not at an earth-shattering rate.

The larger problem may have come in the top down measures. The chart below presents the distribution of district and charter school grades, with one line being the A/B grades and the other D/F grades. The dotted lines represent instances when the state board raised school grading standards.

The setting of these standards represents far more of an art than a science. Set them far too high and disaster follows (this happened in Arizona). Set them too low, and you remove the tension in the system needed to drive improvement. Even after the last increase in grading standards, more than 10 times as many Florida schools received A/B grades as D/F grades.

Florida’s policymakers raised standards four times, and last year (wisely) put in an automatic trigger to raise standards by a preset amount when a certain ratio of schools get A or B grades. In addition, a fresh set of reforms passed the Florida legislature in 2011, revamping teaching and increasing charter school and digital learning options.

Just as it is impossible to exactly pinpoint how much of what caused the gains, it is likewise impossible to say exactly what made them stall. Note however that one of the favorite explanations of the anti-reform crowd, the pre-school, finally saw the advent of children old enough to have participated in the program and age into the 4th grade NAEP sample. I hope that someone is carefully studying variation in participation and corresponding trends in FCAT data, but the results at the aggregate level thus far seem underwhelming.

Plenty of other things, however, have been going on- including the collapse of a housing bubble, cutbacks in public school funding (including of some of the incentive funding programs) and a variety of other very bad things. My advice to Florida policymakers: roll up your sleeves and get back at it. Despite the enormous amount of progress seen on NAEP (and no one loves celebrating it more than me) too great of a gulf lies between a state system awarding ten times as many top grades as low grades but still suffering from large minorities of students scoring below basic on the NAEP exams.

Governor Bush has consistently said for years that success is never final, and reform is never finished. The 2011 pause in progress demonstrates that he called it correctly. Moving the needle on student learning on a meaningful scale and at a sustained basis represents one of the greatest public policy challenges of our times. Governor Bush has passed the torch to a new generation of Florida reformers, and they must now find new ways, and fine-tune the old ways, to push academic progress forward.

Edited for typos

4 Comments | personal tax credits/deductions, school accountability, social promotion, vouchers | Tagged: 2011 NAEP, Florida 2011 NAEP scores | Permalink
Posted by matthewladner

Social Promotion Fig Leaf

March 1, 2011

Matt Ladner and I have been testifying to state legislatures around the country about the effects of Florida’s policy to end social promotion in 3rd grade. The policy default-retains all 3rd graders who score below a certain threshold on the state’s reading test. There are several exemptions to being retained, but about 59% of low-achieving 3rd graders repeated the grade.

Research that Marcus Winters and I have published in the peer-reviewed journal, Education Finance and Policy, finds significant achievement benefits for students retained under the policy. After two years the retained students outperformed their promoted counterparts by about .46 standard deviations, which is the equivalent of receiving about 6.6 additional months of reading instruction. We compared students who barely performed above the test threshold on the 3rd grade test and were default-promoted to students who performed just below the test threshold. This regression-discontinuity design approximates a random assignment experiment.

When we testify about this research we are now commonly being asked about a “study” from the Miami-Dade School District that claims to find the effect fades after two years. Clearly the opponents of the policy (read: the unions) are arming folks with this to dispute our research findings. When people oppose a policy that is supported by rigorous research it is important that they at least have a fig leaf of research to support their opposition. The Miami-Dade report is that fig leaf. The report concludes:

This study has replicated the procedures of theGreene and Winters (2006) paper evaluat ingFlorida’s test-based promotion policy and hasderived very different judgements. Where theyconcluded that the retention policy led to significant improvements in reading for the retained students,this study finds no ultimate advantages. However,it would be a mistake to interpret this study as somekind of indictment of the Greene and Winters work. Their interpretation was valid for the way the datalooked after two years. The picture is quite different after four years

First, it is important to note that the “study” is actually a 4 page document produced by the internal research department of the Miami-Dade School District. It has no descriptive statistics, no detailed description of the methodology, and virtually no literature review. In short, it is extremely hard to judge the accuracy of a “study” that is little more than two graphs that have never been published, reviewed, or fully-described.

Second, the Miami-Dade internal report only claims to analyze data from the Miami-Dade School District, while our research is based on data from the entire state of Florida. It is perfectly possible that Miami-Dade poorly implemented the policy by doing things like granting the exemptions inappropriately or failing to offer effective reading interventions for students who were retained. Even if Miami-Dade did not have successful results with the program, the entire state did.

Third, it is inaccurate to say that the Miami-Dade “study” replicated our positive findings after two years but that those positive effects later disappeared. Their graphs suggest that there was no positive effect of being retained in Miami-Dade 1 and 2 years after the retention decision, and then they show a positive effect in years 3 and 4, which disappears in year 5. We found a small positive effect after one year that grew into a larger effect after two years.

Our results (even in the first two years) are completely different from those in the Miami-Dade report. It is hard to say whether this is because they only looked at Miami-Dade while we looked at the entire state, or because they did not actually replicate our methodology. Four pages and two graphs do not allow for a lot of nuanced analysis of the findings.

We are in the process of extending our analyses to include additional years, so we may have a better idea of whether the benefits we observed state-wide grow, shrink, or remain constant. In the meantime, the unions have provided their research fig leaf to cover state legislators who oppose the policy regardless of what research finds.

2 Comments | social promotion | Tagged: Miami-Dade, social promotion | Permalink
Posted by Jay P. Greene

Jeb Kicks Off the New Year Right

January 3, 2011

Jeb Bush has an op-ed in today’s Wall Street Journal that gets the new year off to the right start. Here’s a taste:

For the last decade, Florida has graded schools on a scale of A to F, based solely on standardized test scores. When we started, many complained that “labeling” a school with an F would demoralize students and do more harm than good. Instead, it energized parents and the community to demand change from the adults running the system. School leadership responded with innovation and a sense of urgency. The number of F schools has since plummeted while the number of A and B schools has quadrupled.

Another reform: Florida ended automatic, “social” promotion for third-grade students who couldn’t read. Again, the opposition to this hard-edged policy was fierce. Holding back illiterate students seemed to generate a far greater outcry than did the disturbing reality that more than 25% of students couldn’t read by the time they entered fourth grade. But today? According to Florida state reading tests, illiteracy in the third grade is down to 16%.

Rewards and consequences work. Florida schools that earn an A or improve by a letter grade are rewarded with cash—up to $100 per pupil annually. If a public school doesn’t measure up, families have an unprecedented array of other options: public school choice, charter schools, vouchers for pre-K students, virtual schools, tax-credit scholarships, and vouchers for students with disabilities.

Choice is the catalytic converter here, accelerating the benefits of other education reforms. Almost 300,000 students opt for one of these alternatives, and research from the Manhattan Institute, Cornell and Harvard shows that Florida’s public schools have improved in the face of competition provided by the many school-choice programs.

Florida’s experience busts the myth that poverty, language barriers, absent parents and broken homes explain failure in school. It is simply not true. Our experience also proves that leadership, courage and an unwavering commitment to reform—not demographics or demagoguery—will determine our destiny as a nation.

6 Comments | charter schools, competitive effects, school accountability, social promotion, special ed, vouchers | Tagged: Jeb Bush | Permalink
Posted by Jay P. Greene

The Irony of Social Promotion

January 9, 2009

In the current issue of the Economics of Education Review, Marcus Winters and I have an article about the use of exemptions to Florida’s test-based promotion policy. Under Florida’s policy students need to perform above a certain level on the 3rd grade reading test to automatically be promoted to 4th grade. If students score below that level they can still be promoted if they are granted one of various exemptions. Some of those exemptions are objectively measured, like scoring well on an alternative test or having certain special ed or English Language Learning classifications. But other exemptions are more subjectively determined, like having a portfolio of work worthy of being promoted.

Marcus and I looked at who received those exemptions and whether being exempted was beneficial. We found that African-American and Hispanic students were less likely to receive exemptions and get promoted, controlling for other factors. That is, minority students with the same test scores and economic status were less likely to be exempted from retention if they fell below the testing threshold. The test-based policy is not racially biased, since all students who lack the academic skills to pass the test may be retained. The bias is introduced in who gets exempted from that test-based policy.

And the irony of it all is that failing to receive an exemption actually benefited those minority students academically. That is, students who were denied the exemption and repeated third grade outperformed their promoted colleagues on achievement tests two years later. The retained students had more academic skill at the end of 4th grade than their comparable promoted peers at the end of 5th grade — despite being exposed to one less grade of curriculum.

Minority students denied the exemptions may have been the vicitms of discrimination, but they ended-up making greater academic progress as a result. Receiving those exemptions wasn’t doing many of the other students any favors.

The St. Pete Times has an article on the study today and had a blog post recently.

2 Comments | social promotion | Tagged: Jay Greene, Jay P. Greene, Marcus Winters, social promotion | Permalink
Posted by Jay P. Greene

The Year That Was

December 15, 2008

It’s getting to be that time when people make lists of good and bad things that happened during the preceding year. Here’s mine from an interview with Michael F. Shaughnessy of EducationNews.org:

What were the 5 most important developments during 2008 that contributed to reform of K-12 education?

1) Barack Obama strongly endorsed the idea that expanding choice and competition is an important part of improving public schools. He limited his support to expanding choice and competition through the introduction of more charter schools, but the theory is not fundamentally different than doing the same with vouchers. Whether Obama follows through on this campaign position or not, it is now clear that it is considered politically desirable among both Democrats and Republicans to support choice and competition. Holdouts from this view, including the teacher unions on the left and curriculum-focused reformers on the right, are being increasingly marginalized.

2) Sarah Palin, in her only major policy speech, pushed the idea of special education vouchers. Like Obama embracing choice and competition, Palin embracing special ed vouchers is a symbol of the political attractiveness of the policy. Special ed voucher programs already exist in Florida, Georgia, Ohio, Utah, and Arizona (pending the resolution of a court case). I’d expect the idea to spread to several more states in the next four years regardless of Sarah Palin’s political prospects.

3) Reform ideas, including choice, merit pay, curbing teacher tenure, and promoting alternative certification, are gaining mainstream acceptance in the Democratic Party largely thanks to the efforts of Democrats for Education Reform. An important indication of this political shift was an event at the Democratic National Convention organized by the Democrats for Education Reform at which an audience of about 500 cheered speakers denouncing the teacher unions and embracing reform ideas. The Democratic supporters of reform largely (but not exclusively) consist of urban minority leaders, including Michelle Rhee, Joel Klein, Adrian Fenty, Cory Booker, Kevin Chavous, Al Sharpton, and Marion Barry. Go ahead and make all the Sharpton and Barry jokes you like, but this (mostly) minority defection of urban Democrats from union orthodoxy is like a political earthquake that will have important implications for future reform politics. And it’s true that some conservatives have begun backtracking on reform ideas, including Sol Stern, Diane Ravitch, and depending on the day of the week, Checker Finn and Mike Petrilli. But if the reform movement has traded some conservatives for the new generation of minority Democratic leadership, I think we’ve come out ahead.

4) We saw a string of new or expanded school choice programs in 2008. Georgia adopted a universal tax-credit supported voucher program. Louisiana adopted a voucher program for New Orleans as well as a personal tax deduction for private school tuition. Florida expanded and decreased burdensome regulation on its tax-credit supported voucher program. And Utah increased and secured a source of funding for its special ed voucher program. For a movement declared dead more times than Generalissimo Francisco Franco, school choice continues to grow.

5) I’ll take the privilege of the final development to brag about the launch of the new doctoral program in education policy in the Department of Education Reform at the University of Arkansas. It may not have been among the 5 most important developments in the whole country, but it was a big development in my little world. With the first cohort of students starting in the Fall of 2009 (supported by a pool of generous fellowships) and a collection of outstanding faculty, we have the potential to significantly increase the number of reform-oriented researchers in academia, think-tanks, and foundations.

What were the 5 most important developments during 2008 that hindered reform of K-12 education?

1) The reform movement lost two great champions this year with the passing of John Brandl and J. Patrick Rooney. Brandl, who had been the Democratic leader of Minnesota’s state senate and Dean of the University of Minnesota’s Hubert Humphrey School of Public Policy, contributed significantly to the argument that choice was not only efficient, but also enhanced opportunities for the disadvantaged. He helped create the state’s pioneering charter school law and other choice programs. Brandl also served as mentor to many of today’s leading choice researchers. Rooney, who had always been active in the civil rights movement, personally sponsored scholarships for disadvantaged students to attend private schools. His privately financed program became a model for publicly funded voucher and tax-credit supported scholarship programs.

2) In 2008 we saw a number of “implementation” problems undermine otherwise promising reform initiatives. For example, Georgia adoption a social promotion policy that required students to pass a test or follow a formal exemption policy to be promoted in certain grades. My research with Marcus Winters on a similar policy in Florida suggested that it would improve student achievement. But in several districts around Georgia more than 90% of students were promoted without passing the test and without following the formal exemption procedure. They simply disregarded the law on a large scale with no consequences for any district or school employee. Another promising idea undermined by implementation was Reading First. There is a lot of rigorous science to support a phonics-based reading approach, but getting public schools to do it well is a completely different matter. Implementation also appears to have done-in a promising teacher mentoring program. I could go on, but the point is that there is no shortage of clever reform practices out there. The problem is that without addressing the lack of proper incentives in the public education system to improve, we regularly see these clever practices fall flat. We need incentive-based reforms along with reform of educational practices.

3) Earlier this year an Arizona court struck down voucher programs for students with disabilities and students in foster care on the grounds that the state constitution forbids aid to private schools. This month defenders of the program argued on appeal to the state Supreme Court that the program aids students, not schools. And the state already sends disabled students to private schools when it is determined that the public schools are unable to provide adequate services. That practice may also be in jeopardy, even though it is actually required by federal law (IDEA). Who knows how this will all be resolved, since courts can adopt any interpretation they like, reasonable or unreasonable. But court action has prevented these beneficial programs from operating and threatens to kill them.

4) A Florida court struck down the ability of a state commission to approve charter schools. If upheld by the (notorious) Florida Supreme Court, only school districts could approve charters and existing charters approved by the state commission may have to be closed. Giving districts the exclusive power to grant charters essentially allows the districts to decide with whom they will have to compete. It’s like giving McDonalds the exclusive power to approve the opening of all new restaurants. The state Supreme Court used the same narrow interpretation of clauses in the state constitution to strike down the Opportunity Scholarship voucher program, so the prospects for a vibrant and competitive charter sector in Florida are not good.

5) And finally the most disappointing development of 2008 is that we spent another half trillion dollars on public education without significantly altering the dysfunctional system that fails to teach a quarter of 8th grade students to read at a basic level or get them to graduate from high school. Results for minority students are significantly worse. The economic bailout may be a $700 billion enterprise, but the public school system spends almost that much each and every year. Every year that we spend that money without fundamentally altering how we operate public education is another fortune wasted and another year lost for millions of students.

(Note: corrected spelling of Marion Barry’s name)

3 Comments | politics, social promotion, vouchers | Tagged: Jay Greene, Jay P. Greene, Michael F. Shaughnessy | Permalink
Posted by Jay P. Greene

PJM on Candidates’ Education Flip-Flops

November 3, 2008

(Guest post by Greg Forster)

Over the weekend Pajamas Media carried my column on how Obama and Palin have flip-flopped on education:

Suppose I told you Candidate A has supported rigorous academic standards, has stood up to the teachers’ unions — even been booed by them at their convention — and proclaimed the free-market principles that schools should compete for students and better teachers should get higher salaries. On the other hand, Candidate B says that competition hurts schools, that kids should be taught a radical left-wing civics curriculum, that we should throw more money at teachers’ unions — excuse me, at schools — and that rigorous academic standards should be replaced with the unions’ old lower-the-bar favorite, “portfolio assessment.”

Candidate A is Barack Obama. So is Candidate B.

Meanwhile, Candidate C has made an alliance with the teachers’ unions, opposed school choice, thrown money at the unions — excuse me, at schools — and even helped undermine a badly needed reform of bloated union pensions. On the other hand, Candidate D has broken with the teachers’ unions, demanded that schools should have to compete for students, and endorsed the most radical federal education reform agenda ever proposed by a national candidate, including a national school choice program for all disabled students.

Candidate C is Sarah Palin. So is Candidate D.

Important disclaimer:

None of this implies anything about the overall merits of any of these candidates. One can love a candidate overall while hating his or her stand on education, and vice versa.

1 Comment | competitive effects, money myth, politics, school accountability, social promotion, special ed, teacher pay, teacher quality, teacher union, vouchers | Tagged: Barack Obama, Greg Forster, Pajamas Media, Sarah Palin | Permalink
Posted by Greg Forster

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Recent Posts

Archives

My Links

Blog Stats

Meta

Follow “Jay P. Greene's Blog”