2011 NAEP: Florida Finally Hits a Wall

November 3, 2011

(Guest Post by Matthew Ladner)

The Florida Age of Public School Improvement hit a wall in the 2011 NAEP. This should not be terribly surprising, as Florida’s improvement seemed certain to plateau in the absence of additional reforms.

Governor Jeb Bush relentlessly pursued a dual strategy- transparency with teeth from the top down, parental choice from the bottom up. Together these reforms drove improvement in the public schools for a number of years.  Accountability measures included school grading (A-F) and earned promotion in the early grades. Parental choice measures included Opportunity Scholarships for children attending F rated schools, the nation’s first special needs voucher program (McKay Scholarships), the nation’s largest scholarship tax credit program (Step Up for Students), a decent charter school law and the nation’s most robust system of digital learning. Florida lawmakers also attempted to thoughtfully incentivize success.

Governor Bush took office in 1999 and left office in 2007. It would be nice if these efforts could indefinitely push progress forward, but there have been plenty of bumps and problems along the way. In 2006, the Florida Supreme Court rendered a logic-free ruling abolishing Opportunity Scholarships (failing school vouchers) for private schools, and followed that up by ruling against a state authorizer for charter schools. Tax-credits, McKay and digital learning continued to incrementally advance, but not at an earth-shattering rate.

The larger problem may have come in the top down measures. The chart below presents the distribution of district and charter school grades, with one line being the A/B grades and the other D/F grades. The dotted lines represent instances when the state board raised school grading standards.

The setting of these standards represents far more of an art than a science. Set them far too high and disaster follows (this happened in Arizona). Set them too low, and you remove the tension in the system needed to drive improvement. Even after the last increase in grading standards, more than 10 times as many Florida schools received A/B grades as D/F grades.

Florida’s policymakers raised standards four times, and last year (wisely) put in an automatic trigger to raise standards by a preset amount when a certain ratio of schools get A or B grades. In addition, a fresh set of reforms passed the Florida legislature in 2011, revamping teaching and increasing charter school and digital learning options.

Just as it is impossible to exactly pinpoint how much of what caused the gains, it is likewise impossible to say exactly what made them stall. Note however that one of the favorite explanations of the anti-reform crowd, the pre-school, finally saw the advent of children old enough to have participated in the program and age into the 4th grade NAEP sample. I hope that someone is carefully studying variation in participation and corresponding trends in FCAT data, but the results at the aggregate level thus far seem underwhelming.

Plenty of other things, however, have been going on- including the collapse of a housing bubble, cutbacks in public school funding (including of some of the incentive funding programs) and a variety of other very bad things. My advice to Florida policymakers: roll up your sleeves and get back at it. Despite the enormous amount of progress seen on NAEP (and no one loves celebrating it more than me) too great of a gulf lies between a state system awarding ten times as many top grades as low grades but still  suffering from large minorities of students scoring below basic on the NAEP exams.

Governor Bush has consistently said for years that success is never final, and reform is never finished. The 2011 pause in progress demonstrates that he called it correctly.  Moving the needle on student learning on a meaningful scale and at a sustained basis represents one of the greatest public policy challenges of our times. Governor Bush has passed the torch to a new generation of Florida reformers, and they must now find new ways, and fine-tune the old ways, to push academic progress forward.

Edited for typos


Oklahoma Legislature Adopts Earned Promotion Policy

April 14, 2011

 

(Guest Post by Matthew Ladner)

Oklahoma lawmakers adopted a policy to curtail the social promotion of children failing to acquire basic literacy skills by the end of the 3rd grade. Oklahoma lawmakers adopted a special needs choice program last year, and are considering a scholarship tax credit program and other far-reaching reforms this year. Oklahoma has got major K-12 reform mojo!

The Foundation for Excellence in Education will be releasing a policy brief on retention soon. Congrats to Oklahoma’s K-12 reformers in joining Florida, New York City, Arizona and Indiana in adopting this tough-love policy. Now comes the hard part: policy implementation. The Devil is in the details on this reform, and others have botched it in the past. Yes, I’m looking at you Georgia…

I spent a few days in Oklahoma recently, and their reformers struck me as resolved, fearless and capable. That’s good, because that is what will be needed to see this policy through.


Social Promotion Fig Leaf

March 1, 2011

Matt Ladner and I have been testifying to state legislatures around the country about the effects of Florida’s policy to end social promotion in 3rd grade.  The policy default-retains all 3rd graders who score below a certain threshold on the state’s reading test.  There are several exemptions to being retained, but about 59% of low-achieving 3rd graders repeated the grade.

Research that Marcus Winters and I have published in the peer-reviewed journal, Education Finance and Policy, finds significant achievement benefits for students retained under the policy.  After two years the retained students outperformed their promoted counterparts by about .46 standard deviations, which is the equivalent of receiving about 6.6 additional months of reading instruction.  We compared students who barely performed above the test threshold on the 3rd grade test and were default-promoted to students who performed just below the test threshold.  This regression-discontinuity design approximates a random assignment experiment.

When we testify about this research we are now commonly being asked about a “study” from the Miami-Dade School District that claims to find the effect fades after two years.  Clearly the opponents of the policy (read: the unions) are arming folks with this to dispute our research findings.  When people oppose a policy that is supported by rigorous research it is important that they at least have a fig leaf of research to support their opposition.  The Miami-Dade report is that fig leaf.  The report concludes:

This study has replicated the procedures of theGreene and Winters  (2006)  paper  evaluat ingFlorida’s test-based promotion policy and hasderived very different judgements. Where theyconcluded that the retention policy led to significant improvements in reading for the retained students,this study finds no ultimate advantages. However,it would be a mistake to interpret this study as somekind of indictment of the Greene and Winters work. Their interpretation was valid for the way the datalooked after two years. The picture is quite different after four years

First, it is important to note that the “study” is actually a 4 page document produced by the internal research department of the Miami-Dade School District.  It has no descriptive statistics, no detailed description of the methodology, and virtually no literature review.  In short, it is extremely hard to judge the accuracy of a “study” that is little more than two graphs that have never been published, reviewed, or fully-described.

Second, the Miami-Dade internal report only claims to analyze data from the Miami-Dade School District, while our research is based on data from the entire state of Florida.  It is perfectly possible that Miami-Dade poorly implemented the policy by doing things like granting the exemptions inappropriately or failing to offer effective reading interventions for students who were retained.  Even if Miami-Dade did not have successful results with the program, the entire state did.

Third, it is inaccurate to say that the Miami-Dade “study” replicated our positive findings after two years but that those positive effects later disappeared.  Their graphs suggest that there was no positive effect of being retained in Miami-Dade 1 and 2 years after the retention decision, and then they show a positive effect in years 3 and 4, which disappears in year 5.  We found a small positive effect after one year that grew into a larger effect after two years.

Our results (even in the first two years) are completely different from those in the Miami-Dade report.  It is hard to say whether this is because they only looked at Miami-Dade while we looked at the entire state, or because they did not actually replicate our methodology.  Four pages and two graphs do not allow for a lot of nuanced analysis of the findings.

We are in the process of extending our analyses to include additional years, so we may have a better idea of whether the benefits we observed state-wide grow, shrink, or remain constant.  In the meantime, the unions have provided their research fig leaf to cover state legislators who oppose the policy regardless of what research finds.


Jeb Kicks Off the New Year Right

January 3, 2011

Jeb Bush has an op-ed in today’s Wall Street Journal that gets the new year off to the right start.  Here’s a taste:

For the last decade, Florida has graded schools on a scale of A to F, based solely on standardized test scores. When we started, many complained that “labeling” a school with an F would demoralize students and do more harm than good. Instead, it energized parents and the community to demand change from the adults running the system. School leadership responded with innovation and a sense of urgency. The number of F schools has since plummeted while the number of A and B schools has quadrupled.

Another reform: Florida ended automatic, “social” promotion for third-grade students who couldn’t read. Again, the opposition to this hard-edged policy was fierce. Holding back illiterate students seemed to generate a far greater outcry than did the disturbing reality that more than 25% of students couldn’t read by the time they entered fourth grade. But today? According to Florida state reading tests, illiteracy in the third grade is down to 16%.

Rewards and consequences work. Florida schools that earn an A or improve by a letter grade are rewarded with cash—up to $100 per pupil annually. If a public school doesn’t measure up, families have an unprecedented array of other options: public school choice, charter schools, vouchers for pre-K students, virtual schools, tax-credit scholarships, and vouchers for students with disabilities.

Choice is the catalytic converter here, accelerating the benefits of other education reforms. Almost 300,000 students opt for one of these alternatives, and research from the Manhattan Institute, Cornell and Harvard shows that Florida’s public schools have improved in the face of competition provided by the many school-choice programs.

Florida’s experience busts the myth that poverty, language barriers, absent parents and broken homes explain failure in school. It is simply not true. Our experience also proves that leadership, courage and an unwavering commitment to reform—not demographics or demagoguery—will determine our destiny as a nation.


Indiana Might Be the Next Florida

February 21, 2010

Matt has written numerous times on the remarkable progress that has been made in Florida, see for example here.  Forces are gathering in Indiana that suggests they may be next to try to full court press of Florida reforms.  The governor, the state superintendent, the Indianapolis newspaper,  and a bipartisan coalition of state legislators on the education committee seem poised to pursue some significant reforms.

First up on their agenda is passage of a bill to end the social promotion of 3rd graders who are unable to read at a basic level.  Patricia Levesque and I each have op-eds in the Indy Star on this topic , with a favorable introduction from the editor.

Check it out.


The Irony of Social Promotion

January 9, 2009

In the current issue of the Economics of Education Review, Marcus Winters and I have an article about the use of exemptions to Florida’s test-based promotion policy.  Under Florida’s policy students need to perform above a certain level on the 3rd grade reading test to automatically be promoted to 4th grade.  If  students score below that level they can still be promoted if they are granted one of various exemptions.  Some of those exemptions are objectively measured, like scoring well on an alternative test or having certain special ed or English Language Learning classifications.  But other exemptions are more subjectively determined, like having a portfolio of work worthy of being promoted.

Marcus and I looked at who received those exemptions and whether being exempted was beneficial.  We found that African-American and Hispanic students were less likely to receive exemptions and get promoted, controlling for other factors.  That is, minority students with the same test scores and economic status were less likely to be exempted from retention if they fell below the testing threshold.  The test-based policy is not racially biased, since all students who lack the academic skills to pass the test may be retained.  The bias is introduced in who gets exempted from that test-based policy.

And the irony of it all is that failing to receive an exemption actually benefited those minority students academically.  That is, students who were denied the exemption and repeated third grade outperformed their promoted colleagues on achievement tests two years later.  The retained students had more academic skill at the end of 4th grade than their comparable promoted peers at the end of 5th grade — despite being exposed to one less grade of curriculum. 

Minority students denied the exemptions may have been the vicitms of discrimination, but they ended-up making greater academic progress as a result.  Receiving those exemptions wasn’t doing many of the other students any favors.

The St. Pete Times has an article on the study today and had a blog post recently.


The Year That Was

December 15, 2008

It’s getting to be that time when people make lists of good and bad things that happened during the preceding year.  Here’s mine from an interview with Michael F. Shaughnessy of EducationNews.org:

What were the 5 most important developments during 2008 that contributed to reform of K-12 education?

 1)     Barack Obama strongly endorsed the idea that expanding choice and competition is an important part of improving public schools.  He limited his support to expanding choice and competition through the introduction of more charter schools, but the theory is not fundamentally different than doing the same with vouchers.  Whether Obama follows through on this campaign position or not, it is now clear that it is considered politically desirable among both Democrats and Republicans to support choice and competition.  Holdouts from this view, including the teacher unions on the left and curriculum-focused reformers on the right, are being increasingly marginalized.

2)     Sarah Palin, in her only major policy speech, pushed the idea of special education vouchers.  Like Obama embracing choice and competition, Palin embracing special ed vouchers is a symbol of the political attractiveness of the policy.  Special ed voucher programs already exist in Florida, Georgia, Ohio, Utah, and Arizona (pending the resolution of a court case).  I’d expect the idea to spread to several more states in the next four years regardless of Sarah Palin’s political prospects.

3)     Reform ideas, including choice, merit pay, curbing teacher tenure, and promoting alternative certification, are gaining mainstream acceptance in the Democratic Party largely thanks to the efforts of Democrats for Education Reform.  An important indication of this political shift was an event at the Democratic National Convention organized by the Democrats for Education Reform at which an audience of about 500 cheered speakers denouncing the teacher unions and embracing reform ideas. The Democratic supporters of reform largely (but not exclusively) consist of urban minority leaders, including Michelle Rhee, Joel Klein, Adrian Fenty, Cory Booker, Kevin Chavous, Al Sharpton, and Marion Barry.  Go ahead and make all the Sharpton and Barry jokes you like, but this (mostly) minority defection of urban Democrats from union orthodoxy is like a political earthquake that will have important implications for future reform politics.  And it’s true that some conservatives have begun backtracking on reform ideas, including Sol Stern, Diane Ravitch, and depending on the day of the week, Checker Finn and Mike Petrilli.  But if the reform movement has traded some conservatives for the new generation of minority Democratic leadership, I think we’ve come out ahead.

4)     We saw a string of new or expanded school choice programs in 2008.  Georgia adopted a universal tax-credit supported voucher program.  Louisiana adopted a voucher program for New Orleans as well as a personal tax deduction for private school tuition.  Florida expanded and decreased burdensome regulation on its tax-credit supported voucher program.  And Utah increased and secured a source of funding for its special ed voucher program.  For a movement declared dead more times than Generalissimo Francisco Franco, school choice continues to grow.

5)     I’ll take the privilege of the final development to brag about the launch of the new doctoral program in education policy in the Department of Education Reform at the University of Arkansas.  It may not have been among the 5 most important developments in the whole country, but it was a big development in my little world.  With the first cohort of students starting in the Fall of 2009 (supported by a pool of generous fellowships) and a collection of outstanding faculty, we have the potential to significantly increase the number of reform-oriented researchers in academia, think-tanks, and foundations.

What were the 5 most important developments during 2008 that hindered reform of K-12 education?

1)     The reform movement lost two great champions this year with the passing of John Brandl and J. Patrick Rooney.  Brandl, who had been the Democratic leader of Minnesota’s state senate and Dean of the University of Minnesota’s Hubert Humphrey School of Public Policy, contributed significantly to the argument that choice was not only efficient, but also enhanced opportunities for the disadvantaged.  He helped create the state’s pioneering charter school law and other choice programs.  Brandl also served as mentor to many of today’s leading choice researchers.  Rooney, who had always been active in the civil rights movement, personally sponsored scholarships for disadvantaged students to attend private schools.  His privately financed program became a model for publicly funded voucher and tax-credit supported scholarship programs.

2)     In 2008 we saw a number of “implementation” problems undermine otherwise promising reform initiatives.  For example, Georgia adoption a social promotion policy that required students to pass a test or follow a formal exemption policy to be promoted in certain grades.  My research with Marcus Winters on a similar policy in Florida suggested that it would improve student achievement.   But in several districts around Georgia more than 90% of students were promoted without passing the test and without following the formal exemption procedure.  They simply disregarded the law on a large scale with no consequences for any district or school employee.  Another promising idea undermined by implementation was Reading First. There is a lot of rigorous science to support a phonics-based reading approach, but getting public schools to do it well is a completely different matter.  Implementation also appears to have done-in a promising teacher mentoring program.  I could go on, but the point is that there is no shortage of clever reform practices out there.  The problem is that without addressing the lack of proper incentives in the public education system to improve, we regularly see these clever practices fall flat. We need incentive-based reforms along with reform of educational practices.

3)     Earlier this year an Arizona court struck down voucher programs for students with disabilities and students in foster care on the grounds that the state constitution forbids aid to private schools.  This month defenders of the program argued on appeal to the state Supreme Court that the program aids students, not schools.  And the state already sends disabled students to private schools when it is determined that the public schools are unable to provide adequate services.  That practice may also be in jeopardy, even though it is actually required by federal law (IDEA).  Who knows how this will all be resolved, since courts can adopt any interpretation they like, reasonable or unreasonable.  But court action has prevented these beneficial programs from operating and threatens to kill them.

4)     A Florida court struck down the ability of a state commission to approve charter schools.  If upheld by the (notorious) Florida Supreme Court, only school districts could approve charters and existing charters approved by the state commission may have to be closed.  Giving districts the exclusive power to grant charters essentially allows the districts to decide with whom they will have to compete.  It’s like giving McDonalds the exclusive power to approve the opening of all new restaurants.  The state Supreme Court used the same narrow interpretation of clauses in the state constitution to strike down the Opportunity Scholarship voucher program, so the prospects for a vibrant and competitive charter sector in Florida are not good.

5)     And finally the most disappointing development of 2008 is that we spent another half trillion dollars on public education without significantly altering the dysfunctional system that fails to teach a quarter of 8th grade students to read at a basic level or get them to graduate from high school.  Results for minority students are significantly worse.  The economic bailout may be a $700 billion enterprise, but the public school system spends almost that much each and every year.  Every year that we spend that money without fundamentally altering how we operate public education is another fortune wasted and another year lost for millions of students.

(Note: corrected spelling of Marion Barry’s name)


PJM on Candidates’ Education Flip-Flops

November 3, 2008

(Guest post by Greg Forster)

Over the weekend Pajamas Media carried my column on how Obama and Palin have flip-flopped on education:

Suppose I told you Candidate A has supported rigorous academic standards, has stood up to the teachers’ unions — even been booed by them at their convention — and proclaimed the free-market principles that schools should compete for students and better teachers should get higher salaries. On the other hand, Candidate B says that competition hurts schools, that kids should be taught a radical left-wing civics curriculum, that we should throw more money at teachers’ unions — excuse me, at schools — and that rigorous academic standards should be replaced with the unions’ old lower-the-bar favorite, “portfolio assessment.”

Candidate A is Barack Obama. So is Candidate B.

Meanwhile, Candidate C has made an alliance with the teachers’ unions, opposed school choice, thrown money at the unions — excuse me, at schools — and even helped undermine a badly needed reform of bloated union pensions. On the other hand, Candidate D has broken with the teachers’ unions, demanded that schools should have to compete for students, and endorsed the most radical federal education reform agenda ever proposed by a national candidate, including a national school choice program for all disabled students.

Candidate C is Sarah Palin. So is Candidate D.

Important disclaimer:

None of this implies anything about the overall merits of any of these candidates. One can love a candidate overall while hating his or her stand on education, and vice versa.


AJC Op-Ed

July 13, 2008

I have an op-ed in today’s Atlanta Journal Constitution on the social promotion issue in Georgia.


Eduwonkette and Eduwonk Aren’t Edumarried?

July 8, 2008

The New York Sun had a nice profile yesterday of Eduwonkette.  Well, it’s not exactly a profile because Eduwonkette writes anonymously.  In the article some folks complain that her anonymity is a problem: “A co-director of the Education Sector think tank, Andrew Rotherham, suggested on his blog Eduwonk that Eduwonkette might be unfairly pretending to be unbiased because she has ‘skin in the game… It’s this issue of you got all this information to readers, without a vital piece of information for them to put it in context.’”

I think Andy’s mistaken on this. (Did they have some kind of edu-break-up?)  The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.  I wish we all spent a whole lot less time analyzing people’s motives and a whole lot more time on their evidence and arguments. 

The only major problem with anonymity is lack of responsibility for being wrong.  There is a reputational price for making bad arguments or getting the evidence wrong that Eduwonkette avoids paying professionally — although she does pay a reputational price to the name brand of Eduwonkette.

Speaking of being wrong, Eduwonkette knocks the study Marcus Winters, Julie Trivitt, and I released today through the Manhattan Institute.  She complains: “It may be an elegantly executed study, or it may be a terrible study. The trouble is that based on the embargoed version released to the press, on which many a news article will appear today, it’s impossible to tell. There is a technical appendix, but that wasn’t provided up front to the press with the glossy embargoed study. Though the embargo has been lifted now and the report is publicly available, the technical appendix is not.”

This isn’t correct.  Embargoed copies of the study were provided to reporters upon their request.  If they requested the technical report, they could get that.  Both were available well in advance to reporters so that they could take time to read it and circulate it to other experts before writing a story.  Both the study and the technical report were made publicly available today (although there seems to be a glitch with the link to the technical report that should be fixed within hours).  The technical report can be found here.

And while we are on the subject of Eduwonkette being wrong, her attacks on test-based promotion policies are overdone.  The Jacob and Lefgren paper does raise concerns, but there is more positive evidence from the experience in Florida.  As I wrote in a previous post: “In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.”

Our results may actually be consistent with what Jacob and Lefgren find.  We find academic benefits for students retained in third grade.  They find: “that grade retention leads to a modest increase in the probability of dropping out for older students, but has no significant effect on younger students.”  It could be that test-based promotion is more beneficial when done with younger students.  It could also be that the policy has positive effects on achievement with some cost to graduation. 

And particularly severe problems with the integrity of test results used for promotion decisions in Chicago may limit the ability to generalize from Chicago’s experience.  In Chicago it may have been easier to move retained students forward by cheating on the next test than actually teaching them the basic skills they need to succeed in the next grade.

Besides, I’m sure that Edwuonkette wouldn’t put too much stock in Jacob and Lefgren’s non-peer-reviewed paper released straight to the public.  I’m sure she would be consistent in her view that: “By the time the study’s main findings already have been widely disseminated, some sucker with expertise in regression discontinuity may find a mistake while combing through that appendix, one that could alter the results of the study. But the news cycle will have moved on by then. Good luck interesting a reporter in that story… So as much as I like to kvetch about peer review and the pain and suffering it inflicts, it makes educational research better. It catches many problems and errors before studies go prime time, even if it doesn’t always work perfectly.”  

Or do these standards only apply to studies whose findings she doesn’t like?   If Eduwonkette isn’t careful she might get a reputation.


Follow

Get every new post delivered to your Inbox.

Join 839 other followers