Grade Retention is Common Nationally but Effective in Florida

February 28, 2012

(Guest Post by Matthew Ladner)

I came across an interesting study from NCES recently concerning the practice of grade retention that creates yet another hole in NEPC boat regarding their Florida theories. In fact, here is a link to a study from the ASU precursor to NEPC by Columbia Teacher College Professor Chatterji (one of the NEPC critics) from 2003 calling on Florida to “rethink sanction and retention policies in light of new and past research showing that retention does not improve student achievement.” 

Now you can look at the below figure and ask yourself just who needs to reconsider what. The red line is FCAT 1 scores for Black students, the Green line is for Hispanic students, and the blue line is for all students.

The NEPC boat is already sitting on the floor of the ocean, but hey, why not drop a depth charge on it?

The main pet theory of the NEPC squad has been that Florida’s 4th grade NAEP scores have been profoundly warped by the state’s retention policy. This beats the daylights out of their Harry Potter theory, but there still is far less to it than meets the eye. Problems with this theory include a substantial improvement in 4th grade NAEP scores before the retention policy went into place, a substantial decline in retentions since the onset of the policy, and a substantial improvement in 3rd grade reading FCAT scores.  Oh and the advent of mid-year promotions and a few other things which NEPC has been either unable or unwilling to address. The peak of any aging effect would have come in 2005 and declined substantially, and yet Florida’s scores continued to rise.

An implicit assumption of this theory was that Florida is doing far more K-3 retention than other states around the country. After seeing this NCES study, I am no longer certain this is the case, especially now that Florida retention has fallen so substantially. Let’s dig into the data and find out.

State level data on grade level retention is very difficult to come by outside of Florida. However, NCES included a question about retention in their parent survey. Low and behold, 10% of parents in the NCES survey report that their child has been retained for one or more grade in grades K-8, more than 20% of low-income parents.

NCES: Students retained in one or more grade, K-8

So first off, this is quite a bit higher than I would have suspected and the trend has been rising. Given the hostility that many College of Education Professors have towards grade retention, it seems apparent that many of the teachers and administrators that go through their programs are not buying what they are selling on retention.

Now that we have a measure of retention nationally, we should explore the question of how prevalent the practice is in Florida. The Florida Department of Education provides this handy chart for the statewide numbers for retention for students in grades K-12. The technical term to describe this chart is “falling off a cliff.”

So if you rummage around in the spreadsheet provided by the Florida Department of Education on retention by grade level and add a few cells together, you can calculate that the total retention figure in Florida in 2009-2010 for Grades K-8 was 54,843.

That sounds like a lot, until you go over to the NCES Common Core Data (note to Jay, Greg and MWAB- not the academic standards, please call off the cruise missle strike :-) and learn that there were over 1.7 million students in the Florida K-8 system in 2009-10. When you do the math, it turns out that 3.9% of Florida K-8 students were retained during the 2009-2010 school year. What about the peak of Florida retention the year the 3rd grade retention policy took place in 2003-04? The total retention rate for that year was (waaaaait for it…..) 5.5%- a little more than half of the national rate that the NCES found in 2007.

We don’t have national data for K-3 retention, which is what we would need to do an ideal comparison, but the data we do have certainly establishes that there is a substantial amount of retention going on around the country, which will be having some impact on NAEP scores of states across the nation, not just Florida. Unless a state is doing far more than average, it retention is likely to be white noise overall- blips in the error term. Furthermore, it is not clear that Florida was doing more K-3 retention than the national average, even during the peak of the practice in 2003-04.

Mind you that I make no claim that retention is necessarily a good practice overall. I think there have been terrible retention practices, such as the practice of “redshirting” 9th graders in Texas back when the state gave a 10th grade exit exam. Redshirting was a widespread district level practice not mandated by state law and it was truly an awful policy basically designed to get students to drop out of school in 9th grade and thereby inflate the passing rate for the 10th grade exit exam.

There was nothing admirable about Texas redshirting. I would venture to guess that both a casual and a sophisticated analysis of data would have found it associated with higher drop out rates.

The Florida policy however is the opposite of the old Texas practice in that it is designed to set kids up to succeed rather than to fail. Not only have there been bad retention practices, there has also been a great deal of bad research done on retention that lacked the statistical rigor to establish causality. Do cancer drugs kill people, or is it the cancer? Most of the retention research doesn’t allow us to answer that sort of question.

Jay, Marcus Winters and the RAND Corp however have been applying sophisticated regression discontinuity designs to retention policies in Florida and New York City. They have found positive academic results. RAND found no self-esteem harm to students, and that NYC educators have generally positive views of the policy, to boot.

The question is not whether retention is “good” or “bad”- that all depends on how it is used. The evidence on the overall literacy effort in Florida-which includes retention as a centerpiece-is overwhelmingly positive.

Case Dismissed: Victory for School Choice in Florida

May 24, 2016


(Guest Post by Jason Bedrick)

The legal attacks on school choice programs are dropping like characters in a George R.R. Martin novel. Last week, a Nevada judge dismissed a case against the state’s education savings account program. Today, a Florida judge dismissed a case against several of the state’s school choice programs.

The Florida lawsuit originally concerned whether the state was adequately funding public education, but in 2014 the plaintiffs amended their suit to challenge a wide range of policies, including state accountability statutes, charter schools, tax-credit scholarships, and the McKay Scholarship Program for Students with Disabilities. Last year, a judge ruled that the plaintiffs did not have standing to challenge the private school choice programs, but this ruling also rejects their substantive claims that the choice programs somehow harm the traditional district school system:

[T]he Court finds no negative effect on the uniformity or efficiency of the State system of public schools due to these choice programs, and indeed, evidence was presented that these school-choice programs are reasonably likely to improve the quality and efficiency of the entire system. […]

Plaintiffs’ specific allegations regarding the constitutional implications of three of Florida’s choice programs- charter schools, the FTC Program, and the McKay Program- are similarly unsupported by the weight of the evidence. […]

The Court has already held that Plaintiffs lack standing to challenge the FTC Program, and the Court further concludes that the weight of the evidence does not support their speculative allegations that the FTC Program diverts state funding or has any material, detrimental effect on Florida’s system of public schools.

The weight of the evidence similarly does not support Plaintiffs’ allegations about the McKay Program, which is limited to “Students with Disabilities” and requires eligible students to have an individual educational or accommodation plan under federal law. […] As indicated by the Florida Supreme Court, parental decisions to send individual children with special needs to private school do not implicate the uniformity of the broader public school system- regardless of whether some of those parents accept scholarship funds from the State.



This decision constitutes one more legal stake in the heart of the canard that educational choice programs harm students attending traditional district schools. As Jon East pointed out at RedefinED yesterday, judges presiding over a separate anti-choice lawsuit in Florida cast a gimlet eye on similar claims during oral arguments last week:

[Florida Education Association] attorney Lynn Hearn: “The loss of $300 million at a minimum to the Florida public school system … is absolutely a fact.”

Presiding Judge Lori S. Rowe: “In your complaint, you haven’t actually alleged that there is a $300 million loss to the Florida education budget, have you?… In fact, the $300 million you’re referring to are the funds that flow into the scholarship program, correct?”

Attorney Hearn: “Well, that’s where the number arrives from, your honor. But we absolutely do allege that that amount has left the public schools in favor of the scholarship program. That’s because of the way the Florida schools are funded. They are funded on a per-student basis. So, during that year, 2013-14, there were 60,000 students who left the Florida public school system.”

Judge Ross L. Bilbrey: “But doesn’t that mean there are 60,000 fewer students that the state has to pay to educate?”

Attorney Hearn: “It does your honor. But the funding of students in our public schools is, uh, we’re not funding widgets, the funding formula for students is not a perfect correlation to the variable cost of funding that student.”

Judge Rowe: “But exactly what is the special injury you are articulating here? You haven’t alleged that any individual student is suffering. You haven’t alleged that per-student funding has been reduced. You haven’t even alleged that the education budget has been reduced.”

Essentially, the union wants to argue the district school system has some special claim on students–and therefore the public funds attached to those students–without openly making that claim. After all, the district school system can’t suffer a “loss” unless they somehow owned those funds to begin with, but parents have no such obligation to enroll their children at their zoned district school, or any district school for that matter. They feel entitled to those children and the corresponding funding, but they know they can’t make that claim explicitly because, well, it’s ludicrous. That’s why the union is having such a hard time articulating any special injury–and why they’re likely to lose that lawsuit as well.

For more information on today’s decision, see Travis Pillow’s write up at RedefinED.


UPDATE: Supplementing his opinion, the judge issued a 179-page Appendix for Findings of Fact which, among many other things, explains that Florida’s tax-credit scholarship program relies on private (not public) funding and explains that the plaintiffs failed to demonstrate any injury resulting from the program:

The Florida Tax Credit Scholarship Program (the ―FTC Program‖) allows Florida taxpayers to apply for tax credits ―to make private, voluntary contributions‖ to fund scholarships for children attending eligible K–12 private schools.767 Plaintiffs allege that the FTC Program violates the uniformity and efficiency requirements of Article IX, Section 1(a) by diverting public funds to private schools that are not subject to the same requirements as schools within the State‘s system of free public schools.

The Court has previously found that the FTC Program, which allows third parties to obtain tax credits for making private donations, does not involve public funds, legislative appropriations, or the State‘s ―provision‖ for a ―system of free public schools‖ under Article IX. Because the private donations that fund the FTC Program are not legislative appropriations, the Court has previously determined that Plaintiffs lack taxpayer standing to assert a challenge to this program under Florida law.

Plaintiffs have also failed to prove any special injury that would allow them to challenge the FTC Program. […]

[A]ny connection between the FTC Program and appropriations to support Florida‘s system of free public schools—not to mention the overall quality of that system—is purely speculative. There was no persuasive evidence presented that the FTC Program has any direct or indirect impact on public-school funding or on the uniformity, efficiency, safety, security, or quality of Florida‘s public schools. […]

Even if tax credits resulted in a decrease in the number of students attending the public schools, local school districts are not responsible for educating students who attend private schools.

The appendix is also is chock-full of citations of some dude named Jay Greene. Here’s a taste:

3rd Grade Retention Policy

Florida‘s third-grade retention policy also is supported by academic research. Dr. Jay Greene, a professor of education and head of the Department of Education Reform at the University of Arkansas, has extensively studied the effect of Florida‘s policy. Dr. Greene‘s studies, which are published in a peer-reviewed journal, concluded that Florida‘s test-based retention policy significantly improves the academic achievement of students who are retained.239 Plaintiffs did not present any evidence countering Dr. Greene‘s findings.

Resources & Results

Plaintiffs allege that the overall level of funding in Florida is not sufficient to provide a uniform, efficient, safe, secure and high quality system of public education.784 Plaintiffs assert that the performance outcomes for certain groups of students indicate that school funding is insufficient.

Plaintiffs, however, have not met their burden of proving a causal relationship between the level of resources available to schools in Florida and student outcomes. Indeed, as described below, the weight of the evidence presented on that issue establishes a lack of any causal relationship between additional financial resources and improved student outcomes. […]

In addition to Dr. Hanushek, Defendants presented findings of Dr. Jay Greene, a professor of education and head of the Department of Education Reform at the University of Arkansas. Dr. Greene statistically analyzed school district-level variables throughout the state of Florida, including per-pupil spending, teacher characteristics, and discipline rates, and found no relationship between these variables and student outcomes.

Specifically, Dr. Greene examined school district per-pupil expenditures and percentages of students proficient on the Florida Comprehensive Assessment Test (―FCAT‖)797 for grades 3 through 10 in reading and math; grades 5, 8, and 11 in science; as well as highschool graduation rates, for school years 2007–08 to 2012–13. The analysis revealed no connection between higher amounts of funding available in school districts and better student performance.

Dr. Greene also conducted regression analyses of spending and performance data, controlling for student demographic differences and prior levels of achievement across school districts. The demographic characteristics that were controlled included the proportion of minority students, proportion of students receiving free or reduced price lunch, the proportion of students classified as English language learners (―ELL‖), and the proportion of students with a disability who had an individual educational plan (―IEP‖), as well as academic outcomes in the prior year. The purpose of these analyses was to examine whether school districts would have better student outcomes if they had more resources, assuming school districts had the same demographic composition and prior year‘s academic outcomes. Dr. Greene‘s regression analyses revealed that there is no pattern between the level of spending in Florida school districts and student performance on the FCAT or high school graduation rates.

Teacher Experience

In addition, Dr. Greene evaluated the assertion by Plaintiffs that teacher qualifications and experience characteristics impact student performance, and that districts with high-minority and low-income student populations have a lower percentage of qualified, experienced teachers. Consistent with his other findings, Dr. Greene found no statistical relationship between the proportion of novice (first-year teachers) or ―highly qualified teachers, as defined by the Florida Department of Education, and student performance on the FCAT or high school graduation rates. Likewise, Dr. Greene found no statistical relationship between the percentage of minority and low-income students in a district and the proportion of novice or highly qualified teachers.

Suspension Rates

Dr. Greene also addressed Plaintiffs‘ assertion that high suspension rates are attributable to a lack of school district resources and lead to lower student performance outcomes. As above, Dr. Greene conducted regression analyses that controlled for student demographic characteristics and prior student outcomes. Dr. Greene found no relationship between the rate at which students are given out-of-school suspensions in Florida school districts and FCAT reading, math, or science proficiency, or graduation rates.

Court’s Conclusion re: the Evidence

The Court accepts Dr. Greene‘s conclusions and finds that they corroborate other evidence in the case showing the lack of causal relationship between the level of resources available in Florida schools and student outcomes, as well as evidence showing that the level of resources available is sufficient for a high quality system.

Although Plaintiffs bear the burden of proof in this case, neither Plaintiffs‘ expert witnesses nor their school-district witnesses presented analyses or studies rebutting the work of Drs. Hanushek and Greene. In fact, the weight of the evidence shows that despite budget cutbacks associated with the Great Recession, student performance continued to improve in the period 2007–08 to 2014–15.


The Race is On: Indiana is the new Florida, but who will be the next Indiana?

September 22, 2011

(Guest Post by Matthew Ladner)

The 2011 legislative sessions set a new standard for K-12 reform, can 2012 hope to compare? The logical response would be something along the lines of “not bloody likely.” The electoral calendar, the fact that many of the reform states are likely to be distracted by policy implementation, and the fact that the molasses states and likely to stay in their torpor all point to a diminished of expectations for next year.

Taking a step back from questions of the pace of reform, it makes for good bloggy fun to speculate where large breakthroughs might occur.

Looking regionally, Big 10 country clearly led the way last year. Indiana engaged in incredible soup to nuts reform, with big reform undertakings in Ohio, Wisconsin and even (gasp) Illinois with tenure reform. The Minnesota legislature passed transformative reforms, but settled for some incremental steps this year. Big things are under discussion in Michigan. Iowa is discussing reform, while Pennsylvania seems to be searching for their sea legs, which I expect them to find.

By comparison to the Big Ten, the South seems stuck in neutral, outside of Florida, Louisiana and Oklahoma. Texas and North Carolina used to be reform leaders, but they faded after plucking the low-hanging fruit of reform (standardized testing). North Carolina shows some signs of rousing. Tennessee has entered into a serious discussion about reform. Reform is on like Donkey Kong in Oklahoma- special needs vouchers followed by school grading and 3rd grade retention and a tuition tax credit program.

The Northeast features some interesting dynamics in Maine, and fascinating struggle between Democrats for Education Reform and the AFT in New York. Lots of small rural schools in the northeast will eventually benefit from digital learning.

When you look out West, you see a clueless giant surrounded by more nimble neighbors. All three states bordering California-Arizona, Nevada and Oregon -have taken steps to enact reform. Yes- even Oregon! Governors Sandoval of Nevada and Martinez of New Mexico have brought a new energy to reform discussions in their states. Arizona, Utah and New Mexico have adopted A-F school grading, with Utah also passing a far-reaching digital learning bill.

Florida enacted comprehensive reform in 1999. Indiana did it in 2011.  Which states will be next? I could tell you, but then I might have to kill you. Feel free to speculate in the comments section.

J.K. Rowling: The Jeb Bush of the NEPC Florida Fantasy

September 13, 2011

(Guest Post by Matthew Ladner)

The Arizona School Boards Association had their annual law conference last week, and had William Mathis from the Think Tank Review Project present on “Are Things as Sunny as They Seem in Florida?”

I went first, and presented charts like this, showing the vast improvement in Florida’s 3rd grade reading scores:

I have repeatedly asked the Think Tank Review Project people to explain why Florida’s 4th Grade NAEP scores continued to rise in 2007 and 2009 even as 3rd grade retention fell substantially. Or for that matter, why their 3rd grade scores have improved so strongly. Dr. Mathis made no attempt to address the issue.

I also presented charts like these:

Now, call me crazy, but when you are the state called “Arizona” in above chart, you might want to make a careful study of what the other state did to get their English Language Learners to read. This phenomenon  of course is not limited to ELL. Another chart I used showed the combined learning gains on all four NAEP tests for children with disabilities for the entire period we have data from all 50 states (2003-2009).

Just in case you are squinting that’s Florida in red with a gain of 69 points and Arizona in green with a decline of two points.

Dr. Mathis proceeded with his presentation unperturbed. He complained about the 3rd grade retention policy without any effort to explain why Florida’s 3rd grade scores had so profoundly improved, and why Florida’s 4th grade NAEP scores continue to increase even as retention rates have significantly declined.

To give Dr. Mathis’ presentation the fairest possible reading, I would say that he was trying to make the following points: that correlation is not causation, and that to use the terminology of Campbell and Stanley, I had not “controlled for history.” That is to say, there could be other possible explanations for Florida’s gains other than the reforms.

Now it is of course the case that correlation can lead us very much astray, and it is the case that “history” has a nasty habit of bedevilling our theories of causality. As I have noted in the past, however, the Florida reforms unfolded in the real world, rather than in a random assignment study. A great many things unfolded all at once. This is called “life” and there is nothing to be done about this but to gather as much data as possible to draw the best informed decisions we can.

Both Chatteriji and Mathis ignored the Education Next piece in which Dan Lips and I examined other possible explanations for Florida’s gains. Huge spending increases (nope), decline in the percentage of low-income or minority students (nope-increases in both), preschool voucher program (nope- students too young to have aged into the NAEP sample) and class size amendment (nope- implemented very slowly, gains already well under way, formal evaluations negative) and retention law (scores continued to rise even as retention fell). This sort of information might be unhelpful if you are simply trying to get the idea in that something other than a set of hated reforms drove the gains.

Mathis however posited other types of “history” and noted other ways that the world had changed after 1998. On his list of other parts of uncontrolled “history” with regards to Florida’s gains were Harry Potter books (kids reading more fiction) and the more widespread availability of personal computers at home.

Sadly, the format of the panel did not provide time for rebuttal. We had two other people on with us, and took questions from the audience. Had there been such time, however, I would have noted that while Arizona may seem backwards to outsiders (Dr. Mathis lives in Colorado) that we do in fact have Harry Potter books and even personal computers in our humble little patch of cactus. In fact, I am rather confident that Harry Potter books and personal computers became increasingly pervasive in all 50 states.

You never know, Harry Potter books could have powerful educational properties that only manifest themselves on massive peninsulas with high rates of humidity and large concentrations of alligators. The children of Arizona, landlocked in an arid climate, and with not much more in the large lizard department than the occasional Gila monster, may have been left behind. I can’t prove that this isn’t the cause after all.

Nevertheless I’m going to stick with my theory that Governor Bush’s success in implementing a varied and comprehensive set of K-12 reforms in 1999 served as the driver for the large increases in academic attainment seen in Florida’s NAEP scores since 1998. Dr. Mathis and his compatriots can continue to play their stategic nihlism game if they wish, ignoring the problems with their arguments and the studies most on point for the subject at hand (like the regression discontinuity studies of Florida’s retention policy).

Until they put forward a plausible explanation for Florida’s gains, I cannot for the life of me find any reason to take them seriously.

How to Improve Alabama’s Schools

August 8, 2016

(Guest Post by Dr. Williamson M. Evers)

Improving education is a combination of (1) teachers effectively conveying to students certain essential information; (2) getting a better match between schools (with different strengths and teaching talents) and students (with different capacities, needs, and interests); and (3) creating learning environments where teachers are motivated to engage the students’ intellects and emotions and students are motivated to learn challenging subject-matter — and learning in a way that embeds it and puts it into practice beyond the level of learning facts that could be looked up on the web. The classroom should be a kind of theater of high-morale learning. In that shared theater of learning, the important components of civilization are passed on, teachers are role models for students, and students’ character is formed.

Curriculum should promote patriotic and liberty-loving citizenship (without ruling out exposure in high school to our country’s problems.) We should always remember that the family is the site and source of the most important education that a child receives. The public school system should avoid undermining the family and related social institutions like churches, charities, and voluntary associations.

There are steps that Alabama should consider to improve student performance:

Reading improvement. The Alabama Reading Initiative (ARI) was a professional development program (started in 1998) with school-level reading coaches. It took a few years to plan and put into place. It was based on the scientific research literature on the teaching of reading and therefore took a phonics-first approach.

By 2007, Alabama’s grade 4 reading score on the National Assessment of Educational Progress (NAEP), a nationwide examination that tests a sample of students, had risen significantly – an eight-point jump from 2005 (almost a grade level). The grade 4 students made another four-point jump in 2011, catching up with the national average. In 2009, Alabama’s grade 8 reading score on NAEP rose three points, and in 2011, it rose another three points. The grade 8 gains were slower and less sharp than the gains for grade 4 and remained  five to six points below the national average. Even the less dramatic grade 8 gains did constitute considerable improvement.

The official history of the ARI says:

“From 2003 to 2011, with a state-funded reading coach in every elementary school, Alabama’s 4th graders made more progress in reading on the National Assessment of Educational Progress (NAEP) than students from any other state in the nation. Alabama met the national average in 4th grade reading for the first time, and Alabama is one of only four states in the nation (second only to Maryland) to show increases in 4th grade reading from 2009 to 2011. The number of Alabama students reading below grade level has been reduced by half.”

Since the coming of Common Core, the State Superintendent has directed the funds for the reading initiative and the STEM initiative to coaches for Common Core. Subsequently, grade 4 reading scores on NAEP in Alabama have declined eight points, to below the national average. Grade 8 reading scores remain stagnant.

I don’t like to made pronouncements about policy issues without talking to a range of people and looking into the details of what changes could be made. But my working hypothesis would be that the reading initiative needs to be restored.

STEM (science, technology, engineering, and mathematics) improvement. Alabama also had a long-running math, science, and technology initiative. In contrast with the reading initiative, it seems to have produced no noticeable effects on student achievement. I think it makes sense for Alabama to look into why there has been such a sharp difference in results between the two initiatives, and make appropriate adjustments.

In both STEM and reading, the Alabama State Department of Education should consider certifying Professional Development programs that are based on science – based on empirical research with control groups and the like. A State superintendent should seek to make sure that districts know about good Professional Development and should nurture the growth of such programs.

Graduation rate and academic attainment. To boost academic attainment, Alabama should put in place a 3rd grade test (Florida currently retains students who cannot pass its 3rd grade reading test) and a 10th-grade/graduation test. Alabama discarded its High School Exit Examination (AHSGE – Alabama High School Graduation Exam) in 2013 and replaced it with ACT/CCRS (College- & Career-Ready Standards).

The 11th grade ACT cut scores in Alabama are 18 for English; 22 for math and reading; and 23 for science. The ACT scores are the standard current national “college readiness” scores. So they are, in fact, quite demanding and above the 10th grade level.

ACT is basically used in Alabama for federal reporting purposes to comply with the federal Elementary & Secondary Education Act.  Graduation is currently based on area requirements and seat-time (measured in Carnegie units).

Alabama should restore its exit exam, but it should operate in a multiple second-chances way, with many opportunities to retake the tests and offering a variety of exemptions and special categories.  The idea is to create a focus and a shared goal, not to deny some large number of students a diploma.  Alabama should also reconsider its One Diploma policy.  Since there are a wide variety of students with different capacities, interests, and needs, there should probably be a variety of diplomas.

Tom Loveless of the Brookings Institution estimates that all the prep time and all the millions spent on the Common Core national curriculum-content standards boosted achievement in the U.S. by only a tiny amount: one scale-score point. The new Every Student Succeeds Act does much to protect against future federal interference with curriculum. When it comes to testing, Alabama is not now with PARCC or SmarterBalanced, but has instead affiliated with ACT. Yet Alabama has a set of state standards largely based on Common Core. It is time, I would think, to revisit those current state standards to see what needs fixing. (I participated in California’s line-by-line review of the Common Core in 2010, and I participated in creating the late-1990s California State Academic Content Standards, judged best in the nation by the Thomas B. Fordham Foundation and the American Federation of Teachers.)

Career and technical education. Students, because they are different individuals, should have different educational pathways in high school. Blouke Carus — a leading children’s magazine publisher (including Cricket, Ladybug, Cobblestone), math and reading textbook developer, and chairman emeritus of the Carus Corporation (a company producing manganese compounds) — has said: “Our schools need to offer each student a choice among six or more challenging and rigorous high school curricula, as do other, higher-performing countries.”

As I and my colleagues wrote in a 2013 manifesto entitled “Closing the Door on Innovation”: “There is no evidence to justify a single high school curriculum for all students. A single set of curriculum guidelines, models, or frameworks cannot be justified at the high school level, given the diversity of interests, talents, and pedagogical needs among adolescents. American schools should not be constrained in the diversity of the curricula they offer to students. Other countries offer adolescents a choice of curricula; Finland, for example, offers all students leaving grade 9 the option of attending a three-year general studies high school or a three-year vocational high school, with about 50% of each age cohort enrolling in each type of high school. We worry that the ‘comprehensive’ American high school may have outlived its usefulness, as a [2011] Harvard report [Pathways to Prosperity] implies.”

Because vocational education has a reputation for being a dumping ground and a place where minority children were channeled because of racial bias, education policy has neglected vocational education and sometimes even eliminated it. Alabama’s “Essentials/Life Skills Pathway” is a separate vocational-education pathway.  For example, instead of English 9, the student would take English Essentials 9. This is intended to prep students for community college or the workforce, not for entry to 4-year colleges.

Alabama needs to avoid a situation which the only high school available for a student is a college-prep comprehensive or, alternatively, vocational-only.  The state should incentivize local districts (through waivers) to create a variety of career-tech program alternatives (including a variety of mission-oriented schools), but these should include an academic component (as they do now). In general, Alabama should look carefully at Arizona where the process of school creation is highly decentralized, and the results have been fruitful.

Teacher Quality. As the 1999 manifesto “The Teachers We Need” (of which I was a co-signer) says, “the surest route to quality is to widen the entryway [and] deregulate the processes.” This would expand the pool of potential teachers. It would open the doors to highly-qualified, but non-traditional recruits who want to become teachers and thereby improve overall quality. New York State currently has a relatively open process allowing alternative certification.  In Arkansas, almost half the incoming teachers come in under alternative certification.

But Alabama gets only a C- from the National Council on Teacher Quality for the extent to which the state’s teaching pool has been expanded by alternative certification and the like. Alabama should also be open to non-traditional teacher training programs like the Relay Graduation School of Education in New York City.

Michael McShane of the American Enterprise Institute has written:

“Recently, I have been influenced by the work of Northwestern University economist Kirabo Jackson, whose fascinating NBER working paper calls into serious question policy’s recent overreliance on math and reading scores as the primary measure of the ‘goodness’ of schools and teachers. As it turns out, teachers have important and measurable impacts on both the cognitive and non-cognitive development of students. While it’s certainly true that test scores can tell us something important about a teacher, what is troubling for the test-score types is that it looks like (1) non-cognitive scores are better predictors of later life success (completing high school, taking the SAT, and going to college) and (2) that it is not the same set of teachers that is good at raising both cognitive and non-cognitive measures.

“Such has to be the same for schools, right? If there are teachers that are increasing non-cognitive, but not cognitive skills, surely there are schools that are doing the same. As a result, trying to assess if a school is ‘good’ or ‘bad’ relies on a complex web of preferences and objective measures that, quite frankly, cannot be taken into account in a centralized accountability system. We need something more sophisticated, and something that can respect a diverse conception of what ‘good’ and ‘bad’ means.”

In citing Michael McShane and Kirabo Jackson, I am suggesting a goal for the future, while not saying that there should be no accountability. We should consider a change in providers of accountability because Alabama has a variety of school districts, localities, and other groupings. As Jason Bedrick of the Cato Institute and Andrew Smarick, chair of the Maryland State Board of Education, have separately pointed out, accountability doesn’t have to be in conformity with One Best Way. School evaluation can be pluralistic and rivalrous and doesn’t have to be solely governmental. Churches could do it, neighborhood associations could do it, chambers of commerce could do it.   As Bedrick writes: “Parents can then evaluate the quality of education providers based on their own experience and the expert evaluations of appropriate external providers, and the entire system evolves as parents select the providers that best meet their children’s needs.”

Currently if a teacher is educated in Alabama teachers colleges and receives bad evaluations during the first three years, the teacher can go back to college for free. This seems like a smart program worth retaining.

Education Next magazine surveyed teachers nationwide and the teachers themselves said that about 5% of teachers deserve an “F” and 8% deserve a “D.” But we should be cautious about grandiose projects to improve teacher quality. The “highly qualified teacher” provisions of the No Child Left Behind law were never successfully put into effect. The elaborate new teacher-evaluation systems put in place (at the cost of millions) in about 20 states have reduced the percentage of teachers rated “satisfactory” from 99% to 97%.

Currently teacher-certification testing is under court supervision. Alabama has low passing scores on the PRAXIS II tests that it expects of teachers. When court supervision is over, the state could slowly raise passing scores (as Texas has). Teachers do need to know the subject-matter they are teaching. But the most important thing is to increase the inflow of good prospects into the teaching pool (from which districts and schools can select), rather than reducing the size of the pool.

Economic growth and student attainment. Both policymakers and the civically active public underestimate the economic gains from school improvement. The differences in rates of growth among states can be matched to the education of the workers in the various states. A multiplicity of causes affect economic growth, but nothing is more influential in the long run than school improvement. Although it is certainly difficult to improve schools, it is easier to improve schools than it is to change other factors that figure in increased productivity.

Equated to PISA international test results, Alabama is currently below Turkey.

Some people think that Alabama’s low results are because of demographics. But the only state whose whites do worse currently is West Virginia.

For Alabama, which is educationally at the bottom of the lowest quartile in the United States, the present value of bringing the state to the level of Kentucky – in the next quartile — would be three times Alabama’s current Gross Domestic Product (according to my Hoover Institution colleague Eric A. Hanushek). Other states at the Kentucky level are Florida, Arizona, Oklahoma, and Rhode Island. To get to the level of Kentucky would mean for Alabama over 6% annual growth in GDP and about 12% annual growth in salaries.

Conclusion: A State Superintendent should not be a Caesar, lording it over local districts and local superintendents. State superintendents should not act as if they are progressive technocrats with coercive utopian powers. It will be harmful to learning communities if state superintendents act that way.  Where sensible, they should devolve much of the responsibility over academic content, teaching methods, and instructional materials to the local districts. The different districts can – and should — try different things. Parallel learning communities will sometimes arise.  If districts and schools endeavor to conscientiously do their best, Alabama can climb out of the cellar and surpass states like Kentucky academically. The State of Alabama will prosper, and Alabama’s children will have more fulfilling lives.


Williamson M. Evers, Ph.D., is a Research Fellow at Stanford University’s Hoover Institution and a former U.S. Assistant Secretary of Education for Planning, Evaluation, and Policy Development. He is a finalist for the position of Alabama State Superintendent of Education.

Chingos Strikes Again

February 22, 2013

Yesterday, I blogged about a new study by Matt Chingos and Marty West about pension reform in Florida.  Now I see that Matt has struck again with a great study about on-line learning in the current issue of Education Next.  Matt, along with co-authors William Bowen, Kelly Lack and Thomas Nygren, conducted a random assignment evaluation of an online statistics course that was offered at six universities.

Students were assigned by lottery either to a traditional course or a course where the bulk of the instruction was provided by inter-active software supplemented by weekly discussion sections.  The bottom line is that students did no better or worse in measured learning outcomes regardless of whether they received the course in the traditional way or via the internet.  The authors suggest that these results should temper wild claims about improved learning from online instruction as well as wild accusations that online fails to deliver.  They seem to be equally effective.  But the authors add that online delivery has significant potential to reduce the cost of delivering education and may have significant benefits for retention of students.

Here’s their conclusion in their own words:

In the case of online learning, where millions of dollars are being invested by a wide variety of entities, we should perhaps expect that there will be inflated claims of spectacular successes. The findings in this study warn against too much hype. To the best of our knowledge, there is no compelling evidence that online learning systems available today—not even highly interactive systems, which are very few in number—can in fact deliver improved educational outcomes across the board, at scale, on campuses other than the one where the system was born, and on a sustainable basis….

We do not mean to suggest that ILO systems are a panacea for this country’s deep-seated education problems. Many claims about “online learning” (especially about simpler variants in their present state of development) are likely to be exaggerated. But it is important not to go to the other extreme and accept equally unfounded assertions that adoption of online systems invariably leads to inferior learning outcomes and puts students at risk. We are persuaded that well-designed interactive systems in higher education have the potential to achieve at least equivalent educational outcomes while opening up the possibility of freeing up significant resources that could be redeployed more productively.

They also consider the implication of this higher education study for online instruction in K-12:

Extrapolating the results of our study to K–12 education is hardly straightforward. College students are expected to have a degree of self-motivation and self-discipline that younger students may not yet have achieved. But the variation among students within any given age cohort is probably much greater than the differences from one age group to the next. At the very least, one could expect that online learning for students planning to enter the higher-education system would be an appropriate experience, especially if colleges and universities continue to expand their online offerings. It is not too soon to seek ways to test experimentally the potential of online learning in secondary schools as well.

You can read the full article here.

[Edited to correct omitted co-author and for clarity]

Shanker Institute Scholar Bounded in a Nutshell but Counts Himself a King of Infinite Space

January 15, 2013

(Guest Post by Matthew Ladner)

Matthew DiCarlo of the Shanker Institute has taken to reviewing the statistical evidence on the Florida K-12 reforms. DiCarlo reaches the conclusion that we ultimately can’t draw much in the way of conclusions regarding aggregate movement of scores.  He’s rather emphatic on the point:

In the meantime, regardless of one’s opinion on whether the “Florida formula” is a success and/or should be exported to other states, the assertion that the reforms are responsible for the state’s increases in NAEP scores and FCAT proficiency rates during the late 1990s and 2000s not only violates basic principles of policy analysis, but it is also, at best, implausible. The reforms’ estimated effects, if any, tend to be quite small, and most of them are, by design, targeted at subgroups (e.g., the “lowest-performing” students and schools). Thus, even large impacts are no guarantee to show up at the aggregate statewide level (see the papers and reviews in the first footnote for more discussion).

DiCarlo obviously has formal training in the statistical dark arts, and the vast majority of academics involved in policy analysis would probably agree with his point of view. What he lacks however is an appreciation of the limitations of social science.

Social scientists are quite rightly obsessed with issues of causality. Statistical training quickly reveals to the student that people are constantly making ad-hoc theories about some X resulting in some Y without much proof. Life abounds with half-baked models of reality and incomplete understandings of phenomena, which have a consistent and nasty habit of proving quite complex.

Social scientists have developed powerful statistical methods to attempt to establish causality techniques like random assignment and regression discontinuity can illuminate issues of causality. These types of studies can bring great value, but it is important to understand their limitations.

DiCarlo for instance reviews the literature on the impact of school choice in Florida. Random assignment school choice studies have consistently found modest but statistically significant test score gains for participating students. Some react to these studies with a bored “meh.” DiCarlo helps himself along in reaching this conclusion by citing some non-random assignment studies. More problematically, he fails to understand the limitations of even the best studies.

For example, even the very best random assignment school choice studies fall apart after a few short years. Students don’t live in social science laboratories but rather in the real world. Random lotteries can divide students into nearly identical groups with the main difference being that one group applied for but did not get to attend a charter or private school. They cannot however stop students in the control group from moving around.

Despite the best efforts of researchers, attrition immediately begins to degrade control groups in random assignment studies. Usually after three years, they are spent. Those seeking a definitive answer on the long-term impact of school choice on student test scores are in for disappointment. Social science has very real limits, and in this case, is only suggestive. Choice students tend to make small but cumulative gains year by year which tend to become statistically significant around year three, which is right around when the random assignment design falls apart. What’s the long-term impact? I’d like to know too, but it is beyond the power of social science to tell us, leading us to look for evidence from persistence rates.

So let’s get back to DiCarlo, who wrote “The reforms’ estimated effects, if any, tend to be quite small, and most of them are, by design, targeted at subgroups (e.g., the “lowest-performing” students and schools). Thus, even large impacts are no guarantee to show up at the aggregate statewide level.”  This is true but fails to recognize the poverty of the social science approach itself.

DiCarlo mentions that “even large impacts are no guarantee to show up at the aggregate statewide level.” This is a reference to the “ecological fallacy” which teaches us to employ extreme caution when travelling between the level of individual and aggregate level data. Read the above link if you want to know all the brutally geeky reasons why this is the case, take my word for it if you don’t.

DiCarlo is correct that connecting the individual level data (e.g. the studies he cites) to aggregate level gains is a dicey business. He however fails to appreciate the limitations of the studies he cites and the fact that the ecological fallacy problem cuts both ways. In other words, while generally positive, we simply don’t know the relationship between individual policies and aggregate gains.

We know for instance that we have a positive study on alternative certification and student learning gains. We do not and essentially cannot know however how many if any NAEP point gains resulted from this policy. The proper reaction for a practical person interested in larger student learning gains should be summarized as “who cares?” The evidence we have indicates that the students who had alternatively certified teacher made larger learning gains. Given the lack of any positive evidence associated with teacher certification, that’s going to be enough for most fair minded people.


The individual impact of particular policies on gains in Florida is not clear. What is crystal clear however is the fact that there were aggregate level gains in Florida. You don’t require a random assignment study or a regression equation, for instance when considering the percentage of FCAT 1 reading scores (aka illiterate) above. When you see the percentage of African American students scoring at the lowest of five achievement levels drop from 41% to 26% on a test with consistent standards, it is little wonder why policymakers around the country have emulated the policy, despite DiCarlo’s skepticism.

I could go on and bomb you with charts showing improving graduation rates, NAEP scores, Advance Placement passing rates, etc. but I’ll spare you. The point is that there are very clear signs of aggregate level improvement in Florida, and also a large number of studies at the individual level showing positive results from individual policies.

The individual level results do not “prove” that the reforms caused the aggregate level gains. DiCarlo’s problem is that they also certainly do not prove that they didn’t. It has therefore been necessary from the beginning to examine other possible explanations for the aggregate gains. The problem here for skeptics is that the evidence weighs very much against them: Florida’s K-12 population became both demographically and economically more challenging since the advent of reform, spending increases were the lowest in the country since the early 1990s (see Figure 4) and other policies favored by skeptics come into play long after the improvement in scores began.

The problem for Florida reform skeptics, in short, is that there simply isn’t any other plausible explanation for Florida’s gains outside of the reforms. They flailed around with an unsophisticated story about 3rd grade retention and NAEP, unable and unwilling to attempt to explain the 3rd grade improvement shown above among other problems. One of NEPC’s crew once theorized that Harry Potter books may have caused Florida’s academic gains at a public forum. DiCarlo has moved on to trying to split hairs with a literature review.

With large aggregate gains and plenty of positive research, the reasonable course is not to avoid doing any of the Florida reforms, but rather to do all of them. In the immortal words of Freud, sometimes a cigar really is just a cigar.

New Social Promotion Study

July 25, 2012

Marcus Winters and I have a new study on the effects of ending social promotion in Florida that appeared this month in the journal, Education Finance and Policy.  In our earlier published research we observed that retained students made greater academic gains in subsequent grades than did promoted students who were just like them.  But we could only track students for 2 years after the intervention, so we didn’t know if the benefits we observed compounded or faded over time.

In the new study we track students for as many as 5 years after retention.  The benefits of the policy do diminish, but they remain statistically significant and educationally substantial through middle school.  We hope to continue tracking these students through high school, graduation, and even college, but so far it looks like there are enduring benefits to ending social promotion.

Scenes from the Transformation: Reactionaries Crying in their Beer

May 10, 2012

(Guest Post by Matthew Ladner)

Carpe Diem is moving into Indianapolis with their blended learning model that produced the biggest learning gains in Arizona. Result: teacher unions babble about the school not having enough teachers and a Tucson reactionary attempts to peddle already discredited criticisms.

Over at NEPC, Kevin Welner rather assuredly asserts that retention is bad for students based upon methodologically unsophisticated studies caried out on bad policies. The claim that retention increases dropout rates is approximately as well established as the belief that cancer drugs kill people with cancer and that rooster crowing causes the sun to rise. Or that Harry Potter books caused NAEP gains in Florida for that matter. Par for the course, Welner ignores the statistically sophisticated studies nearing a random assignment study from Florida and NYC that show significant benefits from those policies.

Over at Ed Week, Little Ramona is drinking the Vegetarian Conspiracy Theory kool-aid on ALEC.

Bless their little reactionary hearts, but at least all of this makes for good comic relief.

The Arizona Republic: Arizona Prods Schools to Focus on Struggling Students

May 2, 2012

 (Guest Post by Matthew Ladner)

The Arizona Republic put in a great story on A-F school grading on the front page above the fold today. Notice the role of double weighting the learning gains of the students in the bottom 25% in the formula. This year Arizona lawmakers closed loopholes to the 3rd grade retention law and provided new state funding for reading intervention. The expansion of the ESA program has been revised and again sits on Governor Brewer’s desk. With a signature, Arizona will have a major new choice program focusing on special needs students and children attending D/F graded districts and schools- a combination of Florida’s McKay and Opportunity Scholarship programs with some new 21st Century upgrades.

Arizona’s relative poverty is masked by a very large number of wealthy retirees who live here, often only part of the year. If we tried to spend like CT with its blessings of old money and hedge fund billionaires, we’d drive those retirees elsewhere and likely have little to show for it in terms of academic achievement. We also have a relatively challenging student demographic profile, with more low-income and ELL challenges than average. If we want to make progress, we are going to need to play Moneyball and embrace policies that increase the bang for our buck in the system.

Arizona embraced the parental choice strategy beginning in 1994 with a liberal charter school law, and followed that up with the tuition tax credit program. In the aggregate, these programs combined mostly to take the edge off of public school enrollment growth. For every child attending charter schools or using a tax credit scholarship, three new kids were pouring into the districts and the state continued to spend hundreds of millions of dollars annually on new district schools- even in very poor performing districts.

The charter school law succeeded in providing a number of very high quality schools and the tax credit programs helped Arizona’s private schools survive the creation of 500+ charters. The parental choice programs were vitally important for tens of thousands of families, but they alone were not at a scale to move the needle much on public school improvement. This was especially the case as policymakers botched other major areas of K-12 policy.

Arizona’s K-12 testing system, for instance, is in recovery from having devolved into a cruel joke on kids. Arizona had the nation’s biggest dummy down vis-a-vis the NAEP on our state AIMS test. The state fielded a deeply flawed version of the Terra Nova exam that curiously found Arizona students to be above the national average in every grade and subject every year, when we as a state had never exceeded a national average on any NAEP exam. Our policymakers put a stop to it. Worse still, we literally still have schools with giant banners out front in 9000 point font boasting of being a “PERFORMING SCHOOL” when in reality “Performing” was the second lowest label possible. The legislature passed a law this session to forbid the use of these deceptive labels going forward.

Things have changed substantially during the downturn. A housing bust was tailor-made to humble Arizona’s economy, and enrollment growth has slowed to a trickle. One silver lining in this very dark cloud is that some of the most successful Arizona charter school networks have executed/are executing ambitious plans to open new schools- real estate is cheaper, they have earned access to capital, and they are making their moves. This is very much for the good. State policymakers have made far-reaching reforms on not only transparency and parental choice, but in teacher evaluation and the curtailment of social promotion.

We’ve still got miles to go and challenges ahead, but Arizona is on the way to improved learning.



%d bloggers like this: