Say It All Together Now Class: TESTING ≠ ACCOUNTABILITY

March 28, 2014

(Guest Post by Matthew Ladner)

So I tuned into the Fordham Foundation podcast yesterday, only to find that Fordham is stubbornly holding onto a misapprehension that their own research ought to have disabused them of long ago, namely that standardized testing equates to “accountability.”

This came up in a discussion of the Arizona ESA court ruling.  Broad misunderstandings of the program were on display, especially regarding the term “accountability.”

Sigh. Let’s start with the basics. The dictionary defines the word accountability as:

the quality or state of being accountable; especially :  an obligation or willingness to accept responsibility or to account for one’s actions.

Standardized testing is widespread in education, but “accountability” is scarce indeed.  So in my state more or less every public school student takes the AIMS test, but you would struggle to find anyone who is held “accountable” for the results.  Forty percent of 4th graders in Arizona scored below basic on the 2013 NAEP reading test, but good luck finding a policymaker, superintendent, teacher or parent who was held “accountable.”  Strangely enough, no one accepted responsibility for this sad state of affairs, making this a very unique form of accountability one where no one  is ever held responsible.

So what Mike and Michelle seem to actually be talking about is academic transparency to the public.  Arizona’s ESA program is indeed lacking in any form of academic transparency to the public.  There are a variety of forms this could take, some much more sensible than others, and Arizona policymakers would do well to pick one of them.  If they don’t pick one of the reasonable models, one must assume that an unreasonable model will be imposed sooner or later.  I’ve testified on a number of occasions at the Arizona legislature that policymakers should embrace transparency in the program. I will keep on doing it in the future.

Now let’s discuss the difference between faux and vrai accountability.

Kathy Visser, the mother of Jordan Visser, an ESA student featured in the above video, testified before the Arizona legislature regarding accountability recently.  She more or less noted that for all this shallow talk of “accountability” in this particular hearing (holding the same flawed understanding of the term displayed by Fordham) that there is in fact accountability in the ESA program. Everyone who educates Jordan is directly accountable to her.

Good luck getting that level of accountability in the public school system.

I followed up with Ms. Visser in a subsequent conversation. She experienced a number of difficulties in the public school system that are sadly common for special needs families. She had an open enrollment request denied without explanation with a public school official going so far as to hang up the phone on her. She consulted a specialized attorney who helps special needs families, but found the $15,000 retainer financially out of reach.  Fortunately the attorney told her about the ESA program.

Ms. Visser first tried a private school for Jordan.  She related that Jordan did not have a terrible experience in the private school, but that she decided to try the customized education approach with private tutors and therapists featured in the video above. Ms. Visser agreed that with a school voucher like the McKay Scholarship Program, she would have been able to hold the public schools accountable for the services they provided Jordan.  With an ESA, she can hold all providers accountable private schools, tutors, therapists, you name it.

That my friends is true accountability, you know, the kind where people actually get held responsible for their results.   Not the largely phony kind of accountability where states administer dummied down academic exams with massive item exposure, dropping cut scores, and all sorts of statistical games and tricks and other problems that I have read about in Fordham reports with most states obscuring things further behind fuzzy labels whose scale almost no one understands.

The type of “accountability” that Fordham is talking about however has proven to be baloney in most states for decades now.  Even in states with the most useful testing systems, like Massachusetts and Florida, you won’t find any parents wielding the type of authority exercised by Kathy Visser. It’s long past time for us to recognize the difference between genuine accountability and mere bullshit accountability.

UPDATE: In the interest of fairness please note that Mike did say he supports the ESA program in the podcast and expressed that we should let this experiment play out.  My point is not to claim that the ESA program is perfect (it isn’t) but rather that our notions of what constitutes “accountability” badly need a reboot.

Wonks of the World Unite!

January 22, 2014

Ah the undergrad years….ok, yes, that’s me with the lampshade hat. Hook ‘Em Horns!

(Guest Post by Matthew Ladner)

My email inbox has been filling up since Mike P. let loose with the “comrade” reference.  Bob Bowden interviewed me over differences in dogma between myself and Commissar Petrilli here, wherein I attempt to correct ересь in an effort to restore harmony within the Politburo. Glasnost and perestroika must continue comrades, but with a human face.

Can Mike Petrilli Get a “Hell Yeah”?

November 15, 2013

(Guest post by Greg Forster)

Mike is now auditioning to be the Stone Cold Steve Austin of education reform; since the Mr. T slot is already taken by George Clowes, I suppose that’s not a bad move. I’ll leave it to others to explain to Mike that there are other positions besides favoring no change and favoring centrally controlled change (backfill on that here and here, for starters). What I want to stress is that I am the sole author of “if you like your curriculum, you can keep your curriculum,” and Jason Bedrick can have it when he pries it from my cold, dead fingers.

Mike Petrilli is Rocking the Suburbs Just Like Quiet Riot Did

July 20, 2012

(Guest Post by Matthew Ladner)

Mike Petrilli is rocking the suburbs for school choice.

School Choice Researchers Unite in Ed Week

February 22, 2012

Pictured (L to R): Rick Hess, Jay Greene, Greg Forster, Mike Petrilli and Matt Ladner

(Guest post by Greg Forster)

Today, Education Week carries a joint editorial signed by nine scholars and analysists. We came together to agree that Mom and apple pie are good, Nazis and Commies are bad, and the empirical research supports the expansion of school choice:

Choice’s track record so far is promising and provides support for continuing expansion of school choice policies…Among voucher programs, random-assignment studies generally find modest improvements in reading or math scores, or both. Achievement gains are typically small in each year, but cumulative over time. Graduation rates have been studied less often, but the available evidence indicates a substantial positive impact. None of these studies has found a negative impact…Other research questions regarding voucher program participants have included student safety, parent satisfaction, racial integration, services for students with disabilities, and outcomes related to civic participation and values. Results from these studies are consistently positive…

In addition to effects on participating students, another major topic of research has been the impact of school choice on academic outcomes in the public school system…Among voucher programs, these studies consistently find that vouchers are associated with improved test scores in the affected public schools. The size of the effect in these studies varies from modest to large. No study has found a negative impact.

We have diverse viewpoints on many issues, but we share a common commitment to helping inform public decisions with such evidence as science is legitimately able to provide. We do not offer false certainty about a future none of us knows. But the early evidence is promising, and the grounds for concern have been shown to be largely baseless. The case for expanding our ongoing national experiment with school choice is strong.

This may well be the most important part:

The most important limitation on all of this evidence is that it only studies the programs we now have; it does not study the programs that we could have some day. Existing school choice programs are severely limited, providing educational options only to a targeted population of students, and those available options are highly constrained.

These limitations need to be taken seriously if policymakers wish to consider how these studies might inform their deliberations. The impact of current school choice programs does not exhaust the potential of school choice.

On the other hand, the goal of school choice should be not simply to move students from existing public schools into existing private schools, but to facilitate the emergence of new school entrants; i.e., entrepreneurs creating more effective solutions to educational challenges. This requires better-designed choice policies and the alignment of many other factors—such as human capital, private funding, and consumer-information sources—that extend beyond public policy. Public policy by itself will not fulfill the full potential of school choice.

Although I also feel particularly strongly about this:

Finally, we fear that political pressure is leading people on both sides of the issue to demand things from “science” that science is not, by its nature, able to provide. The temptation of technocracy—the idea that scientists can provide authoritative answers to public questions—is dangerous to democracy and science itself. Public debates should be based on norms, logic, and evidence drawn from beyond just the scientific sphere.


Kenneth Campbell is the president of the Black Alliance for Educational Options, in Washington.

Paul Diperna is the research director for the Friedman Foundation for Educational Choice, in Indianapolis.

Robert C. Enlow is the president and chief executive officer of the Friedman Foundation for Educational Choice.

Greg Forster is a senior fellow at the Friedman Foundation for Educational Choice.

Jay P. Greene is the department head and holder of the 21st-century endowed chair in education reform at the University of Arkansas, at Fayetteville, and a fellow in education policy at the George W. Bush Institute, in Dallas.

Frederick M. Hess is a resident scholar and the director of education policy studies at the American Enterprise Institute, in Washington, as well as a blogger for Education Week.

Matthew Ladner is a senior adviser for policy and research at the Foundation for Excellence in Education, in Tallahassee, Fla.

Michael J. Petrilli is the executive vice president of the Thomas B. Fordham Institute, in Washington.

Patrick J. Wolf is a professor and holder of the 21st-century endowed chair in school choice at the University of Arkansas, at Fayetteville.

Our color-coordinated mechanical lion battle chariots that join together into a giant robot are still under construction.

Defender of the empirical research universe!

NEPC’s attempt at strategic nihilism

August 4, 2011

(Guest Post by Matthew Ladner)

In the American film classic Animal House there is a scene where students smoke marijuana with their English professor, played by Donald Sutherland, and speculate that it could be the case that the molecules in your fingernails each contain a microscopic universe.

You can’t prove that there aren’t microscopic universes in your fingernails, after all, so they might be in there!

A nice post from Mike Petrilli on the Florida NAEP score gains prompted a response from Kevin Welner from NEPC that shows that the spirit of Sutherland’s Professor Dave Jennings is alive and well at the University of Colorado.

Again there is no attempt to address any of the gaping holes in retention theory. These holes include the fact that Florida’s 4th grade reading scores had improved substantially before the retention policy went into effect, and that they have continued to rise even as retention has fallen off substantially, and that they have fallen off substantially because of a very large improvement in 3rd grade scores.

Welner attempts to tiptoe around this by noting that our EdNext article addressing these points were addressed to a previous Walter Haney paper on the subject rather than the NEPC stuff, which is a distinction without much of a difference. The Chatterji paper contains a carbon copy of the Haney analysis. Amazingly, Chatterji dinged Burke and I for not doing a literature review (not the norm in our tribe) and then cites neither the Education Next paper nor Haney’s analysis. At best, she employed a double standard and at worst, she owes Professor Haney an apology.

Welner’s broader project is to attempt to use the causation problem as a shield. We don’t know, after all, exactly what caused Florida’s remarkable learning gains. Florida’s reformers had to implement their reforms in the real world rather than in a petri dish or in an Intention to Treat Random Assignment study. Welner believes that this allows him the opportunity for strategic nihilism:

The truth might be: (a) there are not actual improvements (the current study is too weak to say whether or not there are), (b) there are improvements, and they’re caused by a combination of all these things, (c) there are improvements, and they’re caused by something none of us pointed to (perhaps the green shirts??), or (d) there are improvements, and they’re caused by some of the things we’re pointing to BUT some of the other things we’re pointing to are actually harming students (just not enough harm to overcome the benefits of the other things).

In other words, when it comes to understanding the FL package of reforms, we are flying blind.

Welner is flying blind all right, but it is by choice. Let’s take each of these little gems on one at a time:

A. The NAEP results show very substantial improvements, as do other indicators.

B. I have always held that the exact cause for the improvement is impossible to know, because Florida’s reformers enacted multiple reforms simultaneously. The logical response to this is not to do none of the Florida reforms, but to do all of them.

C. Florida lurked near the bottom on NAEP for many years, enacted reforms in 1999, and then enjoyed sustained gains over time. While it could be the case that some mysterious X-factor caused the improvement, I’ve yet to hear a plausible theory regarding this. Dan Lips and I addressed multiple possibilities in the Education Next article, including demographic change, spending, etc, and found no evidence to support them.

D. This could be the case, but I haven’t seen a single scrap of evidence to suggest that it is actually the case- return to B above.

Welner is of course correct that there is a correlation and causation problem to consider. As a practical matter, there is nothing else to do but to carefully examine the evidence and history and draw the best conclusions that we can. Dan Lips and I did this in the Education Next article. Florida’s reforms coincided with the student population becoming poorer and less Anglo. State lawmakers increased funding per pupil, but it wasn’t by much and is still below the national average. NEPC complains about a lack of mention of the preschool voucher program when those kids have yet to age into the 4th grade NAEP sample. The class size amendment was implemented very slowly, long after Florida’s scores had begun to rise.

If Dr. Welner would like to provide a plausible explanation for why Florida’s NAEP scores increased so much after 1998, I’d be very interested to read it.

If he prefers to attempt to continue to play games, NEPC’s credibility will go on double secret probation.





Don’t Go Wobbly, Mike

June 9, 2011

(Guest Post by Matthew Ladner)

Over at the Ed Next blog, Mike Petrilli asks the question: if not the 100% proficiency requirement of NCLB, then what? Mike concludes:

So let’s get specific. Assuming that these 1 million kids remain poor over the next 12 years, what outcomes would indicate “success” for education reform? Right now the high school graduation rate in poor districts is generally about  50 percent. What if we moved that to 60 percent? Right now the reading proficiency rate for 12th graders with parents who dropped out of high school is 17 percent. What if we moved that to 25 percent? The same rate for math is 8 percent. What if we moved that to 15 percent?

To my eye, these are stretch goals–challenging but attainable. Yet to adopt them would mean to expect about 400,000 Kindergarteners not to graduate from  high school 12 years from now. And of the 600,000 that do graduate, we would  expect only 150,000 to reach proficiency in reading (25 percent) and just 90,000 of them to be proficient in math (15 percent).

90,000 out of 1 million doesn’t sound so good, but without improving our graduation or proficiency rates for these children, we’d only be taking about 40,000 kids. So these modest improvements would mean twice as many poor children making it–9 percent instead of 4 percent.

And what about the other 91 percent of our Kindergarteners? We don’t want to write them off, so what goals would be appropriate for them? Getting more of  them to the “basic” level on NAEP? Preparing them for decent-paying jobs instead of the lowest-paid jobs? Driving down the teen pregnancy rate? Lowering the incarceration rate?

Is this making you uncomfortable? Good. If we are to get beyond the “100 percent proficiency” or “all students college and career ready” rhetoric, these are the conversations we need to have. And if we’re not willing to do so, don’t complain when Diane Ravitch and her armies of angry teachers complain that we are asking them to perform miracles.

I agree that the 2014 cliff was utopian and counterproductive, and further that the safe-harbor provision does little to rescue NCLB as originally formulated. As Congress dithers on reauthorization (and when have we ever known Congress not to dither?) the 2014 event horizon approaches. Many states back loaded their proficiency requirements to the 2012-2014 period, and ooops, here it comes.

Our goal should be a system which encourages systemic improvement and academic growth rather than a system which requires “perfection on a deadline.” No one is better at creative insubordination than school administrators, making perfection on a deadline a dangerous proposition.

Rather than set goals, we need to focus on aligning the incentives of the adults in the system to match the interests of children and taxpayers. Let’s not bother with any Soviet style 5 year targets, and focus on incentivizing the behaviors we want, and disincentivizing the behaviors we don’t desire. Rather than bemoan a lack of parental involvement, let us promote policies that strongly encourage it. If we can do this, improvement will follow. Stretch goals should come in the form of raising cut scores over time and other forms of raising the bar. All the while, important incentive pieces like parental choice and financially incentivizing academic success must proceed.

Florida pursued this course, and coincidentally the ur-reactionary Ravitch is down there today. The St. Pete Times reports:

“Particularly in  Florida, it’s a disaster,” she said during a visit Wednesday with the St. Petersburg Times editorial board. “What we are doing is killing creativity, originality, divergent thinking. All the things we need in the 21st century are what we’re squeezing out of a generation of children.”

In a speech today at the Florida School Boards Association annual meeting in Tampa, Ravitch plans to continue her full-throated campaign to “save public education” from its obsession with testing.

“This is institutionalized fraud,” she said, referring to the phenomenon of ever-rising scores. “Because we are graduating just as many kids who can’t read as we did 10 years ago.”

She acknowledged that Florida’s focus on reading has produced real gains. But she said other test improvements may have come about partly from the state’s focus on reducing class

Ravitch and her “armies of angry teachers” are living in an alternative universe where she gets to make wild allegations about destroying the creativity of a generation of children without offering any evidence, make claims about education policy (in this case class size) which have been clearly refuted by empirical investigation and label the state which has produced more combined NAEP gains than any other for low-income children “a disaster.” Her point about 12th grade scores may be true in some states, but is not the case in Florida, where FCAT scores, AP passing rates and graduation rates are all improving. Why bother looking anything up if you can simply confidently assert nonsense?

Ravitch is noisily preaching to her reactionary choir as history blows past her, making her the George Wallace of the soft bigotry of low expectations-a sad but ultimately unimportant figure. Meanwhile, the serious conversation of K-12 carries on without her. Getting back to Mike’s post, I think the reactionary he should be worried about is not Diane Ravitch and her army of angry teachers but rather Charles Murray and his potential army of angry taxpayers.

The country after all spends about $10,000 per year per child-amounting to about $50,000 by the end of 4th grade-more if there was public pre-school provided. For that amount of money, which is largely the envy of the rest of the planet, it seems reasonable to teach the vast majority of children how to read. If it can’t be done because of “poverty” then why are we spending so much money going through the motions of pretending to try? Only educating an elite may offend our sensibilities, a Murrayite could argue, but only educating an elite while spending trillions of dollars on maintaining an illusion of educating the uneducable is far, far worse.

Far left meets far right at far gone, so to speak.

Americans are not quitters, and we are not going to give up on public education. Nor are we going to embrace some dorm-room bull session pipe dream of embracing state socialism to fix our education problems, which is just as well, because it wouldn’t work anyway. The grown ups in the K-12 reform conversation, both on the left and right, are pursuing greater productivity for the existing enormous investment in education.

I can forgive Mike for assuming we need some sort of gosplan, and that the gosplan needs to have an assumed rate of failure- he works in DC, and there is something in the water. Focusing on aligning the interests of adults with the interests of children while increasing parental involvement in a variety of ways will produce improvement. We’ve had enough utopian exercises (Goals 2000, NCLB 2014 with Common Core on hot standby). Our focus should be on thoughtful management of incentives in order to produce improvement. This is mostly going to involve sustained hand to hand combat in state capitals- a long hard slog.

Let’s get on with it-sometimes the hard way is the only way. Forget about a master plan or a schedule for improvement Mike- let’s get as much improvement as fast as we can get it.