The Proficiency Illusion

November 13, 2008

(Guest Post by Matthew Ladner)

I had a chance to see John Cronin from the Northwest Evaluation Association present on the Fordham Foundation’s study The Proficiency Illusion at the Arizona Education Research Organization conference last week. It was more than interesting enough to have me check out the study. From the forward by Checker and Mike:

Standards-based education reform is in deeper trouble than we knew, both the Washington-driven, No Child Left Behind version and the older versions that most states undertook for themselves in the years since A Nation at Risk (1983) and the Charlottesville education summit (1989). It’s in trouble for multiple reasons. Foremost among these: on the whole, states do a bad job of setting (and maintaining) the standards that
matter most—those that define student proficiency for purposes of NCLB and states’ own results-based accountability systems.

In short, the accountability and standards reform strategy has morphed into a pig’s breakfast. We’ve all known for some time that most states have failed to set globally competitive standards, and have monkeyed about with their cut scores. One of the revelations of the Proficiency Illusion (to me) is that many states have proficiency standards lacking internal consistency. For example, some states have incredibly low cut scores in the elementary grades, only to amp them way up in 8th grade. Parents will receive multiple notices saying that their child is “at grade level” only to shocked to learn later that they are well short.

Other types of problems exist as well. Two years ago at the same AERO conference, I saw a presentation showing that Arizona writing AIMS test had bell curves that stacked on top of each other rather than being horizontally linked across grades. In short, it was impossible to tell whether 4th graders were writing any better than 7th graders with the state exam.

Cronin’s presentation contained other insights- including just how arbitrary AYP can be. It depends hugely on the N requirement for subgroups state by state- some schools wind up with lots of subgroups and some don’t. This means that some relatively high performing schools miss AYP. In fact, Cronin demonstrated what I take to be a fairly common scenario where middle schools miss AYP but in which they perform at a higher level than all of the public school transfer options in the vicinity.

Checker and Mike go on to argue for national standards as a solution to these problems, but concede that it doesn’t seem likely. My modest suggestion on this front would be to adopt the A-Plus plan, and as states sought alternatives to AYP, to have the US Department require the creation of internally consistent standards as a starting point for negotiations. Given that the states would be able to determine their own set of sanctions (or lack thereof) I can’t see why an increase in rigor would be outside the realm of these discussions for states with absurdly easy to pass tests.

Deeply wedded to inconsistent standards? Fine- have fun with AYP and the 2014 train wreck.

In other words, if the feds would abandon Utopian nonsense like 2014 and the high quality teacher provision, they might be able to play a productive role in providing technical guidance and nudging states into better directions with their testing programs.

I am not a fan of NCLB, but even I will concede that it has to date had a net positive impact increasing transparency in public schooling. This will be lost, however, if the 2014 problem isn’t addressed, or if we go down the absurd road of portfolio assessments, and I do view transparency as vitally important.

The Proficiency Illusion shows us that much of the data we’ve been getting from state testing programs isn’t nearly as useful or reliable as imagined. This is a problem, and it must be addressed.