I Only Know One Truth-It is Time for Bossy McBossypants Testing to End

September 5, 2017

(Guest Post by Matthew Ladner)

Last spring I was on my bike and came across this in front of a local middle school. I found it striking enough to take a picture with my phone:

In case you are squinting at your iPhone, the sign says “AZ Merit Testing 4/17-5/3.” Now mind you that the test that these students take to determine whether or not they go to college, and if so what sort of college, takes 4 hours. Comprehensive exams for a Ph.D. took me three days. Somehow in the awesome logic of 2017 it came to pass that it would make some sort of “sense” to disrupt schools for two weeks to give…AZMerit.

I think we all know what tends to happen to the school year starting (in this case) 5/4.

Weeks later I had the opportunity to observe a number of focus groups held on parental choice policy. The groups were from different parts of the country, and included parents, teachers and opinion leaders. Despite the fact that the topic of the convening was never testing, everyone made their feelings on the subject clear during conversations. All groups everywhere deeply dislike the current practice of standardized testing.

I can’t emphasize the next point strongly enough: I never once heard anyone use the phrase “Common Core” or burst into a fit of conspiracy mongering. Rather what I saw repeatedly was that people feel that schooling has become overly fixated on test preparation. People have a rather strongly held belief that schooling is supposed to be more than test prep. Something has gone terribly wrong with education in their view, and they want it to stop. Across the groups I saw, the consensus seemed to be that we should drive a stake through the heart of the current system, fill the mouth with holy wafers, and then burn the sarcophagus to fine ash.

I have seen remarkably little evidence that today’s heavy-handed, standards based testing system is of much utility. There is some suggestive evidence that states that had been doing nothing on the testing front before NCLB got a modest bump in results when they started testing. They may however have received a similar bump from a system with a much lighter footprint. Moreover no less than Hanuskek and Loveless have concluded that the heavy-handed Common Core project resulted in approximately nothing in the way of improved student learning. Given that we live in a democracy, a lighter footprint system seems like a fine idea.

So here is mine:

Preserving campus level academic transparency should be the central goal of testing. The Demos would apparently be happy to sacrifice it in return for slaying the testing vampire, but it would be a terrible loss in my view. States can adopt whatever standards they want (I suggest the old Massachusetts standards) but give their students a three-hour national norm reference exam on the second to last day of school. The last day rather than last month of school can now be the write-off. Do a good job teaching the MA standards, your students will do well/show progress on the nnr test.

Some will want to have their state officials grade or otherwise label schools based on the results. Have at it-but it is worth noting that the defacto accountability system in this country has become the Greatschools rating system given that is where the eyeball traffic resides. State ratings have become little more than an obsession internal to the system. Some will want to continue on the troubled path of trying to move the number of teachers fired for low performance from 1% to 1.5%. My view is that this is an unworkable path to hold schools accountable, but if some state or locality wants to keep it up feel free.

I know some of you continue to feel motivated by the idea that standards are going to lead us to profound improvement and narrower achievement gaps.  Decades into the project it is time to ask- where’s the beef? If you are willing to impose a deeply unpopular system of testing upon American families I must ask why? The burden of proof lies with you. If you (like me) would like to preserve campus level transparency I ask what is your plan? My plan is to adopt a system that is less intrusive and prescriptive and hold for dear life to campus level data-now tell me your plan. If your plan is to hold onto dear life to a system that the public abhors, I want to suggest that you need a new plan.

In my view, voting with your feet represents the most robust form of accountability by a very wide margin. I would like to have those voting decisions informed by test scores, and a great many other things including parent reviews (score another touchdown for Greatschools). Watching the focus group discussions made me realize that the United States House’s decision to enact a deeply misguided federal opt-out was not a fluke, but rather fit with the democratic sentiments of their constituents.

Opt-outs lead to nudge outs which leads to completely unreliable and thus worthless data. They will be passing at the state level soon unless transparency supporters pull their heads out of the sand. As Corwallis wrote to Clinton before the Battle of Yorktown “What is our plan? If we don’t have one, what are we doing here?”

Perhaps I’ve got this all wrong. If so, the comment section awaits.

The Mystery of the Math Swoon

October 30, 2015

FL Charter 2015 NAEP 8m

(Guest Post by Matthew Ladner)

So nationally 8th grade math scores declined by three points. That’s not good but on a 500 scale point test it isn’t clear that it is anything to get too excited about, although it interrupted a long-term positive trend. Florida’s 8th grade math scores however declined by six points. That’s more worrisome.

So digging around in the data reveals that Florida charter schools were unaffected by the swoon- their 8th grade math scores were flat between 2013 and 2015- and delightfully high to boot (see chart).

The Trial Urban District Assessment has information for Miami-Dade and Hillsborough County (Tampa area). Here’s where the mystery deepens- Miami was also unaffected with flat scores between 2013 and 2015. Hillsborough however:

8 point drop in the Tampa area according to TUDA. If there is any rhyme or reason to this I can’t discern it. Some of the national meta-explanations I have seen bandied about don’t seem to work to explain trends in Florida. For instance some have pointed the finger at Obama’s state waivers. There may or may not be something to that nationally, but Florida schools all operated under the same waiver. Standards/testing transition issues likewise impacted all schools-is there some reason why Miami and charter schools should brush this off while the state as a whole did not? Something peculiar may have happened in Hillsborough but Tampa is not big enough to do a huge amount of damage to Florida’s statewide average.

I’m stumped, but always happy to employ the wisdom of the crowd. If you have bright ideas or wild speculation to offer, that’s what the Jayblog comment section was made for!


Scenes from the Seattle Waiver Riots

July 16, 2015

(Guest Post by Matthew Ladner)

A reporter caught this gripping moment from the Seattle Washington riots following the state’s decision to drop their NCLB waiv…what? Seattle police don’t carry shields with Greek words written on them? Few in Seattle took any notice of the waiver business?

Sorry my bad.

It Depends on What the Meaning of “Testing” Is

July 9, 2015

Bill wagging finger

(Guest post by Greg Forster)

Lots of really good back and forth about NCLB testing and the federal opt-out over the past few days, in response to Matt’s posts. I just want to step in and point out something that seems to be getting lost in the discussion.

Testing of all students (other than those that get an opt-out) is not the only kind of NCLB-related testing. NCLB also required all states, for the first time, to participate in the Nation’s Report Card. NRC participation created the “academic transparency” Matt is looking for, but without raising any concerns about opt-outs, because it’s given to a representative sample of students rather than to all students. If you want to measure how states are doing at serving subgroups of students, this can be done by testing representative samples of those subgroups via the NRC.

My position is that the feds should not throw huge piles of money at schools, but if they’re going to do so (and it seems nothing can stop them) they can and should require the kind of “transparency” NRC provides without pushing states to test every child – and also without interfering with states’ ability to test every child in public schools if they wish to do so. Testing a representative sample of students provides “transparency” without forcing any particular child to take the test.

Unfortunately, the Common Core people have destroyed the bipartisan consensus for “transparency” of even the NRC kind, because now all testing has become suspect. Well done!

K-12 Reformers Need to stare in the Mirror after 251-178

July 9, 2015

(Guest Post by Matthew Ladner)

Perhaps it can be delayed until after the reauth drama has reached a conclusion, and by no means to I wish to exonerate members of the House, but there is a very real need for K-12 reformers to look at themselves in the mirror and ask “how did we lose a bipartisan super-majority that favored academic transparency in the United States House?”

It’s been clear to me that the transparency consensus has been collapsing since at least 2013.  Even now, many in Washington seemed not to have noticed, and continue to carry supporting poorly considered technocratic tweaks to NCLB as if the consensus still exists.

251-178 folks-it’s gone.  How it was lost is certainly complex and I am not in a position to render definitive judgement. I’m calling it though that the canary just died in the coal mine.

To me there has never been a stronger case made for reducing the federal footprint in K-12 than yesterday’s vote. Yesterday the United State House of Representatives voted in overwhelming fashion to mandate the gathering and collection of student testing data that would be of absolutely no value whatsoever- would compromise the ability of parents to compare schools in a reliable fashion, would never withstand challenge in a legal proceeding like Vegara etc. Both past experience and simple logic would lead one to the conclusion that creating a federal parental opt-out for state testing systems will create a powerful incentive for school officials to nudge low-performing students (read: Black, brown, children with disabilities) out of standardized testing to improve their scores. They managed to do this in an attempt to reauthorize an important piece of legislation 8 years behind schedule. I don’t know about you but if this is the best we can get out of our alleged federal Olympians color me more ready than ever to take my chances with state legislatures.

But I digress. If reformers want there to be a consensus on transparency, we apparently need to rethink our efforts. What we are doing now obviously is not working.

The Kind of Control You Are Attempting…

May 11, 2015

(Guest Post by Matthew Ladner)

Is there much at stake in the fight over academic standards? Studies from no less than Hanushek and Loveless basically show that the standards movement has largely been pushing on a string. There is some evidence that suggests that states that were doing absolutely nothing on testing before NCLB saw above average math gains, but the fact is most states were testing before NCLB, the gains may have been a one time step increase, and evidence linking the quality of standards and/or tests to academic gains is in short supply.

NCLB’s attempt to test the nation’s kids to 100% proficiency (or as Andy Rotherham insists something more like close to it if you read the fine print, which few outside of Andy did) by a date certain ended in tears waivers.

My impression is that the standards movement basically hangs its hat on the Massachusetts experience. Massachusetts has the highest NAEP scores and thus is a good example to study. Massachusetts however introduced a multifaceted reform strategy in the early 1990s, but scholars seem remarkably incurious about which policy changes helped to drive how much improvement. Of course, like the Florida experience, we can never know what policy changes drove aggregate level improvements, but we have a great deal of micro-level evidence on the impact of individual policies. If any of this exists for Massachusetts, I’ve not seen it discussed. Even if we did have a good sense of this based upon a large body of studies, the question of external validity must be considered. Last time I checked MA was one of four states with an average family income for a family of four in the six figures and I’d wager draws an unusually high number of teachers from selective universities.

Why has the standards movement been pushing on a string? No it is not just that states set the test cut scores at incredibly low levels, although they did that:

It’s not just that states held a repulsive 35% of schools responsible for the scores of their special education kids scores in 2009-10, although they did just that:

After all of those things and others most states took the further step of obscuring the results behind a set of fuzzy labels, like Texas:

Some states have pulled this off much better than others, and a high quality system of transparency should be every policymakers goal. The idea that the country has meaningful, widespread “accountability” through state testing is a demonstrably simplistic notion. The greatest trick the devil ever pulled was conflating minimal skills testing in math and reading with robust accountability. While this is obviously absurd given a moment or two of reflection, it is also deeply ingrained in people’s thinking that you can do things like show a legislative committee a chart like the one immediately above, only to have a member of that committee berate you a mere few minutes later that private schools “lack accountability.”

Er, lack accountability compared to what? I may have missed it but I’m putting the number of people in Texas having been held responsible for the state’s 28% reading proficiency rate over/under at zero unless you want to blame it on the kids themselves, most of whom have been labeled “proficient” on state tests that the Wall Street Stock Picking Chicken might pass on a good day (see Figure 1).

Well yes, but the Common Core will fix all of this. Except of course it won’t. If you’ve been paying attention, you may have noticed that states all over the place have been adopting their own tests and cut scores and discussing withdrawing all together.  Meet the new boss, same as the old boss?

The current chaos shares an origin with the wrecking of the NCLB-era state tests. It is the same reason your tax dollars get used to pay farmers not to grow food so that you can pay higher grocery bills.  Agribusiness is organized and politically active, while eaters are disorganized and politically inactive.  Organized/active beats disorganized/inactive 99 times out of a 100.

So in theory, the state sets out grade level academic standards, and then tests children against those standards. Schools thereby follow a coherent flow of content such that you do simple addition before complex addition etc. In theory teachers and schools that fail to teach the standards get held accountable. In theory, there is no unauthorized breeding on Jurassic Park, but…

As long as you are going to have academic standards and tests, you ought to fight not to have horribly deceptive systems. You should rather fight for informative tests and clear labels, but with the full knowledge that the dinosaurs on your island will constantly be breaking out of your fences in any number of ways. They may even convince some people in the leafy suburbs that the substitution of one set of standards and tests for another constitutes oppression, er, somehow…how? I’m not entirely sure but…ah…stick it to THE MAN!

Bureaucratic accountability, in short, will always face severe political limitations, and even under the best of circumstances is no substitute for parents possessing an exit option. Even under the best theoretical systems there will always be kids who would be better off somewhere else for both academic and non-academic reasons. Decentralized accountability works best with transparency to inform choices, but centralized accountability without choice will inevitably face the gravity well of regulatory capture.

The level of control you are attempting is not possible.



The Anti-Testing Zombie Apocalypse

December 1, 2014

Grrrrrr….testing ruin flavor of BRAINSSSZZZSSSS!!!!

(Guest Post by Matthew Ladner)

While some of the strongest supporters of standardized testing have allowed their minds to wander to counter-productive uses of overstretched waiver authority in the already dying days of a lame-duck administration, rumors have reached my ears of growing support for eliminating annual testing as a requirement under NCLB in Congress.

This may seem implausible to some, but after watching a 30+ year bipartisan consensus on transparency fold like a house of cards in Texas, nothing seems impossible. Discuss among yourselves…

The hour is later than you think…

The Rebels in the Hills Throw the Capital into Disarray

August 17, 2011

(Guest Post by Matthew Ladner)

Libya? Well yes at the moment but also NCLB as the Department has decided to allow states to retroactively “reset” their proficiency goals.

Over at Eduwonk, Andy grouses that if you have your attorneys study the fine print, it is actually 92 percent proficiency, and not 2014. He may be right, but the state departments of education either don’t agree or don’t realize it. The AMO charts I have seen all end with 100 percent proficiency in 2014.

McNeil and Klein write:

By letting a state retroactively revise its proficiency targets so that schools do better under the law, the department is setting a precedent that it’s willing to use any loophole or technicality to, depending on your perspective, help states out or avoid making tough decisions against states. This, too, despite vows in June that the Education Department would “enforce” the law.

After a similar faceoff with Idaho chief Tom Luna, the department also let that state keep its proficiency targets level, too, because Idaho hadn’t taken advantage of the three-years-in-a-row allowance.

Department officials say they want to give states breathing room until the details of the package come out next month. But one question I have is: If states can just go back and redo their proficiency targets so schools keep making AYP, why apply for a waiver, especially if you have to adopt reforms prescribed by the Obama administration?

Why indeed? State officials seem likely to draw the conclusion that the Department is profoundly reluctant to employ their only real weapon (withdraw of federal funds) in pursuit of a goal which Secretary Duncan has (correctly) described as utopian. A great loophole hunt may be silly, but it beats having states simply drop their cut scores or openly defy federal law while still taking federal money.

Let’s see what happens next…

Testing, Cheating, Culture and Corruption

July 21, 2011

(Guest post by Greg Forster)

Matt draws our attention to some of the broader issues raised by the APS scandal. Cheating is not just about cheating.

Here’s another one of those broader issues I think we should take note of. To call this “cheating” is really inadequate. This was a whole institutional culture in which cheating had become not just acceptable, but normal. This was way beyond teachers subtly indicating the correct answers (such as through tone of voice) or deliberately seating bad students next to good ones (so they could copy). Those things happened, but much more happened.

Teachers had “cheating parties” in which they sat around erasing and remarking student answer sheets. There was one guy whose job was to open test booklets, copy the contents, reseal them (using a lighter to melt the plastic back into place) and then distribute the contents to everybody. This was a huge, pervasive, known-to-everybody cheating system.

And cheating was not just normal but mandatory. Hark ye, my bretheren, unto the Atlanta Journal-Constitution:

For teachers, a culture of fear ensured the deception would continue.

“APS is run like the mob,” one teacher told investigators, saying she cheated because she feared retaliation if she didn’t.

Cheat – or else!

What’s going on here? This is not just the undifferentiated “corruption of human nature.” This is a very specific dynamic of institutional culture. This is a system whose organizational culture responded to NCLB by systematically embracing cheating at all levels, even to the extent of viewing non-cheaters (i.e. honest teachers) as threats to the integrity of the system.

We should think carefully about how that kind of thing happens. There is one hypothesis that sticks out to me as clearly plausible: This happened because the testing requirements of NCLB were percieved as evil, tyrannical and a threat to the integrity of education. Personnel at all levels actually viewed cheating as morally virtuous because it was necessary to protect an essential good (education) from being undermined by vicious oppressors with evil agendas. And given widespread teacher cynicism about the value of standardized tests as a metric of learning, in their perception nothing valuable was lost in the process.

This is about more than cheating. This is a wakeup call to our thinking about how reform works.

I have always been in favor of the aspect of NCLB that uses tests to create transparency. Remember, before NCLB you didn’t even have all states participating in NAEP. Anyone want to go back to that? No? Well, then, let’s not throw the baby out with the bathwater.

However, it is now pretty clear that NCLB does not work as an accountability tool. Might the systemic, institutional extent of the cheating in APS help explain why? Teachers and administrators don’t percieve the tests as legitimate – they see them as inaccurate metrics being imposed by evil oppressors as tools of exploitation – and thus don’t respond to them in positive ways. (On net, that is. Bad responses cancel out good ones.)

Contrast that with the use of testing for accountability in two other contexts. Jeb Bush’s A+ accountabiliy testing system in Florida did produce positive results. Could that be because Florida had spent years at the bottom of the national listings for education and was sick of it, and had spent years trying to improve through the tried and true ideas of the unions and was sick of failing, and was thus more open to new directions? In the context of this openness, Jeb Bush’s leadership, and his partnership with the right stakeholders, framed the reforms in a way that caused them to be experienced as legitimate at the school level.

Even more impressive, consider the use of testing in innovative charter schools like KIPP. Remember that David Brooks column blasting Ravitch? Brooks identifies what he calls “a core tension,” namely: “Teaching is humane. Testing is mechanistic.”

However, in schools where the entire institutional culture has been reinvented from the ground up around personal relationships between teacher and student that are centered around leadership, mentorship and accountability, testing isn’t experienced as mechanistic at all. Where the students really see the teachers caring about them, and vice versa, standardized testing is accepted as a tool that empowers this relationship:

The schools that best represent the reform movement, like the KIPP academies or the Harlem Success schools, put tremendous emphasis on testing. But these schools are also the places where students are most likely to participate in chess and dance. They are the places where they are mostlikely to read Shakespeare and argue about philosophy and physics. In these places, tests are not the end. They are a lever to begin the process of change…

Ravitch thinks the solution is to get rid of the tests. But that way just leads to lethargy and perpetual mediocrity. The real answer is to keep the tests and the accountability but make sure every school has a clear sense of mission, an outstanding principal and an invigorating moral culture that hits you when you walk in the door.

I think this means it’s essential that the use of tests for accountability purposes must be implemented only in contexts of institutional culture where they will be experienced as legitimate – and the degree to which the tests are used must be controlled by the degree to which the institutional culture permits this experiential legitimacy.

In some cases (as with Jeb in Florida) that could be accomplished statewide. In others it can’t. Sometimes it will have to be districts, or a network of charter schools. In many contexts it won’t work at any level. It certainly won’t work nationally, since the institutional context of the federal role in education could never permit this kind of thing to develop in a way that would be seen as legitimate.

How, then, do we drive accountability? Choice and competition, obviously. And guess what? Once schools face the disruptive threat of choice, they will be more likely to start using tests for accountability voluntarily – because they want to survive and they’ll be ready to reconsider their options.

You know, it strikes me that this principle might have application to other issues besides accountability testing. In general, the higher you go up the ladder of power – from school to district, from district to state, and from state to national – the less likely you will really be implementing your reform, and the more likely you will just be playing power games, and be seen to be playing power games, and thus cause those below you on the ladder to respond by playing power games of their own. As in Atlanta.

Heading to the Heart of Cygnus, Headlong into Mystery…

July 7, 2011

(Guest Post by Matthew Ladner)

Ed Week reports that a growing number of states have signaled their intention to ignore the 2014 deadline. USDoE threatens action against these states, and is hoping to leverage the 2014 deadline to spur Congress to act on reauthorization. Congress seems disinterested. Tennessee and others have announced that they will seek waivers, which Secretary Duncan is willing to grant in return for reform, but which Chairman Kline seems to oppose. Duncan wants a reauthorization, but it isn’t in the cards.

Where is all this heading?

Actually, a full-blown train-wreck is not inevitable there is still time to reauthorize ESEA, even if they wait until after the election. Seeing states engage in what could either be described as civil disobedience or lawlessness does send a clear signal that Congress and the administration need to deal with the 2014 event horizon, and that the Safe Harbor loophole is insufficient.

Closer…..move a little closer….a little more….GOTCHA!

Reauthorization beats waivers, and waivers beat the status-quo, which runs the risk of a great cut score dummy down. Washington would be awfully dull without some brinkmanship every now and then, so let’s see how they work this out. Something that would allow states with a system to nudge improvement out of their schools (which NCLB is doing very little of) to run their own testing systems still seems like a sensible idea to me.