A Harsh Apology to the Arizona Class of 2006 and the True Meaning of Accountability

September 2, 2015

(Guest Post by Matthew Ladner)

I had the chance to make a presentation on testing and private choice programs recently, and received a request to share a few slides from that presentation here on JPGB. So the first is from Arizona, circa 2006. I chose 2006 because we have a study that follows the entire public school Class of 2006 through the higher education system.


So for those of you squinting at your Ipad- the columns read: Kids attending AZ public schools taking state math and reading tests (100%!), AZ Class of 2006 who read proficiently as 8th graders on the 2002 NAEP 8th grade reading exam (errr, 23%), Percent of Class of 2006 graduating class who went on to earn a Bachelor degree by the end of 2012 (errr 18.6%) and finally the percent of Arizona public schools who earned an “Underperforming” or “Failing” label in 2006 (*cough* 6.5%).

So who was held “accountable” in this slide. Not the Governor she was reelected by a wide margin in 2006. Oh what about the Superintendent of Public Instruction? Nope- he was reelected as well. Did Arizona have a mass culling of ineffective school superintendents in 2006? What about teachers? Nope and nope- it was business as usual.

Let’s compare the accountability for the staff at the 6.5% of schools who received a nice-so-nice label compared to that of the students. Now that it is 2015, what is the chance that any of the adults in those 6.5% of schools carry around a nine-year old label around with them as a burden, even if they remain in education and are still remain employed at the same school? Right- now what about the 81.4 percent of the Class of 2006 who either never attended college or who were among the waves of people who dropped out of college in debt with little to show for it?

The latter scenario constitutes a much harsher form of accountability than Arizona’s former “whip truly terrible schools with a wet noodle accountability.”  Sorry Class of 2006- I know that the state of Arizona gave you the equivalent of a Good Housekeeping Seal of Approval based on your results on a laughably simple AIMS test, then they kicked the can down the road on an high-school exit exam, then they labelled your school “performing” when it was actually anything but academically performing. That “performing” actually meant that it was performing its duty to employ adults and might occasionally facilitate learning if a student had a deep motivation to do it on their own. You were very foolish not to clue into this and now you will have to pay for it.

We the taxpayers and adult policymakers of Arizona feel some regret about all of this Class of 2006 for all these things, but ultimately you should have ignored the fact that the adults in your life were telling you everything was fine, and you should have studied harder especially when you were learning to read in K-3, since 77 percent of you failed to reach proficiency as 8th graders. It was really absurd so many of you thought you could do college level work, but the universities disabused you of that notion quickly didn’t they? Don’t worry the Class of 2007 temporarily filled your spots after you dropped out.

In the next life you should not be so trusting of adults and study harder. Perhaps you will take this lesson to heart as parents.  Go suffer the consequences of your actions and just think of how much worse this might have turned out if Arizona did not hold school systems accountable. The greatest trick the public school lobby ever pulled was convincing the world that the publication of scores on minimal skills math and reading tests constituted “accountability.” And like that **poof**

…meaningful accountability was gone, unless by “accountability” we mean watching helplessly as students suffer the long-term consequences for failing to acquire competitive skills.

Now as a post-script, things have improved somewhat since 2006 in Arizona. Instead of handing out “performing” labels, the state uses letter grades. Grades of C or D are closer to truth in advertising than “Performing.” The wretched AIMS test has finally received the mercy killing it so richly deserved.  Sorry I-hate-CC-with-a-purple-passion tribe, the new test aligns much closer to Arizona’s performance on NAEP so it represents an undeniable upgrade over AIMS, at least so far. Yes, it could have been accomplished by other means etc. etc. but the sad reality is that we sat around indifferently for years as the fraud described above played out.  My humble suggestion at this point would be to offer constructive and rigorous counter proposals to AZ Merit because I hope that if you’ve reached this part of the post you’ll at least acknowledge the true horror of the AIMS regime. I mean it was cooked up by a group of Arizona teachers in 1994, which makes it near sacred and all, but that can’t make up for the system being horribly mismanaged by the AZ Department of Ed and State Board of Education after that. It devolved into a cruel joke on children.

Yes Jay I get it they probably will do the same with the new test sooner or later.  How long do you expect this peace to last?

…as long as it can.

In the end, this too shall pass, so the most enduring accountability going on in Arizona today involves parental choice. Parental choice in fact represents the ultimate form of accountability that no system of aggregate test scores and school labels can ever replace.  Even at its best such accountability is an aggregate phenomenon, whereas parental choice allows parents to hold schools responsible at the individual level by voting with their feet.

Since 2006, AZ charter schools have reached a more meaningful scale. Arizona now has the highest percentage of students attending charter schools (almost 18%) of any state. Parents have used their contacts and Greatschools to figure out that even their allegedly swell schools leave much to be desired and have commenced to pounding on the doors of high quality charter operators, developing huge waiting lists.

The highest rated general enrollment school in the Arizona Board of Regents analysis of higher education outcomes Tempe Prep- was the forerunner of the Great Hearts system of schools that now has 22 campuses and mile long waiting lists. These schools did not appear in the 2006 analysis because they either had not opened yet or did not have a senior class by 2006. Stay tuned-the Board of Regents will soon have an updated analysis. Our private choice programs in the aggregate are mostly helping private schools to remain viable against the rise of charters. We need to do more on that front, and we need to help high quality charters replicate.

Meanwhile, for the first time ever, Arizona districts find their enrollment in decline in absolute terms. Before the great recession charters and choice were simply taking the edge off of district enrollment growth. In the last couple of years there has been district enrollment declines. Enrollment growth will eventually reverse this, but for now the charters are  basically absorbing all of it and more. Oh by the way, while Arizona’s NAEP scores are not high, they were higher than they have ever been in 2013. Sweet are the uses of adversity…

Arizona’s growing choice sector has created a constituency and will not be dispatched as easily as the well-meaning but ultimately failed efforts of the AIMS regime. Keep hope alive!


TUDA and School Grading

May 2, 2013

(Guest Post by Matthew Ladner)

I had the opportunity to discuss A-F school grading with a thoughtful skeptic yesterday. Sadly my doubting Thomas remained a skeptic at the end of our discussion.  I showed him data about the trend for improving grades in Florida, and he produced data to show improving fuzzy labels from his state. I told him that Florida’s progress is confirmed by improving NAEP data, whereas his state has flatlined on NAEP over the last decade despite improved state scores. He wasn’t buying it.

My failure to persuade however got me to thinking about the Trial Urban District Assessment NAEP data. I ran the proficiency numbers for free and reduced lunch eligible students in all the districts and found the following for 4th grade reading:

TUDA 4thNote that the top 3 performers all operate under an A-F school grading system Hillsborough (Tampa), Miami-Dade and New York City (NYC has operated under A-F longer than any non-Florida district). Obviously there are plenty of other factors at play than school grading, but note that a poor child in Tampa is almost six times more likely to be reading at a proficient level than a poor child in Detroit.

So I decided to run the numbers for 8th grade reading, and found the same basic result, with the same top 3 districts, just in a slightly different order. Top performers NYC and Miami have 8th grade students more than three times more likely to read proficiently than students in the lowest performer (again Detroit).

TUDA 8th

Why do the A-F districts top the list? A 2007 study by the Urban Institute is quite suggestive:

We analyze the impact of the accountability system on Florida’s students and schools using a three-part analysis. First, we estimate the effect of the accountability system and the threat of becoming voucher eligible on student test score performance, both in the short-run and in the longer term. Second, we study the effects of the reform on school policies and practices. Finally, we attempt to determine if the policies appear to affect student achievement or explain the change in student performance. We find that student achievement significantly increased in elementary schools that received an “F” grade by between 6 to 14 percent of a standard deviation in math and between 6 to 10 percent of a standard deviation in reading in the first year. Three years later the impacts persist.

Importantly, we also detect specific school policy changes implemented by the schools that explain part of these increases. Specifically, when faced with increased accountability pressure, schools appear to focus on low-performing students, lengthen the amount of time devoted to instruction, adopt different ways of organizing the day and learning environment of the students and teachers, increase resources available to teachers, and decrease principal control. These, combined with other policies, explain more than 15 percent of the test scores gains of students in reading and over 38 percent of the test scores gains of students in math, depending on the model specification. As such we find evidence that schools respond to accountability pressure in educationally meaningful ways.

So if the powers that be mandated that you were going to come back as an urban poor child, would you want to take your chances in Miami Dade or Dallas or Detroit?

Me too- and school grading is a part of the reason why.


A Once Proud People Begin a Fight Against Hopelessness

April 30, 2013

(Guest Post by Matthew Ladner)

The Arizona Republic ran a fantastic story on their front page of this Sunday’s edition on Navajo schools in Arizona and the efforts underway to turn them around. The story shows how school grading, digital learning and immigration reform can help people who have taken a courageous decision to help themselves.

Background: schools located on the reservations in Arizona face enormous challenges and have truly abysmal test scores to show for it. Isolation, poverty and rampant alcoholism probably constitute the top three problems, though not necessarily in that order. Arizona has the lowest Native American scores on NAEP and they are not only abysmal they have been declining.

In K-12 policy discussions in Phoenix, the subject of “the Res” comes up frequently. Often people will claim that you can’t do this, that or the other thing because of “the Res.” Problems as deep as those caused in large part by a century of having the federal government “take care” of you don’t lend themselves to quick or easy solutions.

It is a long article that focuses on the personal story of Harold Begay, the Navajo Superintendent who returned to run Tuba City school district determined to turn things around. Here are the policy related parts of the story:

When the State Department of Education started assigning letter grades two years ago, Tuba City High School got a D.

It could fall to the bottom or head higher. Begay chose to go higher.When he was named superintendent, he pledged that the district would achieve the top letter grade of A.

Skip ahead….

Last summer, Tuba City High School’s grade improved from a D to a B. In addition to a better performance on standardized tests, the school showed more improvement than other low-performing schools. Navarre was honored at the state Department of Education’s office in Phoenix.

People are starting to believe what Begay told them two years ago”‘We’re going to become an ‘A’ district'” 

As a card-carrying member of the K-12 policy discussion going on in Arizona’s capital, let me be the first to confess that not me nor anyone else down in Phoenix could have ever dreamed up the policy solutions that Begay implemented in Tuba City. That is as it should be – A-F school grading was intended to put a focus on problems and call them by their proper names. Solutions come as a decentralized process.

Most of the conversations I have heard about “Res schools” have involved a sad air of resignation. The article mentions that Arizona Superintendent of Public Instruction John Huppenthal, who carried the A-F bill in the Arizona Senate and implemented it as Superintendent, became the first person in his position in twenty years to visit schools on the reservation. I don’t know whether that is accurate or not, but I think it is fair to say that if anyone has had a serious plan about what to do about reservation schools in Phoenix it has been well concealed for a very long time.

Read the article however and you’ll the solutions that Superintendent Begay developed on his own: a new emphasis on Navajo culture, hiring teachers from the Philipines and use of a digital learning platform know as Beyond Textbooks. Beyond Textbooks is a product developed by the incredibly impressive Vail Arizona school district, located at the opposite end of Arizona from Tuba City in southern Arizona.

Recruiting teachers to extremely isolated and troubled areas is a real challenge. Tuba City is 75 miles north of Flagstaff out in the middle of a very desolate nowhere. If you want a small vignette into the idiocy of our immigration laws, note that Begay is losing half of his Filipino teachers to expiring visas. We ought to be throwing these teachers a ticker-tape parade, but instead we’ve decided to boot them out of the country.

By the way, don’t hold your breath waiting for American nativists to rush to Tuba City to provide the instruction these children need.  They are ummm busy, or something. But I digress.

Tuba City High Schools jump from a D to a B grade was possible because of the emphasis on student learning gains. Twenty-five percent of a school’s grade comes from the gains of the overall student body, and another 25% from the gains of the lowest performing quartile from the previous test. If you get gains your grade gets moving. Arizona will need to nudge up the grading standards in the future but for now the system just may be working as intended by meeting the worst schools where they are at the moment.

Tuba City schools face many challenges and have a long, long way to go, but don’t make the mistake of betting against them- they are back in the fight.

The Dark Days of Educational Measurement in the Sunshine State Ended in 1999

February 8, 2012

(Guest Post by Matthew Ladner)

Over on the Shanker Blog of the American Federation for Teachers, Matthew DiCarlo writes a thoughtful but ultimately misguided post A Dark Day for Education Measurement in the Sunshine State.

DiCarlo is obviously very bright, but a few critical misinterpretations have led him astray. DiCarlo demonstrates that family income is highly correlated with student test scores in Florida. No surprise- the same is true everywhere.

Having demonstrated this, DiCarlo develops a critique of Florida’s school grading system. The Florida school grading system carefully balances overall performance on state exams with academic growth over time. Specifically, the formula weights student proficiency on state exams as 50% of a schools grade, 25% on the growth of all students, and the final 25% on the growth of students who scored in the bottom quartile on last year’s exam.

The last bit is the clever part of the formula. By double weighting the gains of students who are behind, they become the most important children in the building. Only the bottom quartile from last year’s test count in all three categories.

DiCarlo goes into the devilish details about how the state determines these gains, and concludes that some of the gains measures don’t actually measure academic growth but actually effectively measure academic proficiency. The use of proficiency levels in determining gains is critical because students are taking a higher grade level assessment with more rigorous content.  If a student achieves a proficient score on the eighth grade FCAT and then again on the ninth grade FCAT, the student is performing at a higher level because the content is more difficult.  Florida’s system does not provide credit for a learning gain for students performing Advanced in one year but Proficient the next year.

DiCarlo has failed to appreciate that the mastery of more challenging academic material from one grade to the next itself constitutes a form of academic growth.

The 9th grade student has now studied the mathematics curriculum of both 8th and 9th grade and has demonstrated  proficiency of the 8th grade material and  proficiency of the 9th grade material. Given the valid system of testing, we can feel assured that the 9th grader knows more about math than he or she knew as an 8th grader. The growth in this case is staying on track in a progressively more challenging sequential curriculum.

The Florida system, in essence, makes use of proficiency levels in order to give definition to gains and drops as meaningful. There of course is no “correct” way to structure such a system, and if 100 different people examined any given system they would likely have 500 different suggestions for improvement to match their preferences.

DiCarlo’s notion of “fairness” seems to have distracted him from a far larger and more important issue: the utility of the Florida grading system, seen best at the school grading level, has improved student achievement for all students.

If you go back as far as the FCAT data system will take you for results by Free and Reduced lunch eligibility for 3rd grade reading, you’ll find that in 2002 48% of Florida’s free and reduced lunch students scored FCAT 3 or better. In the most recent data available from 2010, 64% scored FCAT 3 or better. That is an enormous improvement in the percentage of students scoring at grade level or better.

In 2002, 60% of all Florida students scored Level 3 or above, and in 2010, 72% scored Level 3 or above. Free and reduced lunch eligible kids in 2010 outperformed ALL kids in 2002 by 4 percentage points. That’s real progress.  And the free and reduced lunch eligible children overtake the 2002 general population averages in a large majority of grades tested.

The same pattern can be found in Florida’s NAEP data. For instance, in 1998, 48% of Florida’s free and reduced lunch eligible students scored “Below Basic” on the NAEP 8th grade reading test. In 2011, that number had fallen to 35%. If an “unfair” system helps to produce a 27% decline in the illiteracy rate among low-income students, I’d like to order up a grave injustice.

The “Dark Days of Education Measurement in Florida” in my view were before school grades. Academic failure lied concealed behind a fog of fuzzy labels, and Florida wallowed near the bottom of the NAEP exams. Back when there was little transparency and even less accountability, far more students failed to acquire the basic academic skills needed to succeed in life. While perhaps a lost golden age for educators and administrators wishing to avoid any responsibility for academic outcomes, it was a Dark Age for students, parents and taxpayers.

Ironically, DiCarlo has decried a system which has weakened the link between family income and academic outcomes demonstrated in his post. Yes it is still strong in Florida, but it used to be much, much stronger.

Finally, can one truly complain about the “fairness” of a system providing more than ten times as many A/B grades as D/F grades? If anything, the Florida school grading system has grown too soft in my view (see chart above).

I’ve read enough of DiCarlo’s work to know that he is a thoughtful person, so I hope he will examine the evidence for himself and reconsider his stance. I don’t have any reason to think that the Florida system is perfect. I don’t think a perfect system exists, and I suspect that there are some changes to the Florida system that DiCarlo and I might actually agree on.

It seems however difficult to argue that the Florida system hasn’t been useful if one gives appropriate weight to the interests of students, parents and taxpayers to balance those of school staff.