Hitt, McShane and Wolf Meta-Analysis leads to a call for humility

March 19, 2018

(Guest Post by Matthew Ladner)

My favorite quote from Hitt, McShane and Wolf’s new study:

Even with these caveats in mind, the policy implications from this analysis are clear. The most obvious implication is that policymakers need to be much more humble in what they believe that test scores tell them about the performance of schools of choice. Test scores
are not giving us the whole picture. Insofar as test scores are used to make determinations in “portfolio” governance structures or are used to close (or expand) schools, policymakers might be making errors. This is not to say that test scores should be wholly discarded.
Rather, test scores should be put in context and should not automatically occupy a privileged place over parental demand and satisfaction as short-term measures of school choice success or failure.

P.S. Letting parents take the lead on which schools expand and/or close can work out fine on the types of tests schools have almost no ability/incentive to game:

The implications of this meta-analysis of the research literature could stretch far beyond the choice sector in time. If test scores continue to show a weak and inconsistent relationship with long-term outcomes, broad rethinking will be required. Let’s see what happens next.


Systematic Review of the Disconnect Between Test Scores and Later Life Outcomes

March 19, 2018

Today AEI released a systematic review by Collin Hitt, Mike McShane, and Pat Wolf on the relationship between changes in test scores and changes in later educational attainment in rigorous studies of school choice programs.  I’ve been writing and talking about this for some time now, inspired to a large degree by informal conversations with the authors of this new report.  Now they have made the point more systematically.

They examined every study of school choice programs with both test score and attainment effects, consisting of “39 unique impact estimates across studies of more than 20 programs.”  They examine whether the direction and significance of the estimated effects of those programs on test scores are consistent with the direction and significance on attainment.  They are not.

They find: “Across the studies we examine, there is no significant or meaningful association between school choice impacts on math scores and high school graduation or college attendance. Nor are ELA impacts meaningfully associated with high school graduation rates. Under some tests, the relationship between ELA impacts and college attendance are significant—but the relationship is weak in magnitude, and the sample of studies is far narrower for college attainment than for high school graduation.”

Keep in mind that the policy relevant question is not whether individual changes in test scores are correlated with individual changes in attainment.  There is some research that has found this relationship (see for example Chetty, et al), but a surprising number of studies find no or only a weak relationship between individual gains on these near-term and later-term measures of success.  But none of them directly address the policy relevant question of whether aggregate test score changes at the school or program level are predictive of aggregate changes in attainment.

If we are going to judge schools or programs as good or bad based on changes in test scores, then those aggregate measures (not individual results) should be predictive of later success.  The fact that they are not, at least when judging school choice programs and schools, suggests that there is something fundamentally wrong with how we have approached public regulation (wrongly called “accountability”) of those programs.  You can’t regulate the quality of schools and programs if you can’t predict their quality.

Arizona Teacher Pay Frustration is Genuine but Misdirected

March 15, 2018

(Guest Post by Matthew Ladner)

A 7-year veteran teacher teaching in Arizona’s Paradise Valley Unified School District has created a social media firestorm by posting her pay stub on social media- $35,621. Before writing anything else, I like anyone else want to pay dedicated teachers much better than this. Teacher compensation decisions however are made at the district/campus level, and an examination of PVUSD finances just makes things look worse. Much worse.

The Arizona Auditor General reports on district finances. Paradise Valley received total revenue of $10,143 per pupil in 2017. This figure is above the state-wide average.  A class of 30 students produces over $300,000 in revenue. Even with benefit costs included, teacher compensation will not even sniff 20% of the revenue generated by this class of 30. Where did the rest of the money go?

This chart from the Heritage Foundation tracks national data, and provides a big part of the answer- American schools have seen a vast increase in the hiring of non-teachers. This is not to say that any school can ever do without non-teachers, but at one point we had 2 teachers for every non-teacher in American schools. These days it looks more like 1 to 1. Despite a substantial increase in inflation adjusted spending per pupil since the 2-1 days, the large increase in non-teaching staff places a limit on teacher salaries.

Ms. Milich works for a district that receives over $10,000 per pupil in revenue. That district has both an elected school board and a chapter of the Arizona Education Association that is very active in the politics of the district. A teacher with seven years under her belt getting paid $35,000 does not suggest that either the district or the association has been placing a priority on teacher compensation.

Impossible? Just take another look at the chart. American schools have spent decades placing a priority on increasing school district employment, with a much stronger focus on non-teachers. What we refer to as “teacher unions” are actually “school district employee unions” and just as a quick mental exercise close your eyes and ask yourself: do you think the chart above would look that way if the NEA and AFT opposed these trends? What if they had been supporting them for decades?

Ok open your eyes now. Get it? Good.

This teacher has every right to feel frustrated. I hope the district decides to compensate her in a fashion commensurate with her contribution to her school. I strongly suspect that contribution is far greater than a low double digit percentage of the revenue her class generates.

Yes state spending and taxing decisions also play into this- but we should never forget that Arizona is not a wealthy state, has an unusually small working age population, and decides on school funding levels through direct and indirect democracy. In 2012 statewide voters declined to raise taxes for schools in Prop. 204 by a very wide margin. A few years ago voters (very narrowly) chose to increase school funding by increasing the payout rate from state land trust.

The lopsided loss in Prop. 204 and the narrow margin on the Prop. 123 vote both relate to a deep skepticism on the part of the public that increased funding will reach teachers like this one. This skeptical attitude is entirely justified. For instance, between fiscal year 2016 and 2017 per pupil revenue increased in PVUSD by $664 per pupil (from $9,497 to $10,143) and this teacher received a raise of $131.25 and this was after completing professional development.

Arizona voters decide school funding democratically, and they will have the opportunity to increase funding again in the future. Under the current district setup, the bucket looks to have multiple large leaks if the aim is to increase teacher compensation. Some of my tribe on the center right here in Arizona tends to think that these spontaneous teacher protest movements are secretly the work of the Arizona Education Association. I don’t believe this is the case. Although much of this frustration is misdirected at choice programs, things like this social media paystub are indicators of genuinely felt grievances.

It would be a huge mistake however to continue avoiding questions about district priorities.


Charter Regulation Keeps Out Minority Charter Operators

March 9, 2018

Image result for minority charter school operators nacsa

My student, Ian Kingsbury, will be presenting a paper next week at the annual meeting of the Association for Education Finance and Policy examining factors that help explain which applications to operate charter schools are more likely to be approved.  He is still in the early stages of this project and I’m sure will benefit from feedback on how to improve the work, but he has already analyzed nearly 400 applications to operate charter schools in 7 states.  His basic findings, which seem unlikely to change as he gets feedback, should surprise no one but should shock everyone interested in charter schools — the more burdensome the regulatory environment for approving charters, the less likely charters led by minority applicants are to be approved by authorizers.

Using the score that the National Association of Charter School Authorizers (NACSA) gives to each state’s charter policies as a proxy for regulation, Ian finds that for each 1 point increase in a state’s NACSA score (on a scale from 0 to 33), African-American and Hispanic-led charter applications are 1.7 percentage points less likely to be approved.  Given that one state included in the study has had a NACSA score as low as 9 and other states, like Indiana and Nevada, have a 33, the variation in regulatory environments Ian observed is associated with about a 41 percentage point difference in the probability that minority-led charter applications would be approved.

Of course, the charter regulations favored by NACSA, the National Alliance of Public Charter Schools, and most of the charter establishment are meant to promote quality.  Unfortunately, there is no evidence that these regulations are in fact associated with higher quality charter schools. But they clearly make it harder for charter applications to get approved, especially for minority-led charter applicants. Even when Ian controls for whether the minority-led charter applicants attended more selective colleges or were affiliated with CMOs/EMOs, which might be proxies for the quality of those applications, minority-led applicants were still 1.7 percentage points less likely to get approved for each 1 point increase in NACSA score.  In other words, if the purpose of regulations is quality control, those regulations still tend to keep out minority-led charter schools even after adjusting for reasonable proxies of the quality of those proposed charter schools.

This pattern of regulations in the name of quality posing a disproportionate barrier to minorities without actually being related to quality should sound familiar to anyone who has paid attention to the issue of occupational licensure.  A variety of groups, from the Obama White House to the Institute for Justice, have noted that raising requirements to enter many occupations has been an important barrier to opportunity, especially for disadvantaged groups.  Requiring people to spend 2,100 hours and about $22,000 to obtain a cosmetology license before they can braid hair has little to do with quality but is an important obstacle to opportunity.  The same can be said of the type of regulations favored by NACSA and the charter establishment — they have little to do with quality but seem to be large obstacles to minority operated charter schools.

Keeping out minority-led charter schools has potentially serious educational and political implications.  There is some evidence that minority students fare better when educators are of their same race/ethnicity.  Minority-led charter schools may be more likely to provide this type of educational benefit for minority students.  In addition, excluding minority leaders of charter schools severely damages the political prospects for charter schools by making minority community leaders significantly less invested in the growth and success of the charter sector.

If any of you will be at the Association for Education Finance and Policy conference next week, I would encourage you to stop by Ian’s panel on Thursday (March 15) at 10:15.  While Ian’s project is not finished, the evidence is becoming clear enough that NACSA and the rest of the charter establishment need to explain why the policies they favor have such a negative effect on minority-led charter schools.  And if they are going to defend that negative effect by claiming that the policies they favor promote quality, they need to provide evidence to support that claim.  The way it looks now, the types of regulations favored by NACSA and others seem to just keep minorities out without producing any increase in quality.

Update — I’ve added a link to the paper, which is available here.

Old Dog Learns New Trick

March 7, 2018

(Guest Post by Matthew Ladner)

Ok so there is this thing inside Microsoft Excel called a “macro” and today I had Dr. Youtube teach me how to do it so I can make data labels without manually typing them in. Next step- learn how to set a VCR DVR to record!

Arizona Kids Winning in the District Competition for Enrollment

March 6, 2018

(Guest Post by Matthew Ladner)

Looking through the Arizona Auditor General report on Arizona school districts, I noticed interesting variation in enrollment trends. Arizona has been the second fastest growing state for decades, and Maricopa County has become the fastest growing county in the nation. Nevertheless, many school districts show steady enrollment trends, and some even declining enrollment. Of course there are others where enrollment is growing like mad, like Chandler (a Phoenix area suburb):

A relatively small percentage of districts are taking the lead on absorbing enrollment growth statewide, along with the charter sector. The enrollment surge in Chandler seems to be working out academically: more kids and more kids performing at a high level:

Other suburban districts have not fared as well in the competition for enrollment such as Scottsdale Unified:

This decrease in enrollment is despite of 4,000 open enrollment kids who have transferred into Scottsdale Unified. According to a district demographic study 9,000 families have paid the real estate premium to live in Scottsdale Unified boundaries but choose to send their children to school elsewhere (mostly charter schools). Scottsdale Unified has had a string of recent scandals, with the Superintendent placed on administrative leave, but we should not let this distract us from improving academic trends in Scottsdale Unified:

We can see similar trends in southern Arizona. Tucson Unified also has a declining enrollment trend:

The academic results in Tucson Unified could look better compared to peer districts (middle columns) or statewide averages (right columns):

Vail Unified is one of the Tucson area surrounding districts taking in a number of open enrollment students. Accommodating growth rather than coping with enrollment decline has been the issue in Vail:

Vail getting additional students seems to be working out academically for many students as well, with proficiency rates substantially higher than in TUSD:

Recently we learned that open-enrollment students outnumber charter students approximately 2-1 in Maricopa County. Choice in Arizona is not being done to districts, but rather by districts. Note as well that rather than a zero sum game, there are positive academic trends in Arizona districts including both districts with declining and increasing enrollment. In this competition, the kids are the winners!


The Abyss Gazes into TFA…

March 5, 2018

(Guest Post by Matthew Ladner)

Max Eden provides a helpful summary of the various scandals plaguing the District of Columbia Public School system. After three Teach for America alumni chancellors we find the district scandal-ridden and clinging to a vastly overrated NAEP record as a life preserver. Upon close examination however that life preserver looks more like an anvil than a life-vest:

The best proxy for disadvantage is parental education. The achievement of students whose parents had high school diplomas but no postsecondary education decreased by three points in math and one point in reading. That puts DCPS two points below Detroit Public Schools in reading and one point above in math. But at least Detroit improved by two points in reading and 10 points in math.

Although there’s apparently little difference between DCPS and Detroit for disadvantaged students, there should be for “evidence-based” policy experts. Given that DCPS’s spending is about twice that of Detroit, DCPS appears to be one of the worst school districts in the country for serving disadvantaged students.

The two bright spots for DC K-12 are as follows: in the choice sector for disadvantaged students and in select pockets of excellence in the District. Momma always said that DCPS would be good at something- who knew that it would be educating highly advantaged kids?

Take a good long look at the above chart. DCPS 8th grade Black students are had nowhere even close to the level mathematics achievement that DCPS White students had as 4th graders (272 for 4th grade White students in 2011 compared to 248 for 8th grade Black students in 2015).

Perhaps some of our friends who remain committed to the DCPS teacher evaluation system could grace the comment section to explain why Black students in DCPS only progressed 36 points (very meh) on NAEP math between 2011 and 2015. DCPS Black students made 38 points of math progress between 2003 and 2007, which isn’t meaningfully different- oh brave new world!

The question isn’t whether the DCPS teacher evaluation system is a magic bullet- it isn’t. Given these numbers I’m wondering if the teacher evaluation system constitutes a bullet at all. Not only did DCPS achievement remain stalled in DCPS, it actually slowed in DC charter schools. Between 2003 and 2007 Black students in DC charter schools displayed a cohort NAEP gains of 54 points, but only 46 between 2011 and 2015. I’ll put it on my to-do list to look at these numbers again when the 2013-2017 cohort gains become available.

For now, teacher evaluation looks more like a pea that got stuck in someone’s straw than a magic bullet imo. In any case, DCPS provides a cautionary tale of those holding a torch for better living through proper management. Chubb and Moe called this almost three decades ago: the fundamental problem with American K-12 is politics and the sad conclusion to draw from the DCPS experience is that the ability of school district politics to corrupt the TFA alumni network >>> than the network’s ability to redeem school district politics. Would that it were otherwise, but this conclusion is as unavoidable as it is disappointing.

Where to go from here? Eden lays out a compelling case:

DCPS is no longer a case study for education reformers, but for teachers unions. Union leaders can look at what weakened job protections and metric-chasing mandates have wrought and say, “I told you so.”

So, what should come next? Admitting a problem is the first step toward fixing it. A movement that talks incessantly about “accountability” ought to practice it within its own ranks. To maintain basic credibility, reformers must admit failure and ostracize, rather than celebrate, those responsible.