Cohort NAEP Gains by Spending

October 11, 2017


(Guest Post by Matthew Ladner)

So I’ve been cooking up some new charts on NAEP statewide cohort gains for math and reading by state K-12 per pupil spending trends. Imo the cohort gains are pretty good overall measure of statewide school quality, albeit not a perfect one. Student demographics influence all scores, but if school quality is going to assert itself they should have less of an influence on 8th grade scores than 4th grade scores simply because the kids have been in school longer. Thus question addressed along the horizon in this chart is how much math did your state’s students learn between 4th grade in 2009 and 8th grade in 2013? This is plotted against the trend in per capita spending between 2007 and 2014 per NCES.

So let’s note a few things here. First once again there is a lack of a discernible relationship between spending trend and academic outcomes. You had some states that made bid increases that bombed, and others that made big cuts and lead the nation in gains (take a bow Arizona educators and policymakers!)

Maybe this was a fluke. What happens if you do it again for math cohort gains between 2011 4th graders and 2015 8th graders?

Using my Professor X mutant super-power, I am reading your thoughts. You were thinking “Okay Ladner we get it something good is going in math. What about other subjects?” Fair enough- we can only do cohort gains in math and reading, so here is the cohort reading gains:

Well would you look at that- tied for second. Cohort gains are one method for measuring gains, but we can also look at over time gains for different cohorts of students, which allows us to bring in 4th and 8th grade science. Here is what those look like for the entire period we can get readings on all six tests (new results will be released in January 2017):

Can Arizona keep it up? I certainly hope so and we will find out in January.


Arts Integration Is a Sucker’s Game

October 11, 2017

I have a piece in Education Week that is part of a forum on arts education.  You can also find my piece on a newly launched blog that is part of the National Endowment for the Arts Research Lab that I direct at the University of Arkansas.  Since readers of this blog may not travel in circles that would take them to Ed Week or to my NEA Research Lab site, I’ve reproduced the piece below.

Arts advocates are earnest in their support of arts integration through science, technology, engineering, art, and math instruction. But as a strategy for promoting arts education, STEAM is almost certainly counterproductive as well as pedagogically unsound.

Instead, the best way to ensure that students are exposed to the arts is to set aside regular times in the school day for arts education to be taught by designated arts teachers in separate arts classrooms. If arts instruction is integrated into science, math, and other subjects, schools will be tempted to curtail separate arts classes and staff. School leaders could claim that the arts are being covered at other times, in other places, and by other staff, so there is less need to set aside specific time for the arts. By trying to put the arts almost everywhere, integration is likely to result in arts education almost nowhere.

Pursuing arts integration would be like most new-age religions: initially attractive but quickly forgotten without the regular reinforcement provided by ritual and sacred spaces. Most religious movements have learned that the best way to encourage people to contemplate the divine is to designate special places and times for worship. We might say that people should integrate religion into all aspects of their lives at all times, but the reality is that most people have a hard time doing so without reserving a time, place, and leadership for religious practice.

If you find the religious analogy unpersuasive, consider this thought experiment: Imagine that educators suggested teaching math by integrating it into the rest of the curricula. Do you think math education advocates would believe humanities teachers could cover the topic just fine? It is telling that math educators are generally not seeking to have their subject taught in arts classes.

Of course, some educators seem to think that all subjects should be integrated into all others. Why stop at STEAM? Why not add history to make it SHTEAM or add physical education to make it PHSTEAM?

The problem with this approach is that academic disciplines help organize and convey knowledge more effectively. It is pedagogically unsound to integrate all disciplines, especially when teaching young children, because it demands that students combine knowledge they do not yet possess. Students cannot gain new insights from the connections between geometry and the arts until they first have some mastery of those subjects. We cannot expect students to run before they can walk.

This type of interdisciplinary instruction also places too many demands on teachers, who would need expert knowledge in multiple fields as well as command of effective techniques for integrating them. There are exceptional teachers who can pull off true interdisciplinary instruction, but those cases are rare.

“There are exceptional teachers who can pull off true interdisciplinary instruction, but those cases are rare.”

Most arts advocates are trying to turn STEM into STEAM not because they find interdisciplinary instruction so attractive, but because they recognize that the arts are being squeezed out of the curriculum and hope to protect them by joining other, more “popular” subjects. Like the plot of a teen movie, the bullied kid hopes to find protection by joining the cool group. That story is unlikely to turn out any better for the arts than it does in those films. The only way for the arts to ensure their place in the curricula is for advocates of the arts to stand up for themselves and argue for why the arts are important on their own.

Arts advocates have made this mistake before, trying to demonstrate arts education’s value by claiming that it increases performance in math and reading. Unfortunately, as researchers Ellen Winner, Thalia R. Goldstein, and Stéphan Vincent-Lancrin documented in a 2013 report for the Organization for Economic Cooperation and Development, there is little evidence that arts instruction improves outcomes in math or reading. Some arts advocates saw priority being given to math and reading, so they hoped they could borrow some of that protection by claiming that the arts contributed to those subjects. By failing to emphasize the demonstrable benefits of arts education for those subjects, the strategy undermined the arts’ case for integration.

Arts advocates need to make the positive case for what arts education teaches, not hide behind the skirts of math and science. The arts teach particular ways of thinking about and viewing the world. The arts teach some vocationally useful skills. And most importantly, the arts connect us to our cultural heritage and teach us how to be civilized human beings. Education is not entirely about the pragmatic, but should also convey the beautiful and profound—something the arts do well. That is why arts education should be preserved in its own right.


Two Minute Face Validity Check on AZ Preliminary School Grades 2-Arcadia versus Shadow Mountain

October 7, 2017

(Guest Post by Matthew Ladner)

Yesterday I did a quick face validity test on Arizona’s new school grades by comparing the schools closest to where I live. That didn’t go terrible well for the grading system. This morning before my bike ride I decided to do a second two minute test: comparing the district high school in the neighborhood I used to live (Shadow Mountain High School in the Paradise Valley district) to the one I live near today (Arcadia High School in Scottsdale Unified).

If someone would like to justify the red columns getting a “B” while the blue columns get a “C” the comment section awaits. If you are running Arcadia High, this is going to look to you like Shadow Mountain had a larger swoon on ELA than you from 2016 to 2017, had less math improvement than you (you had a smidge and they were flat). Moreover you outscored them in both math and reading in 2017, but they got a “B” and you got a “C.” Good luck getting the Arcadia High folks on board with this.

Greatschools btw gives both Arcadia and Shadow Mountain a 6 out of 10 for their academics. In other words, Greatschools gives them both the equivalent of a D. With proficiency rates in the twenties for Shadow Mountain and the thirties for Arcadia, it is hard to argue with that assessment. If there is a D-plus to be had here, clearly Arcadia is the school more deserving of it. The state could really, really use higher levels of achievement at both of these schools btw.

The subject at hand however is the grading formula used to create these preliminary grades. It certainly seems to lack face validity to me, but the Greatschools ratings seem to be defensible and thus useful to parents.

Arizona’s New School Grades Lack Face Validity

October 6, 2017

(Guest Post by Matthew Ladner)

Reviewing Arizona’s ESSA plan the Fordham event the other day inspired a growing sense of dread as I trudged through page after page of extreme complexity regarding the state’s plan to grade schools A-F. “If Jurassic Park scientists spliced the DNA of a Franz Kafka nightmare with a Rube Goldberg machine, it would look something like this…” I recall thinking to myself six or so pages in (less than halfway) through the description of the formula.

When we discussed the concept with Arizona lawmakers originally, the idea was a straightforward mix of 50% proficiency rates, 25% gains for all students, and 25% gains for the students who scored at the bottom 25% on the previous year’s test. This formulation is both easily understood and provides an incentive for schools to avoid having students falling hopelessly behind. Why settle for something tried, true and elegant when you can develop your own new and improved version?

An ultra-complex formula provides opportunity for things to go wrong. Grades were released today, they are almost entirely based on AZMerit scores, and the grades failed the very first check of face validity I tried from schools in my own neighborhood. The above charts show AZ Merit scores from Ingleside Middle School (Scottsdale Unified) and Archway Veritas. Respectively these are the closest and the second closest schools to where I live.

Greatschools gives Archway Veritas a 10 out of 10 ranking and Ingleside an 8 out of 10. Hard to quibble with that- Ingleside’s scores are above state averages while Veritas scores are far above state averages.

Under the newly released Arizona grades, Ingleside received a “B” from the state, while Archway Veritas received a “C.”  Despite the fact that Archway students had a 30% proficiency advantage in math, and a 24% proficiency advantage in ELA, they received a lower grade. Hmmm. This alas is simply the first test of face validity I ran, which lasted all of two minutes. If I were to spend a bit more time, I’m confident I could find even more inexplicable results.

Arizona suspended letter grades years ago due to the introduction of new tests. I know many of the people involved in the effort to revise the formula to be outstanding people who care deeply about improving Arizona K-12. Nevertheless, these grades lack face validity in my book and ought to be revised using a straightforward formula. Failing that, the legislature should adopt new school labels along the lines of “Blue” “Green” and “Poka Dot” to satisfy the feds and leave the task of ranking schools to private platforms such as Greatschools and others, which is where the eyeball traffic resides already. The state, alas, seems unequal to this task.

Anyone from either districts or charter schools will find plenty of things to complain about with a bit of examination. They will be upset. They will have a right to be upset. Fortunately Arizona’s nation leading academic progress is being driven from the bottom up.

Advice to the Arnold Foundation

October 3, 2017

The Laura and John Arnold Foundation is impressive for its intellectual honesty and curiosity.  They have an education reform strategy with which I have some important differences, but they are nevertheless interested in hearing criticism of their approach, so they invited me to present my critique to their board.  Below is the essence of what I prepared for that meeting.  I don’t expect that this will cause them to alter course, nor should it.  It’s their money and they should do whatever they think best.  But the amazing thing about the Arnolds and the head of their education effort, Neerav Kingsland,  is that they are at least open to the possibility of being wrong and want to hear criticism in case they would like to reconsider any aspects of their strategy.

The heart of the Arnold reform strategy is Portfolio Management, which is a term with which they are not enamored, but is essentially a rapid expansion of choices across different sectors with a centralized and muscular system for engaging in quality control.  The Portfolio Manager would govern schools of all types in a location — traditional, charter, and perhaps private — and select which schools should be allowed to operate, which should be closed, and police certain aspects of their operations, including admissions, transportation, and perhaps special education, discipline, and other issues.  I’m a fan of the rapid expansion of choices, but I believe that the centralized and muscular quality control system produces significant educational and political damage.  I am not suggesting that the Arnold Foundation (or the charter movement in general ) abandon all quality control efforts, but I think quality is best promoted by relying heavily on parent judgement and otherwise relying on a decentralized system of authorizers with the most contextual information to make decisions about opening and closing schools if parents seem to have difficulty assessing quality on their own.  The problem with Portfolio Management is the centralized and overly-active nature of a single quality-control entity.  Here is my case in 7 points:

  1. Conceptually, Portfolio Management is no different from School Districts, so there is no reason to expect it to be any better at quality control than School Districts are.  People often claim that PM is different because its mission is only to maintain the quality of the portfolio of schools, not interfere in their operations.  But social scientists think about organizations based on their powers and incentives, not their mission.  Regardless of what PM is supposed to do, we should focus on what it can do and what is in its interest to do.  There is nothing that a PM can do or should want to do, given its organizational interests, that is not also the case for a School District.  PMs can open and close schools, just like  School Districts do.  PMs can set policies that affect school operations, just like School Districts do.  Remember that PMs have already crept into setting policies about admissions, transportation, special education, and discipline — all of which affect school operations.  And the types of schools they decide to let open or force to close shape the curriculum and pedagogy of those schools.  Also remember that School District boards do not actually operate schools, just like PMs don’t.   School District boards just set policies and decide which schools should open and close, just like PMs.  Given that they have the same powers and organizational interests, the only difference I can see between PM and School District boards is that the PM is imagined to be a good guy, who will properly be motivated by quality and avoid interfering unproductively in school operations, while School District board members (even if appointed) are imagined to be bad guys who are more concerned with satisfying special interests and following procedures than with school quality.
  2. Even if you can manage to get a PM system in place (and there are very few), and even if you manage to get “good guys” in charge of it, the good guys won’t stay in charge for very long.  The poster boy for PM, New Orleans, with its exceptional hurricane origin and large charter sector to advocate on its behalf, reverted to control by the previously reviled and inept locally elected school board in about a decade.  Yes, there is a law that swears that the school board now serving as PM will not interfere in charter operations, but these oaths of non-interference hardly provide any protection.  As discussed in 1., PMs already take actions that affect school operations by regulating their admissions, transportation, special education, and discipline.  And their ability to open and close schools can effectively control any other aspect of school operations that they wish.
  3. Even if you can get PM and keep the “good guys” in charge for longer than a decade, the PM is unable to effectively control quality because it has no tools that reliable identify quality.  Basically, the only tool available is the level of test scores.  We all fantasize about a world in which student learning growth on math and reading tests is calculated and used by central authorities to judge quality, but the reality is that very few school systems actually rely heavily on value-added measures (VAM).  In New Orleans, for example, only 5% of the school quality grade is based on VAM.  The rest is based on the level of student performance.  No reasonable person believes that the level of student performance is a reliable proxy for school quality.  Instead, the level of performance is largely a function of the severity of disadvantage among the students.  And yet the PM in New Orleans is making judgments about school closure based on a flawed measure that effectively punishes schools for trying to serve a high concentration of kids who are too disadvantaged.  And even in the imaginary world in which VAM is used, learning growth on math and reading tests only captures a narrow portion of school quality, which is why those measures are not consistent predictors of later life outcomes, like graduation, college attendance, and earnings.  As I’ve written before, you can’t manage quality if you can’t predict it, and PM does not possess any tools to reliably predict school quality.
  4. Even if we thought test score levels or the imaginary future of VAM were good enough for PMs to manage the quality of their portfolio, the heavy reliance on those measures distorts schools in ways that are educationally harmful.  To avoid the risk of being judged low quality, schools will tend to narrow their curriculum to tested subjects and even within those subjects focus more narrowly on tested items.  Other subjects and non-tested material can produce important benefits for students, but PM provides incentives for schools to neglect those benefits.  The result is that we get a homogeneous set of schools that are narrowly focused on improving test outcomes.  That kind of school might be good for some kids, but is certainly not good for all.
  5. The homogeneous set of school options that results from PM is very unlikely to offer anything that appeals to more advantaged families.  It basically results in a charter movement that is designed to serve certain urban students with no-excuses-type schools.  Suburban and more advantaged families have no interest in this kind of schooling for their own children.  By failing to offer more advantaged families any benefits, the charter movement then loses their political support, and advantaged families have much more political power than disadvantaged families.  PM advocates seem to have forgotten that politics is driven largely by self-interest manifested in organized groups.  The crushing defeat of the charter referendum in Massachusetts is at least partially explained by the political foolishness of narrowly focusing the charter movement on a certain type of school to serve disadvantaged students.  No matter what science you present to prove that those schools are good and no matter what appeals to justice you make, advantaged families will not support a movement that poses any risks to their own children and offers them no benefits.
  6. In addition to alienating advantaged, mostly white, families, PM has also alienated minority community leaders.  First, a centralized and muscular system of quality control, like PM, that is only established in urban districts clearly communicates to minority communities a lack of trust in their ability to judge quality as parents or even to judge it as decentralized charter authorizers.  It effectively says that suburbanites can choose whatever they like, but folks in big cities can’t be trusted.  Even worse, raising the barrier to entry for operating a charter school (without actually improving quality, as we already discussed) disproportionately excludes minority community leaders from operating charter schools.  It is the same principle as occupational licensure.  Things that make it hard to enter and have nothing to do with quality typically have the effect of keeping minorities and more disadvantaged people out.  Not surprisingly, we get a charter sector that is largely developed and run by white folks from elite college.  If Rev. Johnson would like to open a charter school in the classrooms he uses for Sunday school, he will have a particularly hard time completing the 700 page application in a way that satisfies the PM’s rubrics and his educational background may not appear as impressive.  But he may know his community well and his political support could be helpful.  Centralized and muscular quality control, like PM, tends to turn these folks away.
  7. The Arnold Foundation invests heavily in another initiative that promotes rigorous science for medical and policy decision-making, yet they do not seem to apply that same standard of proof to their own education strategy.  When pressed, the main evidence they point to in support of PM is a study by Doug Harris that shows that New Orleans made significant gains post-Katrina that cannot fully be explained by changes in the composition of students in the district.  Even if true, however, that study cannot tell us what New Orleans did to produce this improvement.  Perhaps the huge expansion in school choice deserves the credit and the muscular quality control added no benefit or even hurt.  Perhaps New Orleans produced gains because it imported a small army of elite college kids, greatly increasing human capital in the school system.  Maybe the large increase in spending in New Orleans deserves some of the credit.  The point is that attributing the gain to PM is unscientific, since Harris’ research was not designed to address this question.  Saying that we should pursue PM nationwide because New Orleans has it and has improved is roughly the equivalent of saying that you should wear copper bracelets because I wore them and my arthritis feel much better.  If someone made the later claim, the Arnold Foundation would (rightly) scoff at them as quacks.  I understand that foundations cannot have rigorous evidence to support all steps in their reform theory, but before pursuing a reform strategy that promises to close a bunch of schools that parents want and alienating both advantaged suburbanites and minority community leaders, they might want more evidence for the strategy than they have.

Fordham Debate on Future State Gains

October 2, 2017

(Guest Post by Matthew Ladner)

So you’re hanging out in DC, sad because you can’t get your wonk on, when suddenly you remember that Fordham is hosting a debate on whether Arizona, California, Louisiana or Tennessee have the best prospects for academic gains going forward at 3 pm! !!Spoiler alert!! Special nerdy sneak preview chart above.

Somin: Choice Is More Democratic Than Political Control

October 2, 2017


(Guest Post by Jason Bedrick)

Critics of school choice frequently argue that schools are a “public good” that requires public control via democratic institutions (i.e., school boards and state legislatures). Proponents of choice generally respond that school board elections are too easily captured by special interests, that private schools tend to do a better job of inculcating civic knowledge and values, and that school choice better respects the high value our society places on freedom and pluralism.

In a recent blog post, law professor Ilya Somin argues that if we understand democracy more broadly as “people having a say in the decisions that affect their lives” rather than mere majoritarianism (“one person/one vote”), as some modern democratic theorists argue, then school choice is also more democratic:

Much depends on exactly what it means for people “to have a say in the decisions that affected their lives.” If it merely means having some minimal opportunity to participate in the decision-making process, then African-Americans in 1950s Topeka had enough “say” to qualify. After all, they, like whites, could vote in local elections that decided who would get to direct education policy. True, they rarely actually prevailed on issues related to segregation. But repeated defeats are a standard part of the political process, especially for unpopular minorities.

Blacks living in Topeka under segregation had the right to vote, but how much of a “say” did they really have over the public policies affecting their lives? Somin continues:

But perhaps “having a say” means more than just the right to participate, but actually requires people to have a substantial likelihood of influencing the outcome. In that sense, blacks in Topeka obviously did not enjoy true “democracy.” But their painful situation was just an extreme case of a standard feature of electoral processes. In all but the smallest and most local elections, the individual voter has only an infinitesimal chance of actually influencing the result, about 1 in 60 million in a US presidential election, for example. A small minority of citizens have influence that goes well beyond the ability to cast a vote – politicians, influential activists, pundits, powerful bureaucrats, important campaign donors, and so on. But the overwhelming majority do not.

If having a say means having substantial influence over the content of public policy, most of us almost never have a genuine say. Obviously, most voters are not as dissatisfied with the resulting policies as African-Americans in the 1950s had reason to be. But that is largely because their preferences and interests happen to line up more closely with the dominant political majority, not because they actually have more than infinitesimal influence.

Perhaps you “have a say” if enough other voters share your preferences that the government is forced to follow them. But in that event, the government is still enacting your preferred policies only because powerful political forces advocate for them, not because you have any significant influence of your own. In the same way, a person who agrees with the king’s views might be said to “have a say” in the policies of an absolute monarchy. And if, as Glickman suggests, the goal is to give “all people” a say (emphasis added), then any electoral process will necessary leave many people out. There are almost always substantial minorities who strongly oppose the status quo, but have little prospect of changing it.

In short, in a majoritarian system — even one that has significant protections for minorities — you only “have a say” when a significant number of other people agree with you (either to enact your preferred policy or at least to affect whatever policy is ultimately enacted against your will).

The powerlessness of the individual voter is one of the reasons why many libertarians favor making fewer decisions at the ballot box and more by “voting with your feet.” When making choices in the market and civil society, ordinary people generally have much greater ability to make decisive choices than at the ballot box. When you decide what products to buy, which civil society organizations to join, or where you want to live, you generally have a far greater than 1 in 60 million chance of affecting the outcome. Whether or not it is more “democratic” than ballot box voting, foot voting gives individuals greater opportunity to exercise meaningful choice.

Taking the “having a say” standard seriously also entails cutting back on the powers of government bureaucracies. The latter wield vast power over many important aspects of people’s lives, often without much constraint from either foot voting or ballot box voting.

In other words, for people to truly “have a say,” then we must shift the locus of control away from politicians and bureaucrats toward individuals and families.

If having a meaningful say is the relevant criterion, it also turns out that […] school choice […] is more “democratic” than conventional public schools. In the case of the latter, most individual parents have very limited ability to influence the content of the public education available to their children. They can only do so in the rare case where they can exercise decisive influence over education policy, or by moving to a different school district. By contrast, school choice enables them to choose from a wide range of different options, both public and private. And they can do so without having to either move or develop sufficient political clout to change government policy.

If we truly want the most disenfranchised and powerless among us to have a say over their own lives, we should favor an education system that empowers them to make choices about where and how their children are educated. It’s the democratic thing to do.