I’ve been having a debate over the last few weeks with Neerav Kingland and others about the dangers of a high-regulation approach to school choice.  (You can see my posts on this so far here, here, here, here, here, and here).  I know this seems like a lot of posts on a topic, but as one grad student observed, it took more than 100 pieces about the obvious error of government reported high school graduation rates before people fully acknowledged the error and too significant steps to correct it.  Let’s hope convincing ed reform foundations and advocates to scale way back on their infatuation with heavy regulation does not require the same effort as moving more obstinate and dim-witted government officials.

One heavy-handed regulatory approach that is particularly worrisome is the strategy of portfolio management.  Under this strategy, a portfolio manager, harbor master, or some other regulator actively manages the set of school options that are available to families by closing those believed to be sub-par and expanding or replicating those that are thought to be more effective.  This approach is being implemented in New Orleans and the city appears to be experiencing significant gains in achievement tests, so Neerav and others are puzzled as to why I don’t support it.

I’ve tried to express my reasons for opposing portfolio management in several ways.  I tried mocking it: “If education reform could be accomplished simply by identifying and closing bad schools while expanding good ones, everything could be fixed already without any need for school choice.  We would just issue regulations to forbid bad schools and to mandate good ones.  See?  Problem solved.”  That clearly didn’t work because folks like NOLA advocate Josh McCarty replied: “moving the left end of the performance curve to the right through regs has gotten more kids in higher perf schools.”

So, let me try again.  A portfolio manager can only move “the left end of the performance curve” if the regulator can reliably identify which schools are likely to harm students’ long-term outcomes and which ones are likely to improve them.  If you don’t really know whether schools are on the left or right end of some curve of quality, closing schools just limits options without improving long-term outcomes.  But backers of portfolio management are not lacking in confidence.  They have achievement test results, so they think they know which are the good and bad schools.

Unfortunately, they are suffering from over-confidence.  Achievement tests are useful but they are not nearly strong enough predictors of later life outcomes to empower a portfolio manager to close a significant number of schools because he or she “knows” that those schools are “bad.”  In fact, the research I reviewed on rigorous evaluations of long-term outcomes from choice programs suggests that using test scores to decide whether a bunch of schools should be closed or expanded would lead to significant Type 1 and Type 2 errors.  That is, in their effort to close bad schools, portfolio managers may very well close schools with lower test performance that actually improve high school graduation, college-attendance, and lifetime earnings.  And they may expand or replicate schools that have high test performance but do little to improve these later life outcomes.

If there were an active portfolio manager of Florida charter schools, they would have closed a bunch of charter schools that were doing a great job of improving students’ later life outcomes.  As Booker, et al’s research shows, relying solely on test scores to distinguish good from bad schools would lead to serious errors by an active portfolio manager:

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…. Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement. (pp. 27-8)

Conversely, foundations and portfolio managers are pouring more resources into certain types of schools with strong test performance that are failing to show much benefit for students’ long-term outcomes.  As Angirst, et al, Dobbie and Fryer, and Tuttle, et al show, a bunch of charter schools with large achievement test gains, including Boston “no-excuses” schools, Harlem Promise Academy, and KIPP, have produced little or nothing in terms of high school graduation and college-attendance rates.

Portfolio management guided solely by test scores would seriously harm students by unwittingly closing a bunch of successful schools, like those Booker, et al studied in Florida, while expanding and pouring more resources into ones with less impressive long-term results, like those studied by Angirst, et al, Dobbie and Fryer, and Tuttle, et al.

Matt Barnum challenged me on Twitter to describe what evidence would persuade me to support portfolio management.  At a minimum I would want to see that portfolio managers have reliable tools for predicting long-term outcomes for students so they knew which choice schools should be closed and which should be expanded or replicated.  The evidence I’ve reviewed here and in more detail in this prior post suggests that they do not have a reliable tool and so the entire theory of portfolio management falls apart. I’m not making the strawman argument that test scores are useless or that no school should ever be closed by regulators.  I’m just arguing that portfolio management requires confidence in the predictive power of achievement tests that is not even close to being warranted by the evidence.

But what about the impressive achievement gains that Doug Harris and his colleagues find are being produced in New Orleans?  Let’s keep in mind that many reforms have been implemented in New Orleans at the same time.  Even if we were confident that the test score gains in New Orleans are not being driven by changes in the student population following Katrina (and Doug and his colleagues are doing their best with constrained data and research design to show that), and even if these test score gains translate into higher high school graduation and college attendance rates (which Doug and his colleagues have not yet been able to examine), we still would have no idea whether portfolio management and other high regulations in NOLA helped, hurt, or made no difference in producing these results.  In fact, the evidence from the 7 rigorous studies on school choice programs with long-term outcomes suggests that portfolio management and other heavy regulations are neither necessary nor desirable for producing long-term gains for students.

Neerav, Matt Barnum, and Josh McCarty have suggested that I am making overly-broad claims not consistent with evidence.  I think the opposite is true.  I’ve carefully cited and quoted the relevant research and drawn the obvious conclusion — active portfolio management based on achievement tests is likely to make harmful errors and unnecessarily restrict options.  In fact, it seems to me that the burden is on supporters of portfolio management to demonstrate that they are able to reliably distinguish between schools with good and bad long-term outcomes.  If you are going to go around telling families that they can’t choose a certain school because it is bad for them, you had darn better be confident that it really is bad.

A group of researchers at Stanford University led by Eric Bettinger have a new study comparing the effects of online versus in-person university instruction.  There is obvious appeal and lower cost associated with online courses, but rigorous evidence on its educational effectiveness has been limited.

Bettinger and his colleagues examine outcomes for a large number of students at DeVry University who took courses online or in person.  They take advantage of the fact that in-person classes are only offered in certain semesters and students have different distances to travel to get those in-person classes to obtain an exogenous estimate of the effect of online classes.

Here is what they found:

Online college courses are a rapidly expanding feature of higher education, yet little research identifies their effects. Using an instrumental variables approach and data from DeVry University, this study finds that, on average, online course-taking reduces student learning by one-third to one-quarter of a standard deviation compared to conventional in-person classes. Taking a course online also reduces student learning in future courses and persistence in college.

The bright digital future in education may not as sunny as we’ve been told.  Perhaps students are better motivated to learn by a human being in front of them in class than they are by digital methods of instruction.  If so, there may be substantial trade-offs in lost learning in exchange for the lower cost of online instruction.

Neerav Kingsland is smart, honest, and, in a positive development for ed philanthropy, is now leading education efforts at the Arnold Foundation.  I enjoyed his response to my series on the dangers of high regulation of school choice (it started here, and then had parts 1, 2, 3, and 4).  But the bulk of the critical response from Neerav and others seemed to focus on defeating a strawman rather than what I actually wrote.  They depict me as arguing against any regulation when my post was explicitly against “high-regulation” of school choice.

Maybe the mistake was on my part.  Maybe I just wasn’t clear enough, so let me succinctly reiterate here my concerns.  Some of the big reform-oriented foundations have drifted toward a high-regulation approach to school choice that I think is very dangerous and counter-productive.  Many of the features of this high-regulation approach can be seen in Louisiana and New Orleans.  They are:

1. Families should have choices, but they should only choose among quality schools.  So, a portfolio manager, harbor master, or other type of regulator should use test scores to identify who is and is not a quality school operator and eliminate from the set of options a large number of schools that appear to be sub-par.
2. All choice schools should administer state achievement tests so that regulators and families can more easily make comparisons.  Besides, it is normal to expect that government funds bring with them the requirement to demonstrate performance to the government.
3. To ensure equity of access, choice schools should not be allowed to use their own admissions criteria but should be required to take all applicants or admit by lottery.  They should also be required to accept the amount of state funding as payment in full.  And equity would be further enhanced if we targeted choice programs toward low income students in low performing traditional public schools.
4. If private schools are reluctant to go along with this high-regulation approach, maybe it is best just to concentrate on charter schools which have no alternative but to accept whatever regulations come with state funding.  Besides, private schools seem to bring with them other political problems and things uncomfortable to foundations, like religion.

In describing this high-regulation approach increasingly preferred by the big reform foundations I don’t think I am making a strawman out of their views.  This seems like a fair summary of what they believe and it describes what they have put into practice in choice programs in New Orleans and Louisiana.

But I think my objections are being turned into a strawman.  Neerav asks in the title of his post: “To Regulate or Not?”  Joel Klein similarly tweeted: “read you as saying choice per se, no regulation, is solution.”  This is not what I argued.  My argument was against high-regulation, not all regulation. Here is what I was actually trying to argue:

1. Test scores are useful but are not strong enough predictors of later life outcomes to determine which are the “quality” schools that should be among the options available to families.  So, regulators relying on test scores will experience false positives and false negatives if they try to actively manage a portfolio of schools.  This doesn’t mean that regulators should never close a school or should ignore test results (both of which are strawman arguments), but they should be significantly more humble about what they really know about school quality.  It means that they should give a fair amount of deference to parent preference and should only eliminate schools from choice programs if there are multiple indicators of failure, and exercise some human judgment about those indicators.  In the end I think I agree with Ashley Jochim who argued: “It may be that government will be more effective at establishing performance floors – much like they do in the arena of auto safety – rather than driving continuous improvement in school quality.”
2. High regulation does not improve equity of access because it drives away many of the highest quality schools.  Heavy regulation caused two-thirds of the private schools in Louisiana to refuse to participate in the state’s voucher program, including most of the best private schools in the state.  Neerav is right that I should have addressed how equity could be achieved without these regulations. Rather than trying to compel equity of access through regulations that instead drive schools out of the program, we should incentivize equity by having student-weighted voucher amounts.  That is, students who are more expensive to educate (disabled, ELL, low-income) should be provided with larger voucher amounts so that schools are compensated for taking on more expensive to educate students.  Most of our concerns about equity of access are likely to be addressed if we simply empower disadvantaged families with higher resource amounts.
3. Focusing only on charter schools because they are more compliant with a high-regulation approach is a serious mistake.  The evidence suggests that private school choice programs may have stronger later-life outcomes for students than charters.  But more importantly, I argued: “No one knows the ideal political strategy or regulatory scheme, so having a variety of different approaches [vouchers, charters, ESAs, etc…] allows us to learn about how these different methods for expanding choice are doing.  We need choice among choice.”

Just as we should be humble about using test scores to identify quality schools, we should be humble about knowing the ideal political or regulatory strategy.  While I don’t exactly know the right regulation for choice, I’m pretty confident that the high regulation strategy being pursued in LA and New Orleans is a really bad idea.  It deters quality schools from participating, it unnecessarily excludes schools from being options, and it is counter-productive in how it interferes with the operations of choice schools.

Last week Bill Gates gave a speech to articulate his foundation’s strategy for education reform and the basis for that strategy. Unfortunately, throughout the speech Gates misreads and distorts findings from research, including research that the foundation has funded or conducted itself.  Why hasn’t anyone called Gates on these errors?

Look, I’m not surprised that a non-researcher misunderstands research.  And I’m not surprised that a foundation is pursuing a strategy that is not well-supported by research.  Frankly, he’s entitled to pursue any strategy he wants for any reason he prefers.  But Gates gets up there, puts on his metaphorical lab coat, and declares that Science says… It’s then the job of actual scientists, even social scientists, to correct him.  If we don’t, then we are letting Gates corrupt science.  The Gates foundation doesn’t, for the most part, buy false research findings.  But Gates is contributing to the corruption of social science as researchers obviously feel the need to stay quiet when he and other foundation leaders misread and distort findings.

How do I know Gates is distorting and misreading research?  For one thing, several researchers involved in the studies he is referencing have told me so.  They know that he, Vicki Phillips, and other senior Gates officials are mis-describing the results of their work.  It’s no secret among researchers that this is occurring, they are just reluctant to do anything about it.

When I asked one researcher involved in the Measuring Effective Teachers (MET) project why they hadn’t corrected Vicki Phillips’ inaccurate description of their work, he agreed “it was Vicki’s mistake” but “anyone who reads the original paper can see the actual findings. I guess this is like lots of policy-relevant papers…”  That is, he just accepted the corruption and trusted that his results would speak for themselves.  Others with whom I have spoken who have worked on MET and other Gates-funded projects have not been quite so passive, but none feel brave enough to correct the Emperor.

I also know that Gates is distorting and misreading research because what he says clearly contradicts what researchers have found.  For example let’s look at how Gates claims that research supports his foundation’s shift in strategy from small schools of choice to measuring effective teaching:

Early on, we thought smaller schools were the way to drive up college-ready rates. We set out to build the model of a successful school by breaking large high schools into new, smaller ones. Those efforts did raise graduation rates.  But only some of the smaller schools also raised college-readiness rates—and the ones that did put a huge focus on training skilled teachers. So we weren’t going to reach our goals simply by changing the size of the school.

Gates is right that small schools of choice in New York City caused increased high school graduation, college-readiness, and college-attendance.  We know this from a rigorous randomized experiment conducted by MDRC, although it is important to note that these results came out several years after Gates had already abandoned the small schools strategy in 2008.  Gates starts deviating from an accurate reading of the research when he says “only some of the smaller schools also raised college-readiness rates.”  The research says nothing about how only some schools were effective and it certainly never claimed that focusing on “training skilled teachers” accounts for why some might have been more effective than others.

To justify the shift in strategy, Gates appears to be relying on the results of a set of focus group interviews MDRC and its partners conducted with teachers and principals.  These results were released in 2013, so unless the Gates Foundation has a time machine hidden within its headquarters, even this research could not account for the 2008 change in strategy.  Rather than basing the strategy on research as Gates’ speech is designed to suggest, this could at best serve as a post hoc justification.

But even the focus group interviews do not support what Gates claims.  Focus groups were held with educators at 25 of the 30 small schools of choice (SSCs) that MDRC judged to be most effective at raising high school graduation rates, not college-readiness rates (p. 63).  They didn’t compare those most effective schools to less effective ones to see if the presence or absence of any reported characteristic was associated with their effectiveness.  So, this is basically a “best practices” or selection on dependent variable analysis that cannot be used to make causal claims.  The MDRC report explicitly warns readers about this:

A second important caveat is that information from SSC interviews and focus groups only represents respondents’ opinions about the factors that make SSCs effective. This information can only suggest hypotheses; it cannot confirm them. (p. 70)

The report also never mentions “focusing on training skilled teachers” as an explanation offered by educators at the 25 most-effective small schools to explain their effectiveness.  Instead, what it does say is that 84% of principals and 64% of teachers believed that “teachers” were a factor in their school’s success (p. 19).  Teachers were the first factor mentioned most often by principals (52%) but only 8% of teachers mentioned that first.  They gave stronger emphasis to personal relationships (52%), high academic expectations (12%), and leadership (12%).  So, these focus group interviews didn’t mention “training skilled teachers” and did not even clearly give emphasis to “teachers.”

Let’s review.  Gates justifies a 2008 switch in strategy based on a 2013 study.  The study consists of focus group interviews, not a rigorous research design, and only involved the most effective schools, so as in any selection on dependent variable analysis, no causal claims can be made.  And the report doesn’t mention “focusing on training skilled teachers” and doesn’t even clearly suggest teachers as the main reason educators offered for the success of their schools.  Besides that, Mrs. Lincoln, how was the play?

In the speech Gates goes on to assert that Kentucky, Denver, Green Dot schools in LA, and DC all have achieved impressive gains because they have focused on improving teacher quality.  Even if these school systems did improve outcomes and had focused on teacher quality, this would be another example of selection on dependent variable analysis that cannot be used to support causal claims.  Maybe these school systems did other things that account for their success.  And Gates never considers that many other school systems also paid attention to teacher training without improvement in results.  This is not “evidence” to support a shift in the foundation’s strategy to measuring effective teachers.

And then Gates claims that using “multiple measures of effectiveness” when evaluating teachers is “backed by evidence” and has been essential in improving teacher skills.  I’ve dealt with this in several previous posts, but probably most directly here.  It may well be sensible to use multiple measures to assess teacher effectiveness, but this is not “backed by research” and there is no evidence that using multiple measures improves student outcomes.

I’d continue documenting how Gates misreads and distorts research findings, but by now the pattern should be clear to any observant reader.  The question is what is anyone going to do about this?  Will more researchers come forward to question the accuracy of how the Gates Foundation references research?  Will reporters finally start to ask about this?

It is time once again for us to solicit nominations for the Al Copeland Humanitarian Award.  The criteria of the Al Copeland Humanitarian Award can be summarized by quoting our original blog post in which we sang the praises of Al Copeland and all that he did for humanity:

Al Copeland may not have done the most to benefit humanity, but he certainly did more than many people who receive such awards.  Chicago gave Bill Ayers their Citizen of the Year award in 1997.  And the Nobel Peace Prize has too often gone to a motley crew including unrepentant terrorist, Yassir Arafat, and fictional autobiography writer, Rigoberta Menchu.   Local humanitarian awards tend to go to hack politicians or community activists.  From all these award recipients you might think that a humanitarian was someone who stopped throwing bombs… or who you hoped would picket, tax, regulate, or imprison someone else.

Al Copeland never threatened to bomb, picket, tax, regulate, or imprison anyone.  By that standard alone he would be much more of a humanitarian.  But Al Copeland did even more — he gave us spicy chicken.

Last year’s winner of “The Al” was Peter DeComo, the inventor of the Hemolung Respiratory Assist System.  To save a life DeComo had to trick border control officials to bring a model of his artificial lung machine into the US from Canada because the device had not yet been fully approved by the FDA.  DeComo won over a worthy field, including Marcus Persson, the inventor of Minecraft, Ira Goldman, the developer of the “Knee Defender,”  Thomas J. Barratt, the father of modern advertising, and Thibaut Scholasch and Sébastien Payen, wine-makers who improved irrigation methods.

The previous year’s winner of “The Al” was Weird Al Yankovic.  Weird Al beat an impressive set of nominees, including Penn and Teller, Kickstarter, and Bill Knudsen.

The 2012 winner of “The Al” was George P. Mitchell, a pioneer in the use of fracking to obtain more, cheap and clean natural gas. Mitchell won over a group of other worthy nominees:  Banksy, Ransom E. Olds, Stan Honey, and Alfred Fielding and Marc Chavannes.

In 2011 “The Al” went to Earle Haas, the inventor of the modern tampon.  Thanks to Anna for nominating him and recognizing that advances in equal opportunity for women had as much or more to do with entrepreneurs than government mandates.  Haas beat his fellow nominees:  Charles Montesquieu, the political philosopher, David Einhorn, the short-seller, and Steve Wynn, the casino mogul.

The 2010  winner of  “The Al” was Wim Nottroth, the man who resisted Rotterdam police efforts to destroy a mural that read “Thou Shall Not Kill” following the murder of Theo van Gogh by an Islamic extremist.  He beat out  The Most Interesting Man in the World, the fictional spokesman for Dos Equis and model of masculine virtue, Stan Honey, the inventor of the yellow first down line in TV football broadcasts, Herbert Dow, the founder of Dow Chemical and subverter of a German chemicals cartel, and Marion Donovan and Victor Mills, the developers of the disposable diaper.

And the 2009 winner of “The Al” was  Debrilla M. Ratchford, who significantly improved the human condition by inventing the rollerbag.  She won over Steve Henson, who gave us ranch dressing,  Fasi Zaka, who ridiculed the Taliban,  Ralph Teetor, who invented cruise control, and Mary Quant, who popularized the miniskirt.

Nominations can be submitted by emailing a draft of a blog post advocating for your nominee.  If I like it, I will post it with your name attached.  Remember that the basic criteria is that we are looking for someone who significantly improved the human condition even if they made a profit in doing so.  Helping yourself does not nullify helping others.  And, like Al Copeland, nominees need not be perfect or widely recognized people.

(Guest Post by Matthew Ladner)

I spent yesterday up in Prescott attending a meeting of the Classrooms First initiative, which is a panel put together by Governor Ducey to discuss school finance reform in Arizona. Arizona has both online learning programs and CTE programs that basically entail dividing the per pupil funding between a home school and an outside provider. As you might imagine, this gets messy.

On the digital side for instance schools and districts are not required to accept online courses by outside providers for credit. “How are we supposed to know whether the course was any good?” goes the refrain. Of course districts will happily accept online courses which they provided themselves- these of course have all of the necessary quality control and all. An outside observer however would struggle to discern that a district online course was any more effective than one provided by an outside provider.

So Arizona taxpayers foot the bill for all of this, and we have kids completing coursework but finding themselves denied credit for said coursework. Delightful.

CTE training has a different but related set of problems involving division of per student funding.

During this discussion it occurred to me that our state is about a third of the way through the birth process leading to an a la carte education for high school students.

Some people want our mom to stop pushing while we, er, figure things out or something. The contractions however will not agree to a pause. Lobbyist Jay Kaprosy noted that testing is moving in the direction of end of course exams, so we should consider moving the funding from the providers to the student to allow them to divvy it up among providers. A great many details would need working out to move to such a system and the Classrooms First council was understandably cautious (this topic is related but not central to their task) but my reaction:

Here is Frank Zappa during a 1975 concert at the University of Arkansas wearing a hog hat.  And below I am doing the same.  I knew we had a lot in common. (h/t Arkansas Newswire)

In my final piece of this series arguing against the high-regulation approach to school choice, I’m going to discuss whether achievement test results are a reasonable proxy for school quality.  Achievement tests are at the center of the high-regulation approach.  They are used by regulators — whether authorizers, portfolio managers, or harbor masters — to identify good and bad schools, to determine whether they should be included as choice options, and to shape the goals schools should pursue.

There is no question that growth in student learning provides us with some useful information.  The problem is that school quality is much broader than just test score results.  I always understood that achievement tests were only a partial and imperfect indicator of school quality, but I used to believe that other aspects of school quality not captured by achievement tests were largely correlated with those test results.  That is, I used to think that if a school raised scores it probably meant that students were safer, more students would graduate, more students would learn productive values, and more students would go on to become successful adults.

Unfortunately, the evidence is increasingly clear that test scores are only weakly correlated with all of these other desirable outcomes from schools.  All you have to do is look at yesterday’s post.  Schools that produce the largest achievement test gains are not necessarily the ones that produce higher graduation, or college-attendance rates.  And sometimes schools with unimpressive achievement gains make significant contributions to attainment and annual earnings when students join the workforce.  I used to think that this couldn’t be possible.  All of these happy outcomes had to be aligned.  They just aren’t.

If you are not persuaded by the evidence I reviewed yesterday on the disconnect between achievement results and other outcomes, I suggest you read an excellent book written by Nobel Prize winning economist James Heckman and his students called The Myth of Achievement Tests.

The problem is that the high-regulation approach needs achievement tests to be correlated with all of these other good outcomes.  If they are going to pick the school choice winners and losers based on test scores, then test scores need to be strongly predictive of other things we care about.  People have been very slow to accept the fact that test scores are only weakly correlated with later life outcomes because it would be so convenient if readily available and relatively inexpensive test scores could capture something as complex as school quality.  The fact that they don’t throws a monkey wrench into the entire high-regulation machinery.

The reality is that the average low-income parent has more complete information about their kid’s school quality than does the highly-trained regulator armed only with test scores.  When we wonder why parents are choosing schools that regulators and other distant experts deem to be “bad,” it is almost certainly because the parents know more about what is good and bad than do the experts.

The wrong response to recognizing that test scores fail to capture school quality sufficiently is to increase the set of high-stakes measures we collect.  We can’t fix the limits of math and reading achievement tests by adding mandatory “grit” surveys or other measures.  Even informed by a variety of measures, Chinese officials are no more effective in telling state-controlled banks how to allocate capital than portfolio managers are in determining how to allocate school options.  Decentralized decision-making is simply better than central planning.

The school choice movement has to remember that choice is what makes this reform work, not the regulation.  I’m perfectly willing to accept that some regulation is necessary and inevitable.  And I’m willing to make compromises to get programs adopted.  But the cardinal sin of the high-regulation school choice folks is that they believe that heavy regulation is the ideal and should be the starting point for political compromises.