The toy company Lego has refused to fill a bulk order from Chinese artist, Ai Weiwei, claiming that it “cannot approve the use of Legos for political works.”  You see, the artist has been an outspoken critic of the Chinese government and has produced works using Legos depicting fellow dissidents.  Meanwhile, Lego has a financial interest in a company that plans to open LegoLand, a theme park in Shanghai.  Political criticism wouldn’t be good for business.

So, Legos, which had never previously refused to fill a bulk order, has now decided to take a firm stand against being involved in politics, which itself feels very political.  As Ai Weiwei has said, “Everything is art. Everything is politics.”

It’s no small irony that the villain in The Lego Movie was called Lord Business.  The internet is buzzing with offers to donate Legos so that the art works can still be made.  Looks like Lord Business may be defeated again.

There are many ways a nominee for The Al could improve the human condition.  Some, like Matt’s recent nominee Malcolm McLean, are inventors of products or methods that make us better off by increasing economic prosperity.  Others, like Greg’s winning nominee Wim Nottroth, improve the human condition by fighting for greater liberty.  My nominee for this year’s Al didn’t do either of these things.  Gary Gygax, the inventor of Dungeons & Dragons (D&D), made a significant contribution to improving the human condition by giving us license to continue “playing pretend” into our adolescence and adulthood.

How does it improve the human condition for teenagers and adults to continue playing pretend?  Well, mostly because it’s fun.  Children love playing pretend, but traditionally we’ve been socialized to abandon these childish things as we assume adult responsibilities.  That might make sense when every waking moment had to be devoted to making a living, but society has grown remarkably wealthy.  That wealth has created significant opportunities for leisure, most of which people now consume by passively sitting and watching movies and TV.  D&D involves story-telling, just like movies and TV, but it is more active and social.  It is collective story-telling, with each participant adding details and shaping the plot.

I won’t bother to defend the benefits of story-telling other than to note that it is as old as human beings and universal in its appeal.  We obviously need to do it.  By inventing a new way for people to tell stories and giving us a mechanism for continuing that activity beyond childhood, Gary Gygax made a significant contribution to improving the human condition.

Let’s be clear.  D&D really is just playing pretend, not a formal game — at least if you are doing it right.  Even Gary Gygax, who made his living mostly by selling rulebooks for D&D, understood that the rules were really unnecessary when he conceded: “The secret we should never let the gamemasters know is that they don’t need any rules.”  Gygax is also reported to have said, “A DM only rolls the dice because of the noise they make.”  To play D&D you don’t need to follow rules and it doesn’t really matter what numbers the dice roll.  All you need is a group of people willing to tell a story together.

I haven’t played D&D in many years, but I remember it fondly.  In fact, I first played D&D in 1979 — the same year that the first Dungeon Master’s Guide was published.  The “game” for which I was DM was loosely based on David Bowie’s Ziggy Stardust and the Spiders from Mars.  The players had to fight the spiders and ultimately defeat Ziggy by “crush[ing] his sweet hands.”  Yep, we were incredibly geeky… but pretty cool geeks.  And we used to make fun of Gary Gygax at the time for what we thought were his stupid books and rules.  We didn’t understand that he just meant them as suggestions to get us started.  At that they succeeded.  So, thank you Gary Gygax for getting geeks everywhere to keep on playing pretend and telling stories.

I’ve been having a debate over the last few weeks with Neerav Kingland and others about the dangers of a high-regulation approach to school choice.  (You can see my posts on this so far here, here, here, here, here, and here).  I know this seems like a lot of posts on a topic, but as one grad student observed, it took more than 100 pieces about the obvious error of government reported high school graduation rates before people fully acknowledged the error and too significant steps to correct it.  Let’s hope convincing ed reform foundations and advocates to scale way back on their infatuation with heavy regulation does not require the same effort as moving more obstinate and dim-witted government officials.

One heavy-handed regulatory approach that is particularly worrisome is the strategy of portfolio management.  Under this strategy, a portfolio manager, harbor master, or some other regulator actively manages the set of school options that are available to families by closing those believed to be sub-par and expanding or replicating those that are thought to be more effective.  This approach is being implemented in New Orleans and the city appears to be experiencing significant gains in achievement tests, so Neerav and others are puzzled as to why I don’t support it.

I’ve tried to express my reasons for opposing portfolio management in several ways.  I tried mocking it: “If education reform could be accomplished simply by identifying and closing bad schools while expanding good ones, everything could be fixed already without any need for school choice.  We would just issue regulations to forbid bad schools and to mandate good ones.  See?  Problem solved.”  That clearly didn’t work because folks like NOLA advocate Josh McCarty replied: “moving the left end of the performance curve to the right through regs has gotten more kids in higher perf schools.”

So, let me try again.  A portfolio manager can only move “the left end of the performance curve” if the regulator can reliably identify which schools are likely to harm students’ long-term outcomes and which ones are likely to improve them.  If you don’t really know whether schools are on the left or right end of some curve of quality, closing schools just limits options without improving long-term outcomes.  But backers of portfolio management are not lacking in confidence.  They have achievement test results, so they think they know which are the good and bad schools.

Unfortunately, they are suffering from over-confidence.  Achievement tests are useful but they are not nearly strong enough predictors of later life outcomes to empower a portfolio manager to close a significant number of schools because he or she “knows” that those schools are “bad.”  In fact, the research I reviewed on rigorous evaluations of long-term outcomes from choice programs suggests that using test scores to decide whether a bunch of schools should be closed or expanded would lead to significant Type 1 and Type 2 errors.  That is, in their effort to close bad schools, portfolio managers may very well close schools with lower test performance that actually improve high school graduation, college-attendance, and lifetime earnings.  And they may expand or replicate schools that have high test performance but do little to improve these later life outcomes.

If there were an active portfolio manager of Florida charter schools, they would have closed a bunch of charter schools that were doing a great job of improving students’ later life outcomes.  As Booker, et al’s research shows, relying solely on test scores to distinguish good from bad schools would lead to serious errors by an active portfolio manager:

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…. Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement. (pp. 27-8)

Conversely, foundations and portfolio managers are pouring more resources into certain types of schools with strong test performance that are failing to show much benefit for students’ long-term outcomes.  As Angirst, et al, Dobbie and Fryer, and Tuttle, et al show, a bunch of charter schools with large achievement test gains, including Boston “no-excuses” schools, Harlem Promise Academy, and KIPP, have produced little or nothing in terms of high school graduation and college-attendance rates.

Portfolio management guided solely by test scores would seriously harm students by unwittingly closing a bunch of successful schools, like those Booker, et al studied in Florida, while expanding and pouring more resources into ones with less impressive long-term results, like those studied by Angirst, et al, Dobbie and Fryer, and Tuttle, et al.

Matt Barnum challenged me on Twitter to describe what evidence would persuade me to support portfolio management.  At a minimum I would want to see that portfolio managers have reliable tools for predicting long-term outcomes for students so they knew which choice schools should be closed and which should be expanded or replicated.  The evidence I’ve reviewed here and in more detail in this prior post suggests that they do not have a reliable tool and so the entire theory of portfolio management falls apart. I’m not making the strawman argument that test scores are useless or that no school should ever be closed by regulators.  I’m just arguing that portfolio management requires confidence in the predictive power of achievement tests that is not even close to being warranted by the evidence.

But what about the impressive achievement gains that Doug Harris and his colleagues find are being produced in New Orleans?  Let’s keep in mind that many reforms have been implemented in New Orleans at the same time.  Even if we were confident that the test score gains in New Orleans are not being driven by changes in the student population following Katrina (and Doug and his colleagues are doing their best with constrained data and research design to show that), and even if these test score gains translate into higher high school graduation and college attendance rates (which Doug and his colleagues have not yet been able to examine), we still would have no idea whether portfolio management and other high regulations in NOLA helped, hurt, or made no difference in producing these results.  In fact, the evidence from the 7 rigorous studies on school choice programs with long-term outcomes suggests that portfolio management and other heavy regulations are neither necessary nor desirable for producing long-term gains for students.

Neerav, Matt Barnum, and Josh McCarty have suggested that I am making overly-broad claims not consistent with evidence.  I think the opposite is true.  I’ve carefully cited and quoted the relevant research and drawn the obvious conclusion — active portfolio management based on achievement tests is likely to make harmful errors and unnecessarily restrict options.  In fact, it seems to me that the burden is on supporters of portfolio management to demonstrate that they are able to reliably distinguish between schools with good and bad long-term outcomes.  If you are going to go around telling families that they can’t choose a certain school because it is bad for them, you had darn better be confident that it really is bad.

A group of researchers at Stanford University led by Eric Bettinger have a new study comparing the effects of online versus in-person university instruction.  There is obvious appeal and lower cost associated with online courses, but rigorous evidence on its educational effectiveness has been limited.

Bettinger and his colleagues examine outcomes for a large number of students at DeVry University who took courses online or in person.  They take advantage of the fact that in-person classes are only offered in certain semesters and students have different distances to travel to get those in-person classes to obtain an exogenous estimate of the effect of online classes.

Here is what they found:

Online college courses are a rapidly expanding feature of higher education, yet little research identifies their effects. Using an instrumental variables approach and data from DeVry University, this study finds that, on average, online course-taking reduces student learning by one-third to one-quarter of a standard deviation compared to conventional in-person classes. Taking a course online also reduces student learning in future courses and persistence in college.

The bright digital future in education may not as sunny as we’ve been told.  Perhaps students are better motivated to learn by a human being in front of them in class than they are by digital methods of instruction.  If so, there may be substantial trade-offs in lost learning in exchange for the lower cost of online instruction.

Neerav Kingsland is smart, honest, and, in a positive development for ed philanthropy, is now leading education efforts at the Arnold Foundation.  I enjoyed his response to my series on the dangers of high regulation of school choice (it started here, and then had parts 1, 2, 3, and 4).  But the bulk of the critical response from Neerav and others seemed to focus on defeating a strawman rather than what I actually wrote.  They depict me as arguing against any regulation when my post was explicitly against “high-regulation” of school choice.

Maybe the mistake was on my part.  Maybe I just wasn’t clear enough, so let me succinctly reiterate here my concerns.  Some of the big reform-oriented foundations have drifted toward a high-regulation approach to school choice that I think is very dangerous and counter-productive.  Many of the features of this high-regulation approach can be seen in Louisiana and New Orleans.  They are:

1. Families should have choices, but they should only choose among quality schools.  So, a portfolio manager, harbor master, or other type of regulator should use test scores to identify who is and is not a quality school operator and eliminate from the set of options a large number of schools that appear to be sub-par.
2. All choice schools should administer state achievement tests so that regulators and families can more easily make comparisons.  Besides, it is normal to expect that government funds bring with them the requirement to demonstrate performance to the government.
3. To ensure equity of access, choice schools should not be allowed to use their own admissions criteria but should be required to take all applicants or admit by lottery.  They should also be required to accept the amount of state funding as payment in full.  And equity would be further enhanced if we targeted choice programs toward low income students in low performing traditional public schools.
4. If private schools are reluctant to go along with this high-regulation approach, maybe it is best just to concentrate on charter schools which have no alternative but to accept whatever regulations come with state funding.  Besides, private schools seem to bring with them other political problems and things uncomfortable to foundations, like religion.

In describing this high-regulation approach increasingly preferred by the big reform foundations I don’t think I am making a strawman out of their views.  This seems like a fair summary of what they believe and it describes what they have put into practice in choice programs in New Orleans and Louisiana.

But I think my objections are being turned into a strawman.  Neerav asks in the title of his post: “To Regulate or Not?”  Joel Klein similarly tweeted: “read you as saying choice per se, no regulation, is solution.”  This is not what I argued.  My argument was against high-regulation, not all regulation. Here is what I was actually trying to argue:

1. Test scores are useful but are not strong enough predictors of later life outcomes to determine which are the “quality” schools that should be among the options available to families.  So, regulators relying on test scores will experience false positives and false negatives if they try to actively manage a portfolio of schools.  This doesn’t mean that regulators should never close a school or should ignore test results (both of which are strawman arguments), but they should be significantly more humble about what they really know about school quality.  It means that they should give a fair amount of deference to parent preference and should only eliminate schools from choice programs if there are multiple indicators of failure, and exercise some human judgment about those indicators.  In the end I think I agree with Ashley Jochim who argued: “It may be that government will be more effective at establishing performance floors – much like they do in the arena of auto safety – rather than driving continuous improvement in school quality.”
2. High regulation does not improve equity of access because it drives away many of the highest quality schools.  Heavy regulation caused two-thirds of the private schools in Louisiana to refuse to participate in the state’s voucher program, including most of the best private schools in the state.  Neerav is right that I should have addressed how equity could be achieved without these regulations. Rather than trying to compel equity of access through regulations that instead drive schools out of the program, we should incentivize equity by having student-weighted voucher amounts.  That is, students who are more expensive to educate (disabled, ELL, low-income) should be provided with larger voucher amounts so that schools are compensated for taking on more expensive to educate students.  Most of our concerns about equity of access are likely to be addressed if we simply empower disadvantaged families with higher resource amounts.
3. Focusing only on charter schools because they are more compliant with a high-regulation approach is a serious mistake.  The evidence suggests that private school choice programs may have stronger later-life outcomes for students than charters.  But more importantly, I argued: “No one knows the ideal political strategy or regulatory scheme, so having a variety of different approaches [vouchers, charters, ESAs, etc…] allows us to learn about how these different methods for expanding choice are doing.  We need choice among choice.”

Just as we should be humble about using test scores to identify quality schools, we should be humble about knowing the ideal political or regulatory strategy.  While I don’t exactly know the right regulation for choice, I’m pretty confident that the high regulation strategy being pursued in LA and New Orleans is a really bad idea.  It deters quality schools from participating, it unnecessarily excludes schools from being options, and it is counter-productive in how it interferes with the operations of choice schools.

Last week Bill Gates gave a speech to articulate his foundation’s strategy for education reform and the basis for that strategy. Unfortunately, throughout the speech Gates misreads and distorts findings from research, including research that the foundation has funded or conducted itself.  Why hasn’t anyone called Gates on these errors?

Look, I’m not surprised that a non-researcher misunderstands research.  And I’m not surprised that a foundation is pursuing a strategy that is not well-supported by research.  Frankly, he’s entitled to pursue any strategy he wants for any reason he prefers.  But Gates gets up there, puts on his metaphorical lab coat, and declares that Science says… It’s then the job of actual scientists, even social scientists, to correct him.  If we don’t, then we are letting Gates corrupt science.  The Gates foundation doesn’t, for the most part, buy false research findings.  But Gates is contributing to the corruption of social science as researchers obviously feel the need to stay quiet when he and other foundation leaders misread and distort findings.

How do I know Gates is distorting and misreading research?  For one thing, several researchers involved in the studies he is referencing have told me so.  They know that he, Vicki Phillips, and other senior Gates officials are mis-describing the results of their work.  It’s no secret among researchers that this is occurring, they are just reluctant to do anything about it.

When I asked one researcher involved in the Measuring Effective Teachers (MET) project why they hadn’t corrected Vicki Phillips’ inaccurate description of their work, he agreed “it was Vicki’s mistake” but “anyone who reads the original paper can see the actual findings. I guess this is like lots of policy-relevant papers…”  That is, he just accepted the corruption and trusted that his results would speak for themselves.  Others with whom I have spoken who have worked on MET and other Gates-funded projects have not been quite so passive, but none feel brave enough to correct the Emperor.

I also know that Gates is distorting and misreading research because what he says clearly contradicts what researchers have found.  For example let’s look at how Gates claims that research supports his foundation’s shift in strategy from small schools of choice to measuring effective teaching:

Early on, we thought smaller schools were the way to drive up college-ready rates. We set out to build the model of a successful school by breaking large high schools into new, smaller ones. Those efforts did raise graduation rates.  But only some of the smaller schools also raised college-readiness rates—and the ones that did put a huge focus on training skilled teachers. So we weren’t going to reach our goals simply by changing the size of the school.

Gates is right that small schools of choice in New York City caused increased high school graduation, college-readiness, and college-attendance.  We know this from a rigorous randomized experiment conducted by MDRC, although it is important to note that these results came out several years after Gates had already abandoned the small schools strategy in 2008.  Gates starts deviating from an accurate reading of the research when he says “only some of the smaller schools also raised college-readiness rates.”  The research says nothing about how only some schools were effective and it certainly never claimed that focusing on “training skilled teachers” accounts for why some might have been more effective than others.

To justify the shift in strategy, Gates appears to be relying on the results of a set of focus group interviews MDRC and its partners conducted with teachers and principals.  These results were released in 2013, so unless the Gates Foundation has a time machine hidden within its headquarters, even this research could not account for the 2008 change in strategy.  Rather than basing the strategy on research as Gates’ speech is designed to suggest, this could at best serve as a post hoc justification.

But even the focus group interviews do not support what Gates claims.  Focus groups were held with educators at 25 of the 30 small schools of choice (SSCs) that MDRC judged to be most effective at raising high school graduation rates, not college-readiness rates (p. 63).  They didn’t compare those most effective schools to less effective ones to see if the presence or absence of any reported characteristic was associated with their effectiveness.  So, this is basically a “best practices” or selection on dependent variable analysis that cannot be used to make causal claims.  The MDRC report explicitly warns readers about this:

A second important caveat is that information from SSC interviews and focus groups only represents respondents’ opinions about the factors that make SSCs effective. This information can only suggest hypotheses; it cannot confirm them. (p. 70)

The report also never mentions “focusing on training skilled teachers” as an explanation offered by educators at the 25 most-effective small schools to explain their effectiveness.  Instead, what it does say is that 84% of principals and 64% of teachers believed that “teachers” were a factor in their school’s success (p. 19).  Teachers were the first factor mentioned most often by principals (52%) but only 8% of teachers mentioned that first.  They gave stronger emphasis to personal relationships (52%), high academic expectations (12%), and leadership (12%).  So, these focus group interviews didn’t mention “training skilled teachers” and did not even clearly give emphasis to “teachers.”

Let’s review.  Gates justifies a 2008 switch in strategy based on a 2013 study.  The study consists of focus group interviews, not a rigorous research design, and only involved the most effective schools, so as in any selection on dependent variable analysis, no causal claims can be made.  And the report doesn’t mention “focusing on training skilled teachers” and doesn’t even clearly suggest teachers as the main reason educators offered for the success of their schools.  Besides that, Mrs. Lincoln, how was the play?

In the speech Gates goes on to assert that Kentucky, Denver, Green Dot schools in LA, and DC all have achieved impressive gains because they have focused on improving teacher quality.  Even if these school systems did improve outcomes and had focused on teacher quality, this would be another example of selection on dependent variable analysis that cannot be used to support causal claims.  Maybe these school systems did other things that account for their success.  And Gates never considers that many other school systems also paid attention to teacher training without improvement in results.  This is not “evidence” to support a shift in the foundation’s strategy to measuring effective teachers.

And then Gates claims that using “multiple measures of effectiveness” when evaluating teachers is “backed by evidence” and has been essential in improving teacher skills.  I’ve dealt with this in several previous posts, but probably most directly here.  It may well be sensible to use multiple measures to assess teacher effectiveness, but this is not “backed by research” and there is no evidence that using multiple measures improves student outcomes.

I’d continue documenting how Gates misreads and distorts research findings, but by now the pattern should be clear to any observant reader.  The question is what is anyone going to do about this?  Will more researchers come forward to question the accuracy of how the Gates Foundation references research?  Will reporters finally start to ask about this?

It is time once again for us to solicit nominations for the Al Copeland Humanitarian Award.  The criteria of the Al Copeland Humanitarian Award can be summarized by quoting our original blog post in which we sang the praises of Al Copeland and all that he did for humanity:

Al Copeland may not have done the most to benefit humanity, but he certainly did more than many people who receive such awards.  Chicago gave Bill Ayers their Citizen of the Year award in 1997.  And the Nobel Peace Prize has too often gone to a motley crew including unrepentant terrorist, Yassir Arafat, and fictional autobiography writer, Rigoberta Menchu.   Local humanitarian awards tend to go to hack politicians or community activists.  From all these award recipients you might think that a humanitarian was someone who stopped throwing bombs… or who you hoped would picket, tax, regulate, or imprison someone else.

Al Copeland never threatened to bomb, picket, tax, regulate, or imprison anyone.  By that standard alone he would be much more of a humanitarian.  But Al Copeland did even more — he gave us spicy chicken.

Last year’s winner of “The Al” was Peter DeComo, the inventor of the Hemolung Respiratory Assist System.  To save a life DeComo had to trick border control officials to bring a model of his artificial lung machine into the US from Canada because the device had not yet been fully approved by the FDA.  DeComo won over a worthy field, including Marcus Persson, the inventor of Minecraft, Ira Goldman, the developer of the “Knee Defender,”  Thomas J. Barratt, the father of modern advertising, and Thibaut Scholasch and Sébastien Payen, wine-makers who improved irrigation methods.

The previous year’s winner of “The Al” was Weird Al Yankovic.  Weird Al beat an impressive set of nominees, including Penn and Teller, Kickstarter, and Bill Knudsen.

The 2012 winner of “The Al” was George P. Mitchell, a pioneer in the use of fracking to obtain more, cheap and clean natural gas. Mitchell won over a group of other worthy nominees:  Banksy, Ransom E. Olds, Stan Honey, and Alfred Fielding and Marc Chavannes.

In 2011 “The Al” went to Earle Haas, the inventor of the modern tampon.  Thanks to Anna for nominating him and recognizing that advances in equal opportunity for women had as much or more to do with entrepreneurs than government mandates.  Haas beat his fellow nominees:  Charles Montesquieu, the political philosopher, David Einhorn, the short-seller, and Steve Wynn, the casino mogul.

The 2010  winner of  “The Al” was Wim Nottroth, the man who resisted Rotterdam police efforts to destroy a mural that read “Thou Shall Not Kill” following the murder of Theo van Gogh by an Islamic extremist.  He beat out  The Most Interesting Man in the World, the fictional spokesman for Dos Equis and model of masculine virtue, Stan Honey, the inventor of the yellow first down line in TV football broadcasts, Herbert Dow, the founder of Dow Chemical and subverter of a German chemicals cartel, and Marion Donovan and Victor Mills, the developers of the disposable diaper.

And the 2009 winner of “The Al” was  Debrilla M. Ratchford, who significantly improved the human condition by inventing the rollerbag.  She won over Steve Henson, who gave us ranch dressing,  Fasi Zaka, who ridiculed the Taliban,  Ralph Teetor, who invented cruise control, and Mary Quant, who popularized the miniskirt.

Nominations can be submitted by emailing a draft of a blog post advocating for your nominee.  If I like it, I will post it with your name attached.  Remember that the basic criteria is that we are looking for someone who significantly improved the human condition even if they made a profit in doing so.  Helping yourself does not nullify helping others.  And, like Al Copeland, nominees need not be perfect or widely recognized people.