Lies, Damned Lies, and NYT Statistics

January 31, 2017


(Guest Post by Jason Bedrick)

Earlier this month, Max Eden and I showed how three separate data sets employing three different methodologies all reached the same conclusion: Detroit’s charter schools are significantly outperforming Detroit’s district schools.

So how did the New York Times come to paint such a different narrative?

That’s the question Eden tackles at The Seventy-Four this week, and it isn’t pretty.

First, NYT reporter Kate Zernike rejected the findings from a credible center-right think tank purely for political reasons. In an email conversation with Eden, she argued that the Mackinac Center is “a partisan group that is pro–school choice and anti-[Detroit Public Schools],” as though that had a bearing on whether its data were accurate.

Second, she demonstrated little familiarity with either the data source she rejected or the one upon which she relied. She claimed Mackinac “only” used graduation rates as its basis of comparison, but that’s completely false. She also thought that Excellent Schools Detroit (ESD) — her preferred data source — adjusted their data for demographics, but they didn’t. Mackinac did.

Far more egregious is how she portrayed the ESD data. Eden painstakingly takes readers through her calculations, but the short story is this: in calculating the average performance of Detroit’s district schools, she inappropriately excluded the district schools that were so low performing that the state intervened and took over. She also inappropriately included selective-admission magnet schools that require students to maintain a certain GPA and pass a test to gain entrance — something charters and traditional district schools cannot do. She also compared a weighted average for the supposed “district” school performance against the median charter performance. Eden concludes:

If that sounds silly, it’s because comparing an average to a median is statistical nonsense. The “apples to oranges” metaphor is apt but insufficient here. Essentially, Zernike took a basket of apples, pulled out the rotten ones, kept the genetically modified ones, made statistically weighted applesauce, and plopped that applesauce in the middle of a row of organic oranges. Then she drew a false conclusion that’s become central to the case against Betsy DeVos’s nomination for secretary of education.

Eden also took Zernike to task for digging in her heels over her demonstrably false claim that “Ms. DeVos pushed back on any regulation as too much regulation.” As Eden details — and several others have detailed previously — DeVos has supported all sorts of regulations on choice programs. Indeed, I wish DeVos were as libertarian as Zernike portrays her, but the record indicates otherwise. As Eden notes, Zernike should have known better:

In a Detroit News op-ed, to which [Zernike’s] article later links, DeVos called for two additional regulations: A–F school accountability grades and default closure for failing schools, both charter and district. She certainly pushed back on some regulations as too much. But the bill that passed included the additional accountability regulations for which she advocated. In fact, the final legislation boosted Michigan’s accountability score on the National Alliance of Charter School Authorizers index.

Zernike, sadly, still refuses to acknowledge these glaring errors. Instead, in response to criticism, she has tried moving the goalposts and hoping no one would notice. Indeed, she’s even repeating the claim that Detroit’s charter sector “is no one’s model” even though I have repeatedly pointed out to her that the 2015 CREDO study called Detroit’s charter sector — wait for it — a “model to other communities.” As I’ve noted before, I think that’s overstated, but you can’t seriously claim that “no one” thinks Detroit is a model when, in fact, the most wide-ranging study of charter schools conducted by a research center at one of the most respected university’s in the world used that very word to describe Detroit’s charters.

Zernike has her narrative and she’s sticking to it, facts be damned. Moreover, this isn’t the first time Zernike has let her narrative get ahead of her reporting (for example, see pages 33-37 here for a long list of “errors of omission and commission” in her highly flawed reporting on a voucher study by Harvard’s Paul Peterson).

What’s particularly frustrating is that she claims to be an objective, bias-free journalist (“[I] don’t really have an opinion“) when it is obvious from her reporting (or her Twitter feed) that she’s a dyed-in-the-wool liberal. Now, there’s nothing wrong with that. Pretty much everyone has a worldview, especially those who spend a good deal of their time thinking about issues related to public policy. The problem isn’t having a worldview, it’s not admitting it, and therefore not taking steps to make sure that it doesn’t cloud your judgment (or your reporting). As Jonah Goldberg wrote recently:

Reporters routinely call experts they already agree with knowing that their “takes” will line up with what the reporter believes. Sometimes this is lazy or deadline-driven hackery. But more often, it’s not. And that shouldn’t surprise us. Smart liberal reporters are probably inclined to think that smart liberal experts are right when they say things the smart liberal reporters already agree with.

For these and similar reasons, liberal ideas and interpretations of the facts sail through while inconvenient facts and conservative interpretations send up ideological red flags. Think of editors like security guards at a military base. They tend to wave through the people they know and the folks with right ID badges. But when a stranger shows up, or if someone lacks the right credential, then the guards feel like they have to do their job. This is the basic modus operandi for places like Vox, which seek to explain not the facts or the news, but why liberals are right about the facts and the news. […]

And you know what, the same thing is true for conservative journalists, because it’s true of people… The distinction is that there aren’t a great number of conservative journalists, certainly not in print, who don’t openly admit their biases to the reader. There are literally thousands of mainstream journalists, editors, and producers who insist that they are objective — and who actually believe it. And that leaves out the fact that liberalism is besotted with the idea that liberals aren’t ideological at all in the first place, which makes it even harder for them to recognize their ideological biases.

All journalists have is their credibility. Keeping it requires admitting errors when necessary. It should be clear to everyone that Zernike botched her reporting of the data on Detroit’s charter schools and misrepresented DeVos’s views on regulations — significant errors that have had a real impact on the narrative surrounding a cabinet pick shortly before her confirmation hearings and vote.

A responsible and credible news organization would correct the record.

What Advice Might Kahneman and Tversky Offer Foundations?

January 31, 2017

I’d like to continue my review of Michael Lewis’ new book,  The Undoing Project, on the collaboration between Daniel Kahneman and Amos Tversky by speculating about what advice Kahneman and Tversky might offer education foundations.

Foundations have particular challenge in detecting and correcting errors in their own thinking.  Because many people want things from foundations, especially their money, foundations frequently are organized as to limit communication with them.  They generally don’t open their doors, phone lines, and emails to whoever might want to suggest something or ask something of them for fear that they will be overwhelmed.  So they typically hide behind a series of locked doors, don’t make their phone or emails readily available, and insist that all applications follow a specific format, be submitted at a particular time, and be designed to address pre-determined issues.

Insulating themselves from external influence is understandable, but it creates real problems if they ever hope to detect when they are mistaken and need to make changes.  The limited communication that does make it through to foundations tends to re-affirm whatever they are already doing.  Prospective grantees don’t like to tell foundations that they are mistaken and need to change course because that makes getting funded unlikely.  Instead, foundations tend to hear that their farts smell like roses.  To make matters worse, many foundations are run in very top-down ways, which discourage questioning and self-criticism.

The Undoing Project presents a very similar situation having to do with airline pilot errors.  Lewis describes a case in which a commercial airline was experiencing too many accidents caused by pilot error.  The accidents were not fatal, but they were costly and dangerous — things like planes landing at the wrong airport.  So the airline approached Amos Tvsersky and asked for help in improving their training so as to minimize these pilot errors.  They wanted Tversky to design a pilot training method that would make sure pilots had the information and skills to avoid errors.

Tversky told the airline that they were pursuing the wrong goal.  Pilots are going to make errors and no amount of information or training would stop them from committing those mistakes.  We often assume that our errors are always caused by ignorance, but Tversky told them this was not true.  The deeper problem is that once we have a mental model of the world, we tend to downplay or ignore information that is inconsistent with that model and bolster facts that support our model.  If a pilot thinks he is landing at the right airport, he distorts available information to confirm that he is landing in Fort Lauderdale rather than nearby Palm Beach even if that is incorrect.  The problem is not a lack of information, but our tendency to fit information into our pre-conceived beliefs.

Tversky’s advice was to change the cockpit culture to facilitate questioning and self-criticism.  At the time cockpits were very hierarchical based on the belief that co-pilots needed to implement pilot orders quickly and without question lest the delay and doubt promote indecision and disorder.  So the airline implemented Tversky’s suggestions and changed their training to encourage co-pilots to doubt and question and pilots to be more receptive to that criticism.  The results was a marked deline in accidents caused by pilot error.  Apparently we aren’t very good at detecting our own errors, but we are more likely to do so if others are encouraged to point them out.

So what might Tversky suggest to education foundations?  I think he’d recognize that they have exceptional difficulty in detecting their own errors and need intentional, institutional arrangements to address that problem.  In particular, he might suggest that they divide their staff into a Team A and Team B.  Each team would work on a different theory of change — theories that are not necessarily at odds with each other but are also not identical.  For example, one team might focus on promoting school choice and another on promoting test-based accountability.  Or one team may promote tax credits and the other ESAs.  The idea of dividing staff into somewhat competing teams is that they then have incentives to point out shortcomings in the other team’s approach.  Dividing into Team A and Team B could be a useful check on the all-too-common problem of groupthink.

Another potential solution is to hire two or three internal devil’s advocates whose job it is to question the assumptions and evidence believed by foundation staff.  To protect those devil’s advocates, it is probably best to have them report directly to the board rather than the people they are questioning.

Whatever the particular arrangements, the point is that education foundations should strive to promote an internal culture of doubt and self-criticism if they wish to catch and correct their own errors and avoid groupthink.  One foundation that I think has taken steps in this direction is the Arnold Foundation.  They actually hold internal seminars in which they invite outside speakers to come and potentially offer critiques of their work.  Neerav Kingsland, who heads education efforts at Arnold, is also especially available on blogs and twitter for critical discussion.  I don’t always agree with Neerav but I am impressed by his openness to dissent.

The collaboration between Kahneman and Tversky was itself an example of the importance of engaging in tough criticism within an effort.  Like the airline pilots, they developed habits of challenging each other, which made their work together better than it ever could have been individually.

Review of The Undoing Project

January 30, 2017

When I was in graduate school I read a lot of what was then new research by Daniel Kahneman and Amos Tversky.  I found their documentation of the systematic ways in which people deviate from rational decision-making fascinating and I was searching for a way to apply that to political science questions.  In the end, I couldn’t figure out how to build a new theory based on systematic irrationality.

Reading Michael Lewis’ excellent new book, The Undoing Project, about the amazing and eventually problematic collaboration between Kahneman and Tversky brought back a flood of nostalgia but also reminded me of some problems with trying to extend their work.  In particular, I was reminded of two things.  First, while Kahneman and Tversky are remarkably persuasive in demonstrating how people regularly deviate from rationality, neither I nor others have had much success in building new theories based on systematic irrationality.  As it turns out, assuming rationality is clearly an inaccurate description of how people think, but it remains quite useful for building theories that yield accurate predictions.  That is, Kahneman and Tversky may have revolutionized social science much less than Lewis suggests.

Second, much of the work that has tried to build on Kahneman and Tversky seems to violate their basic finding that expert judgement is unreliable.  The development of behavioral economics and its application to a variety of fields, including education, mostly seems to consist of trying to devise ways to correct the systematic irrationality of others.  If low-income students are accepted to college but do not enroll after failing to complete the FAFSA financial aid form, we assume they are behaving counter to their long-term interests and propose interventions to induce them to complete the form and enroll.

As I’ve written elsewhere, this approach has a variety of problems, but the chief of which is that it assumes too much rationality on the part of the social scientist devising the solutions.  How do we know that people would be better off if we could nudge them into doing something other than what they had originally decided to do?  Just as other people may be systematically irrational, so may the social scientists devising plans for improving other people’s lives.  I’m not saying that no interventions are helpful.  I’m just saying that we should be extremely cautious and humble when developing plans for how other people should live their lives.

The need for humility among experts and social scientists was a central theme in Kahneman and Tversky’s work.  Their approach was not, as one critic accused them, a psychology of stupid people; it was a psychology of all people, including experts and social scientists.  In fact, one of their first experiments was to give statisticians problems to see if they would update their priors as if they were Bayesians.  As it turns out, even statisticians who you might think would be particularly familiar with Bayes’ Theorem, do not actually think like Bayesians.  In subsequent experiments they found that even warning subjects of the systematic irrationality to which they might be prone does not prevent them from being systematically irrational.  Greater knowledge and expertise do not prevent us from falling into the same intellectual potholes over and over again.

So Kahneman and Tversky’s research demonstrates that there is no priestly class immune to the shortcomings of others and even foreknowledge and confession of one’s sins of irrationality provide little protection against repeating common errors.  And yet, much of behavioral economics seems to pay little heed to this central finding as they move full steam ahead devising solutions for other people’s irrationality.  They seem to forget that devising solutions, building models, and testing them all require human judgements which are also prone to systematic error.

In his seminal volume, Thinking, Fast and SlowKahneman admits there is no real solution to our tendency to deviate from rationality.  Instead, he suggests some habits to check the errors, mostly involving slowing down, being more cautious and self-critical, as well as inviting the criticism of others.  Let’s not correct for popular mistakes by installing a technocratic elite because that elite are also prone to common errors.

Real Accountability Is Choice, Not Regulation

January 27, 2017


(Guest Post by Jason Bedrick)

Imagine that a lobbyist from the taxi industry argued that rideshare companies like Uber and Lyft are “unaccountable” because they aren’t subject to the same regulations as the taxi industry.

The right response, of course, would be to laugh at the absurdity. Clearly Uber and Lyft drivers are much more accountable than taxi drivers because they are directly accountable to the consumer. Passengers rate their drivers based on the quality of their experience, so drivers tend to work hard to ensure that passengers have a good experience. When was the last time you got into a taxi and were offered candy or your choice of music?

Now imagine that someone responded to the taxi lobbyist with something like the following:

That may have been the case years ago, when Uber first burst onto the scene. Rideshare companies were not generally subject to price controls — not even to prohibit “surge pricing” —and even if they were, there were no requirements to purchase a taxi medallion or obtain a commercial license. And, to be fair, that’s still the case for some rideshare companies, where commercial insurance requirements remain light to nonexistent.

But what some rideshare-doubters might not know is that some of the newest and biggest rideshare markets—like those in New York and Maryland—now have significant accountability provisions that are arguably even stronger than those found in many taxi laws. That’s no accident. Pro-rideshare lawmakers adopted these taxi-like requirements because some of us accountability hawks and advocacy groups pushed for them.

In Austin, Texas, participating rideshare companies must run fingerprint background checks on all their drivers. In New York City, drivers must obtain a Taxi and Limousine Commission permit and license plate that’s used to show that the driver meets the city’s standards. Mayor Bill De Blasio has also pushed for imposing a medallion-like system that would keep the growth in the number of rideshare drivers down to manageable levels.

So if you oppose Uber and Lyft because of lack of accountability, it may be time to change your position.

Sadly, that’s almost exactly what Fordham’s Mike Petrilli wrote this week in attempting to defend choice programs from the spurious charge that they are “unaccountable.” The above paragraphs are merely a revised version of what he actually wrote (as Matt highlighted earlier this week). Rather than explain that the very act of choosing is, itself, a strong form of accountability, Mike instead lists all the ways in which some voucher programs subject schools to top-down government regulations–“just like their public school counterparts.”

This fundamentally misunderstands accountability. As I explained at the Heritage Foundation earlier this week, true accountability is when service providers are directly answerable to the people most affected by their performance. When that isn’t possible, as when a utility company has a monopoly, top-down regulations may be necessary instead. But we shouldn’t confuse the inferior alternative accountability regime for the ideal form of accountability just because that’s what we’re used to. As Thomas Sowell has written, “It is hard to imagine a more stupid or more dangerous way of making decisions than by putting those decisions in the hands of people who pay no price for being wrong.”

Sadly, we’ve become so accustomed to the “accountability” regime imposed on the monopoly district school system that many people have not only come to confuse it for true accountability, but they no longer recognize true accountability when they see it.

UPDATE: For my comments at Heritage this week, plus even wiser commentary from Jay Greene and Yuval Levin, see here:


The Way of the Future: Vertical Farming

January 24, 2017

(Guest Post by Matthew Ladner)

The interwebs are full of predictions of doom for Big Box retail- Amazon continues to surge, Macy’s and Sears closing stores, etc. What the Sears catalog had been to rural general stores, so to is Amazon to retailers of today.

Now Big Box may strike back by finding its online footing and value in their physical locations, and thus news of its death may be greatly exaggerated. But then again, maybe not. Free two-day delivery was impressive, but two-hour delivery is tough to beat. Fasten your seat belt because this ride is going to get very turbulent.

Let’s assume for a moment that Big Box retail continues to flounder as more and more Americans discover the delights of e-commerce. What becomes of all of that real estate?

A recent New Yorker article on new agricultural techniques suggests one possibility: vertical farming. The practice involves stacking of crops in trays indoors and spraying their roots with mineral enhanced water rather than planting them in soil. Based upon the information provided in the article and my Wilson Middle School Algebra, this technique uses 9% of the freshwater of that utilized in conventional farming to produce the same amount of crops. The technique is also hyper efficient in the use of land, potentially freeing large amounts farmland for other uses. In addition, since you can utilize this technique basically anywhere, sellers can reduce shipping costs. If government policy ever had an attack of reason and allowed market forces to play a greater role in agricultural water use, the attractiveness of these techniques would be even greater.

The company featured in the New Yorker article is operating out of Newark New Jersey. The price of real estate is the likely reason for a Newark as opposed to a Manhattan operation. Climate controlled indoor space will be needed to make this practice thrive, and as luck would have it, a great deal more of it may be looking for different uses.

Of course, this speculative piece may seem entirely misguided a couple of decades because someone figured an even better use for the space. I’ve heard for instance that a charter school operator converted a Target into a very nice school facility. Perhaps agriculture will follow a different path. Let’s see what happens next.


Begun the School Choice Week Has

January 23, 2017

(Guest Post by Matthew Ladner)

School Choice Week is upon us, which means it is time for Step Up for Students to update us with their above cool graphic showing the progress of student options in Florida. Btw there is more of this on the way after the Florida Supreme Court wisely decided not to hear the Florida Education Association’s appeal of their lawsuit against the programs.

Jason has a good post over at Cato about action in the states. Jason predicts Year of School Choice, Jr. while Greg gets his steak knives ready.

Finally my favorite tweet of the year (so far) comes from Expect More Arizona Director of Programs and Policy Geoff Espo:




Meanwhile MOOCs continue to grow and evolve…

January 18, 2017

(Guest Post by Matthew Ladner)

The Economist provides a current state of play for the MOOC space, Georgia Tech announces a new ultra-low cost Masters Degree in analytics, and EdX has a summary of who uses their courses, earns credentials etc.