When Evidence and Science are Really Just Assumptions and Ideology

Doug Harris has a new post that attempts to reply to the many critics of his New York Times op-ed, including me.  In the NYT piece Harris claimed that “one well-regarded study found that Detroit’s charter schools performed at about the same dismal level as its traditional public schools.” He also claimed that the relatively light regulatory approach to Detroit’s charter schools has led to “the biggest school reform disaster in the country.”  He prefers instead the heavy-regulation approach adopted in New Orleans, which he says has produced “impressive” results.  He then suggests that failing to believe these claims is “a triumph of ideology over evidence.”

Several critics, including me, noted that the “well-regarded study” Harris cites actually finds that Detroit charter schools are producing significantly greater gains than traditional public school alternatives — gains that are only slightly smaller than those in New Orleans and greater than in another high-regulation darling, Denver.  But Harris wants to continue posturing as the person backed by science and evidence, while describing his opponents as ideologues.  In his reply to critics in Education Next he uses the word “evidence” 16 times.

So, his defense of the claim that Detroit charters perform “at about the same dismal level as its traditional public schools” must be based on evidence, right? Actually, no.  He provides four arguments to rescue his assertion that “the failure of Detroit charter schools to improve student outcomes” is true despite the fact that the CREDO evidence he cited to make that claim shows otherwise.  Each one of those four arguments is based on assumptions, not evidence.  If you believe all of his assumptions, you might believe his claim that Detroit charter schools really are a disaster. But drawing conclusions that depend completely on assumptions and are contrary to the evidence he cites is certainly not science.  It seems more like a faith-based or ideological exercise.

Let’s consider his four arguments.  First he says, “given the lack of oversight in Detroit and evidence from other cities that some charter schools cherry-pick their preferred students, these results may make Detroit’s charter schools look better than they are.”  Got that?  He has no evidence that Detroit charters are cherry-picking students at all, let alone that they are doing so at a higher rate than in New Orleans, but he nevertheless posits that “if it’s happening, then the charter effects on achievement [in Detroit] are inflated.”

Second, given Harris’ assumed concerns about cherry-picking in Detroit charter schools and the inability of the CREDO study to account for that, he examines evidence from the urban NAEP test and finds that the city of Detroit has experienced below average growth in those scores in recent years.  Using NAEP results from the entire city to draw conclusions about Detroit’s charter schools requires a host of assumptions.  He’d have to assume that test results from all schools, charter and traditional, somehow speak to the effectiveness of charter schools.  He’d have to assume that demographic and other non-school factors in Detroit do not affect the comparison of test growth in Detroit relative to other cities.

A number of education analysts have coined a term — misNAEPery —  to capture how unreasonable it is to make the assumptions required to use NAEP to compare policies across jurisdictions.  Oddly, some of those analysts who like to accuse others of misNAEPery, like Morgan Polikoff and Matt Barnum, have somehow failed to denounce Harris’ use of misNAEPery in both the NYT op-ed and in the Ed Next reply to critics.  Both have even “retweeted” Harris’ new post, so we can assume they’ve read it.

And just to anticipate concerns about my own consistency on the use of NAEP, I think comparisons using NAEP that control for observed demographics are about as convincing as the CREDO results, which also rely on comparisons controlling for some observed characteristics.  Information from NAEP or CREDO can be interesting or suggestive, which is why I say that Arizona charters “appear” to be doing very well, even if I am not convinced that any of this is causal.  At the very least, it is useful to offer a disclaimer that neither NAEP nor CREDO provides convincing causal evidence, even if confession does not assure absolution.  Harris does not offer any disclaimer and instead uses his NAEP comparisons to bolster his assumption that Detroit charters may be cherry-picking.

Third, Harris builds on the observation that Detroit city has very low NAEP test scores to assume that “the extraordinarily low standing of the city as a whole, to the degree it is caused by low performance of traditional public schools, should make it easier to improve student outcomes when trying something new.”  It is also quite plausible — perhaps more plausible — that a city with extraordinarily low test scores also has severe social and economic problems that are outside of the control of schools, which would make it harder for charters to improve outcomes.

Lastly, Harris argues that even if Detroit charters have produced gains, the gains are smaller than those in New Orleans, so we should prefer the regulatory approach used in New Orleans to the one in Detroit. But this assumes that any greater gains produced by New Orleans charter schools are caused by the regulatory approach in that city.  In fact, we have no idea whether New Orleans’ regulations helped, hurt, or had no effect on how large the gains in that city were. For all we know, the gains made by New Orleans charters are largely attributable to the importation of top-notch human capital from elite colleges and a huge increase in per pupil spending, and that these gains were made despite the hindrance of burdensome regulations.  The heavy regulations in Denver somehow failed to produce gains as large as those in Detroit.

Other than these four-assumption-dependent arguments, Harris offers very little to defend the strong claims he made in the NYT that Detroit charter schools have been a “failure” and a “disaster.”  He does cite some national “charter-friendly” organizations, like CRPE and NACSA,  as being critical of Detroit charters.  But that is an argument from authority — not evidence.

He also falsely suggests that Detroit has failed to close failing charter schools.  In fact, 30% of Detroit’s charter schools have been shuttered.  And another national charter-friendly organization, NAPCS, gives Michigan higher marks on closure than Louisiana, noting that Michigan has closed more charter schools than Louisiana, 47 to 26.

Harris also cites two voucher studies with negative results — one of which is an RCT and the other not — as proof that lower regulation approaches are less effective.  Leaving aside the distinct possibility that the negative RCT result for Louisiana was actually a function of the over-regulation of that program, it’s important to note that Harris fails to cite the entire literature of rigorous studies on the effects of private school choice programs, which overwhelmingly shows positive outcomes.

I agree with Doug that ideology, which could more kindly be described as “principles” or “values,” has an important role to play in policymaking.  I just disagree with him that the evidence clearly shows the superiority of a high-regulation approach to school choice.  I think it’s more accurate to say that the evidence is unclear on this matter, which means that we may — appropriately — need to rely more on our values, principles, and broader ideology when deciding how to proceed.

[Edited to remove tangential comment that was poorly phrased.]

15 Responses to When Evidence and Science are Really Just Assumptions and Ideology

  1. matthewladner says:

    Ok so I have to confess. If examining NAEP scores across all six 2015 NAEP exam samples, drilling down on subgroup scores, tracking gains and finding consistent data entirely external to the NAEP is “misNAEPery” then I am guilty as charged!

    I will bear the charge with grim stoicism, comforted only by the fact that my state’s majority minority charter sector scored like a New England state on all six exams. Oh cruel fate!

    • You can take solace from the fact that Matt Barnum now seems to think making NAEP comparisons with demographic adjustments to draw policy conclusions is not misNAEPery: https://twitter.com/jaypgreene/status/805881748865171457 . No word from him yet on how this is any different than the previous cases he has denounced as misNAEPery. I believe the answer has something to do with how his goes to 11.

      • matthewladner says:

        That’s nice to know, but since I was careful to note that factors other than school quality almost certainly explain some of the gap between AZ districts and charters, I don’t think I ever made a causal claim. I’ve noticed however that the term “misNAEPery” has slowly morphed over time to mean “I don’t agree with the person referencing NAEP data” rather than “attributing NAEP scores or trends to a specific policy when multiple factors are simultaneously at play.”

  2. Greg Forster says:

    “A faith-based or ideological exercise”?

    Someone is forgetting the lessons of history:


    At least you didn’t paste on a picture of guys goofing around in Hare Krishna Halloween costumes.

  3. […] am thinking this conflict, as well as my current and past positions as I follow a discussion among education scholars regarding Betsy DeVos, and the evidence for and against voucher and charter policies. As is usual […]

    • Has anyone noted whether the charters they or anyone else studied followed a Common Core curriculum or did more? (i.e., how strictly state law required a CC curriculum to address the CC test taken.) That could make a huge difference in outcomes.

  4. Jason Bedrick says:

    Here’s where “just following the evidence” without any ideology gets you:

  5. Patrick says:

    Wasn’t the original definition of misNAEPery drawing conclusions about a single program by comparing NAEP scores? Now it has become using NAEP period? Or using NAEP to argue in favor for something they don’t like?

    • I think the definition is making causal claims about the effectiveness of different policies based on comparing NAEP scores across time or jurisdictions. The problem is that scores may go up or down or be higher in one place than another, but we can’t know with confidence which of the near-infinite differences across time or place is actually responsible for any change in scores. Without a causal research design we can’t make causal claims and NAEP is better suited for descriptive purposes, not causal research designs.

      But given the selective nature of when people accuse others of misNAEPery a more cynical definition would be that misNAEPery is using NAEP to make a policy claim I don’t like.

  6. Mike G says:

    10 years ago, I was at the extreme of wanting aggressive regulation of choice based on test score gains. Have certainly moved towards your view Jay, and I think I just crossed the 50 yard line. (Still not all the way there).

    Read Doug Harris’ post with interest. There was nowhere to comment there, so I’m tacking it on to yours.

    Harris writes: “…they fought against an accountability plan developed by a broad-based, bipartisan group of educators and policymakers, and on top of it, wrote the group off as “big city bosses.”

    Small point: describing the group as a broad-based, bipartisan group….come on. It has 5 key people (head of teachers union, head of NAACP, with no strong pro-charter folks).

    So it’s “broad-based” perhaps….except on the issue at hand!

    My favorite quote from one of the 5: charters can “overpopulate in areas with high population and high parent dissatisfaction.”

    The audacity!

    • Thanks, Mike! I’m glad that my writing has some effect. Of course, I’m also learning from you when you comment. So who knows? We may meet each other sometime soon.

    • Greg Forster says:

      charters can “overpopulate in areas with high population and high parent dissatisfaction.”

      Be right back, just need to get my jaw up off the floor over here.

  7. Kimberly Sheridan says:

    The 30% of charter schools reported closed in Detroit conflates those closed for financial or academic reasons. Any data on how much of the former vs. the latter?

    • It’s not really possible to separate these things. When charter schools are in financial trouble it is quite often because they have low enrollment and they have low enrollment because of academic problems

  8. Aside: Check out Diane Ravitch, “When Public Goes Private, as Trump Wants: What Happens?” (__the New York Review of Books__ 2016-12-08) defending the NEA/AFT/AFSCME cartel’s exclusive position in receipt of taxpayers’ sub-adult education subsidies.

    A slice: “… there is no evidence for the superiority of privatization in education. Privatization divides communities and diminishes commitment to that which we call the common good. When there is a public school system, citizens are obligated to pay taxes to support the education of all children in the community, even if they have no children in the schools themselves. We invest in public education because it is an investment in the future of society.”

    Sad. What happened to the scholarly author of __Left Back: A Century of Failed School Reforms__?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: