Jay P. Greene's Blog

(Guest Post by Jason Bedrick)

The big school choice news today is that Illinois has become the 18th state to enact a tax-credit scholarship program. Here’s the press release from EdChoice’s CEO, Robert Enlow:

This new program is an outstanding example of what happens when elected officials, advocates, state partners and community leaders work together on behalf of families. The Governor, lawmakers and local groups such as One Chance Illinois deserve credit for putting politics aside and students first.

We couldn’t be more excited to welcome Illinois to the school choice family, and we look forward to thousands of families having access to educational equity and options that previously were out of reach.

This is a good opportunity to revisit the famous bet JayBlog’s own Greg Forster made with Jay Mathews of the Washington Post.  Back in 2011, after Mathews had predicted that the school choice movement was basically spent, Forster challenged him:

Tell you what, Jay. Let’s make a bet. You say there won’t be “a wave of pro-voucher votes across the country”…[W]e’ll set a mutually agreed on bar for the number of voucher bills passing chambers this year. If we hit the bar, you have to buy me dinner at a Milwaukee restaurant of my choice. But if we don’t hit the bar, I buy you dinner at a DC restaurant of your choice. That’s pretty lopsided in your favor, dollar-wise. How about it?

They agreed that Forster would win if at least 10 legislative chambers passed bills in 2011 to create or expand a private school choice program. Greg has won that bet every year since, and this year is no exception. Below is the list of bills that were actually signed into law (or, in the case of Illinois, are about to be). Note that this list omits all the bills that passed one legislative chamber but either died in the other or haven’t been passed into law yet ([cough] New Hampshire! [cough cough]).

• Arizona: expanded education savings account program to near-universal eligibility
• Arkansas: expanded eligibility for voucher
• Florida: expanded funding and eligibility for ESA, expanded funding for tax-credit scholarship program
• Kansas: expanded eligibility to receive tax credits for donations to scholarship organizations to individual taxpayers in addition to corporate taxpayers
• Illinois: new tax-credit scholarship program
• Indiana: increased funding for tax-credit scholarship program
• Maryland: increased funding for voucher program
• Mississippi: expanded special-needs ESA
• New Hampshire: new town tuitioning program
• North Carolina: new special-needs ESA
• Ohio: increased funding for Cleveland Scholarship Program, expands phase-in of Income-Based Scholarship Program another year
• Oklahoma: expanded eligibility for special-needs voucher
• South Carolina: increased funding for the tax-credit scholarship program
• Tennessee: expanded eligibility for special-needs ESA

I think that calls for a GIF from Greg’s favorite movie:

UPDATE: Although it’s not a state, Patrick Wolf noted in the comments section below that Congress also re-re-authorized and expanded the D.C. Opportunity Scholarship Program.

(Guest post by Patrick J. Wolf)

New York City public charters have been much in the news of late (see here & here) for hitting it out of Yankee Stadium on student achievement.  When Judge-ing (sorry, couldn’t resist) The Big Apple’s charter school sector, add this little fact to the case:  charters out-slug their peers at a lower cost.

That is the conclusion of my latest study of charter school funding inequity, co-authored with Larry D. Maloney.  It is fun to study New York City, in part because of great potential for wordplay but also because the place is so darn big that you can disaggregate results by borough and still have district v. charter comparisons informed by large samples.  So, “start spreading the news…”

There are over 1,000,000 public school children in The Big Apple.  Seven percent of them attended charter schools during Fiscal Year 2014, the focus of our study.  Cash revenue to charter schools averaged $15,983 per-pupil while payments to district-run schools averaged a much more generous $26,560 per-pupil.  “You just wait a New York Minute Mr. Henley,” you might caution, “The New York City Department of Education actually provides in-kind services to students in charter schools that represent a funding resource not accounted for in your cash calculations.”  You would be right.  After factoring in the cash value of such in-kind services, charter schools receive a mere $4,888 less in per-pupil funding than district schools (Figure 3).  New York City charters schools are outperforming the City’s district schools at about 81 cents on the dollar.

I’ll admit that my figure isn’t nearly as MoMA-worthy as Matt’s post-modernist depiction of the Arizona school districts that refuse to accept students through inter-district choice, but it makes a crucial point.  Even accounting for the value of everything contributed in support of charter schools in New York City, district schools still get more money per student.

Critics of our prior charter school funding studies (available here and here) have claimed that we are making Big-Apple-to-Big-Orange comparisons, since district schools provide more extensive educational services to students than charters.  Our accounting for in-kind district services to charters fully addresses that argument.  After factoring in the value of co-located facilities, transportation, meals, special education services, health services, textbooks, software, etc., all of which are provided to charters in New York City so that the scope of their services is equal to that of district schools, the charters still receive less funding.  We even examined school spending patterns, in addition to funding patterns, and the story is the same.

Surely the student populations in district schools are needier than those in charter schools, thereby justifying the funding gap, right?  Actually no.  The population of charter school students in New York City contains a higher percentage of free-and-reduced price lunch kids than the population of district school students (Figure 4).

The percentage of students with disabilities is only slightly higher in district schools versus charter schools, 18.2% compared to 15.9%.  That means that districts enroll 21,342 “extra” students with disabilities compared to charters.  For the special education enrollment gap favoring districts to explain the entire funding gap favoring districts, each “extra” student with a disability in the district sector would have to cost an additional $214,376 above the cost of educating a student in general education.  It is simply implausible that the slight gap in special education enrollments explains the substantial gap in funding between district and charter schools in New York City.

Like rookie sensation Aaron Judge, this report has lots of hits besides just the homeruns described above, so check it out.  In sum, New York City has made a major commitment to provide material support to students in its public charter schools.  Still, inexplicable funding inequities persist depending simply on whether a child is in a charter or a district school.  Larry and I think this case study provides yet another Reason to support weighted student funding with full portability (see what I did there?).  Switching to such a simple and equitable method for funding all public school students definitely would put us in a “New York State of Mind.”

CREDO has produced a slew of studies comparing test score outcomes for students in charter and traditional public schools.  Those studies have come to dominate public policy and foundation discussions about charter schools and are sometimes thought to be the highest quality studies on charter effects.  They are not.

We actually have more than a dozen random-assignment studies of charter school achievement effects.  For a summary of what those gold-standard studies find, see this systematic review by Cheng, Hitt, Kisida, and Mills (or if you have difficulty with the pay-wall you can find an earlier working paper here).

CREDO’s research design is not gold standard.  It’s not even silver.  Maybe it’s formica.  It would be understandable for you to be confused and think CREDO was gold standard given how much people in policy circles talk about that research as opposed to the set of gold-standard random-assignment experiments.  And you might be further confused by the language CREDO uses when they describe their research design as comparing “virtual twins.”

CREDO’s methodology does not compare twins, virtual or otherwise.  All they are doing is comparing students who are similar on a limited set of observable characteristics — race, age, gender, and prior achievement scores.  “Matching” students on those observable characteristics is just as prone to selection bias as any other observational study that controls statistically for a handful of observed characteristics when comparing students who choose to be in different school sectors.  That is, students who choose to attend charter schools are very likely to be different from those who choose to remain in traditional public schools in ways that are not captured by their race, age, gender, and prior test score.  In particular, their desire to switch to a different kind of school may well be associated with developments in their life that might affect the future trajectory of their test scores.  In short, school choice is prone to bias from selection in observational studies like CREDO.

CREDO overstates the strength of their methodology by referring to their approach as one that compares “virtual twins.”  They say: “a ‘virtual twin’ was constructed for each closure student by drawing on the available records of students with identical traits and identical or very similar baseline test scores.” ( p. 3)  It is probably unintentional, but this description gives the false impression that they are comparing “identical” students in different sectors.  In reality they are only comparing students who are similar on a handful of observed characteristics.  Ladner and I may both have beards, enjoy a malt beverage, and are interested in school choice but that does not make us “twins” nor would it be reasonable to describe us as having “identical traits.”

Unlike CREDO, gold-standard random assignment studies are not subject to selection bias because only chance distinguishes between whether students are in charter or traditional public schools.  On average, the students being compared in randomized control trials (RCTs) are truly identical on all observed and unobserved characteristics.  They really are virtual twins.

Backers of CREDO can point to the fact that the CREDO methodology has produced results that are similar to experimental studies in a few locations and claim that selection bias must therefore not be an important problem.  This is a faulty conclusion.  Finding that CREDO’s observational method and randomized control trials sometimes produce similar results only proves that selection did not bias the results in those cases.  In other cases charters may attract students who are very different in their future achievement trajectory and RCTs would produce results that are very different from an observational study.  Online charters are likely a clear example of where this selection bias would be severe.

The problem is compounded by the fact that policymakers and foundation officials are too eager to use CREDO results for a number of reasons that have nothing to do with the quality of the methodology.  Sometimes they want to use CREDO because it supports their preferred policy conclusions.  They also have a strong preference for studies that name the city or state they are considering.

It’s as if Jonas Salk proved that the polio vaccine works in an RCT, but policymakers and foundation officials want to know if it prevents polio in New Orleans or Detroit.  Rather than rely on a lower quality research design that mentions their town, policymakers and foundation officials should focus on the highest quality charter randomized control trials, of which we have more than a dozen.  If that evidence shows the polio vaccine to work, then they should assume it also works in their town.

(Guest post by Greg Forster)

OCPA’s Perspective carries my latest, on the question of whether sending your children to public schools should be considered mandatory, forbidden or neither under Christian theology.

First I respond to a Presbyterian pastor who says Christians should feel obligated to send their kids to public schools, because hundreds of years ago churches created the schools that public schools were later modeled on:

Moore seems not to have asked why, if churches were so aggressive in creating schools, government stepped in and used its power of taxation to drive churches (mostly) out of the schooling business. One reason was because the church schools taught people to think for themselves and live independently. As industrialists grew more powerful in the 19th century, they enlisted government to create a new school system whose products would provide more submissive and narrow-minded cogs for the factory machine. (Big business and big government colluding to destroy freedom and oppress the poor—it all seems so familiar, almost as if we’ve seen it somewhere before.)

However, the more important reason was religious. Massachusetts created the first government school monopoly in America in the 1830s, partly out of submission to exploitative industrialists, but also because the Unitarian Boston Brahmins were horrified at the unreconstructed Calvinism of the Massachusetts countryside. The new public schools indoctrinated students in a religion of good works without the cross, tearing up the cultural roots of puritanism. (One would think this history would be of interest to a Presbyterian pastor.)

The tool created to destroy Calvinism in Massachusetts was soon deployed nationwide in hopes of destroying Catholicism…

Then I respond to an Anglican priest who says any school that isn’t explicitly Christian must be treated as atheistic and avoided:

It is true, as Fowler argues, that there is no such thing as religious neutrality. All human beings have some kind of cosmic worldview, and everything we do presupposes that worldview. Even to assert “2+2=4” presupposes the view that the human mind is rational, and its spontaneous intuitions about the relationships between numbers are sound (and, for that matter, that other minds exist and I can communicate meaningfully with them). All of these premises are religious, or at least metaphysical.

However, while there is no such thing as religious neutrality, there is religious ambiguity. To assert “2+2=4” is not a religiously neutral statement, but it does not settle all religious questions, either. A Christian, a Muslim, and a Hindu have three very different metaphysical accounts of what the human mind is, yet each can square “2+2=4” with their view.

Thus, a school that is not explicitly Christian need not become explicitly atheist. It need not even become implicitly atheist. It may be implicitly Christian! Or it may be a shared possession, offering an educational discourse that accommodates a diverse population without resolving their religious differences.

I argue we are neither compelled nor forbidden by Christian theology in the matter of sending children to public school:

Parents should evaluate local schools, public and private, and select the one that aligns best with their views and goals. My wife and I send our daughter to the local government school, because we are more satisfied with it on all counts—including spiritually—than the available alternatives. Others, facing other circumstances, may legitimately make other choices.

We live under neither a Babylonian captivity to public schools nor an Egyptian exodus from them. We live in the freedom of Pentecost, sent out into the nations to live in the tension of being in the world but not of it:

The problem with both Babylon and Exodus as social models for Christianity today is that they both come from the Old Testament, before Christ’s coming. With the Great Commission, Christ has sent his people out into the world; he wants disciples of every nation. Where the Jews were called out of Egypt, we are called into Egypt—our road is an eisodus, not an exodus. But we are not called to live passively, like the exiles in Babylon, merely marking time in a foreign land. The church has a mission to build godly ways of life wherever we go, and that means we can’t simply conform to the world around us or bunker down in Christian ghettos.

Whatever your faith and whatever you think of my theology, I welcome your feedback – and I promise not to lie about your views and then fire you!

In my last post I described how Atila Abdulkadiroglu, Parag Pathak, and Christopher Walters (APW) released an evaluation of the 1st year results from Louisiana’s voucher program through NBER dated December 2015 (although actually released in January 2016) that failed to cite or provide appropriate credit to earlier conference presentations by Jonathan Mills and Patrick Wolf and a dissertation by Mills.  In response APW issued a statement that raises many issues, but fails to address the heart of the matter. Before replying to some of those extraneous issues, let’s focus on the key questions:

1. Did Jon and Pat conduct analyses, write papers, and present findings to the academic community of the same program using the same basic methodology and data as APW prior to their December 2015 NBER paper?
2. Were APW aware of this prior work?
3. Did APW fail to cite and give appropriate credit to that prior work in their December 2015 NBER paper?

If you read their statement closely you will see that APW do not deny the existence of prior work, do not deny being aware of that work, and do not deny failing to cite it. Let’s take a look at what they did say.

The No Prior Working Papers Claim

APW do not deny that Jon and Pat had multiple conference papers, starting with one to the Association for Public Policy Analysis and Management on November 8, 2013 followed by 8 more during 2014 and 2015.  Instead they focus on whether there were prior working papers: “Mills and Wolf released a working paper in February 2016. This is the first working paper by the Arkansas team that we are aware of.” Christopher Walters reiterates this point in a comment on the blog: “We would have cited a public working paper had we known of one in December 2015.”

There are two problems with this no prior working paper argument.  First, a working paper is not the only format of prior academic work that requires citation.  There is no exemption to the scholarly obligation to credit prior work if that work is in the form of conference papers and presentations.  In fact, the American Economics Association helpfully provides guidance on the appropriate way to cite “Lectures and Papers Presented at Meetings.”

Second, there is a very good reason why Jon and Pat did not have a working paper for APW to cite and APW are almost certainly aware of that reason: The Louisiana Department of Education (LDE) asked Jon and Pat not to widely disseminate their findings until after the second year of results were complete. LDE had made the same request of APW, who similarly complied with that request before deciding that it would take them too long to finish the 2nd year analyses.  As APW describe it in their statement: “We postponed the release of our paper because the LDE promised us additional data in exchange for delaying public disclosure of our results. We released the paper when we judged that this data was not forthcoming.”

While Jon and Pat did not widely publicize the results of their 1st year evaluation by issuing a working paper online, they did not keep their findings a secret.  They presented their results at numerous conferences and in a dissertation, and they solicited feedback and suggestions from colleagues (including from Abdulkadiroglu).  And they took that time to resolve missing data issues, complete the 2nd year analyses, and release both 1st and 2nd year results in a working paper only a month after APW released their 1st year results.

By contrast, APW operated in a different manner.  As far as I can tell they did not present their work at any meeting of a professional association prior to December 2015.  They did not disclose to Pat and Jon that they were planning to conduct or were already conducting their own study, even when Pat and Jon discussed the research with Abdulkadiroglu.  They also did not take the time to work out data issues with LDE and wait for the 2nd year of results before releasing their NBER report. And the failure of that report to cite and credit the work produced by Jon and Pat cannot be excused because Jon and Pat didn’t put a working paper online.

The Data Sharing Agreement Claim

APW’s statement does not deny being aware of prior work by Jon and Pat.  In Walter’s comment on the blog, however, he does say “We were not aware of the Mills dissertation chapter …”  But Abdulkadiroglu, not Walters, was at the lunch with Jon and Pat in June 2015.  The statement they issued together does not deny that Abdulkadiroglu was told about Jon’s dissertation work at that lunch.  Instead it denies that Jon’s dissertation was the origin of APW’s work: “This conversation centered on the use of school assignment mechanisms for program evaluation. The conversation occurred more than two years after we signed our data agreement to evaluate the LSP using lotteries, so it was clearly not the inspiration for our work.”

Whatever inspired their work does not obviate their scholarly obligation to cite and give appropriate credit to academic work on the same program that had been produced prior to their December 2015 NBER report. The existence and date of APW’s data sharing agreement with LDE is irrelevant to whether they failed to cite and give appropriate credit to previously produced work.

But in a comment on the blog, Walters offers a novel interpretation of what “prior work” means: “Your team’s research is not ‘prior work.’ As shown by the date on the data agreement provided in our response, the two projects were in progress simultaneously.” By prior work I mean research findings that had been produced and shared with the academic community before December 2015.  The fact that both research teams had data sharing agreements does not erase the fact that Jon and Pat had produced, presented, and published (in a dissertation) results prior to the December 2015 NBER report.

In addition, it should be noted that Walters’ description that “the two projects were in progress simultaneously” is very different from APW’s characterization of Jon and Pat’s work as a “followup analysis” in the footnote they added to their NBER report after Jon and Pat released their February 2016 working paper with the 1st and 2nd year results. It should be further noted that even that amended paper does not provide a proper citation because Jon and Pat’s work is missing from the reference section.

Even though the existence and date of the data sharing agreements is irrelevant to the scholarly obligation to cite prior work, the suggestion that APW had an agreement that pre-dated Jon and Pat’s by two years is incorrect.  Jon and Pat started negotiating a data sharing agreement with LDE during the fall of 2012, around the same time as APW, and concluded that agreement on January 8, 2013, two months after APW.  Jon and Pat were completely unaware of the existence of APW’s data sharing agreement or that APW were pursuing a similar line of research with the same data.

The data sharing agreements are further irrelevant because they do not establish when the research teams actually started the research in earnest.  We know that Jon and Pat had started their research during 2013 because they presented preliminary findings at the APPAM conference in November 2013. The earliest we know APW had written-up results, according to reporting by The 74 Million, was in October 2015, when they presented them to LDE.  And when Jon and Pat discussed their research with Abdulkadiroglu in June 2015, it is unclear whether he failed to disclose that they were working on a similar study because APW had not actually started that work yet.

The You Didn’t Cite the Dissertation Either Claim

Finally, the APW statement seems to suggest that they are somehow absolved of the responsibility of citing Jonathan Mills’ dissertation because Jon and Pat also failed to cite that dissertation in some of their subsequent papers, including the February 2016 working paper with 1st and 2nd year results. First, it is an entirely different thing to deny oneself credit than to deny someone else credit.  Whether Jon and Pat made an error in failing to credit all of their own prior work does not excuse APW in making that error about someone else’s work.  This is especially the case because Jon and Pat’s failure to cite Jon’s dissertation does not mislead the reader about who had been the first to produce these results.  Second, Jon and Pat’s February 2016 working paper did cite a series of their own prior conference papers from 2014 and 2015, so they did clearly establish that they had been presenting results on the Louisiana program well before December 2015. APW might have noticed those references and acknowledged that record of prior work when they added the footnote in an update to their December 2015 NBER paper mentioning Jon and Pat’s February 2016 working paper.

Raising this issue is certainly not pleasant.  In fact, it’s down-right nerve-wracking and I can completely understand why Jon was reluctant to press this matter earlier.  But I think credit is being given to other researchers for being the first to produce an evaluation of achievement effects from the Louisiana voucher program that properly belongs to Jon and Pat.  I actually tried to resolve this amicably with Parag Pathak when he came to give a lecture at the University of Arkansas in 2016.  Parag had been invited prior to the December 2015 NBER report and his visit was uncomfortable.  It would have been more uncomfortable if either Jon or Pat were there, but Jon had already left for a post-doc at Tulane and Pat had another obligation.  Contrary to the APW statement, I did raise the bruised feelings over credit and data access with Parag and suggested that he might smooth things over with Jon and Pat and perhaps regain access to LDE data if he were to suggest collaboration on some future project with them.  I suggested at the very least he should call Pat and talk to him.  He never did.