Pondiscio: Choice Is Not About Test Scores

March 6, 2017

il_340x270-1128328451_66jh

(Guest Post by Jason Bedrick)

In case you missed it, in today’s U.S. News & World Reportthe inimitable Robert Pondiscio gently chides fellow school choice advocates for getting caught up in a debate over test scores, which are ancillary to the true value of school choice:

Wonky battles over research studies can be illuminating. They can also be irrelevant or premature. While [school choice] advocates are correct that the preponderance of evidence tends to favor school choice, this entire debate puts the cart before the horse. When we look to test-based evidence – and look no further – to decide whether choice “works,” we are making two rather extraordinary, unquestioned assumptions: that the sole purpose of schooling is to raise test scores, and that district schools have a place of privilege against which all other models must justify themselves.

That’s really not what choice is about. Choice exists to allow parents to educate their children in accordance with their own needs, desires and values. If diversity is a core value of yours, for example, you might seek out a school where your child can learn alongside peers from different backgrounds. If your child is a budding artist, actor or musician, the “evidence” that might persuade you is whether he or she will have the opportunity to study with a working sculptor or to pound the boards in a strong theater or dance program. If your child is an athlete, the number of state titles won by the lacrosse team or sports scholarships earned by graduates might be compelling evidence. If faith is central to your family, you will want a school that allows your child to grow and be guided by your religious beliefs. There can be no doubt that, if you are fortunate enough to select a school based on your child’s talents or interests or your family’s values and traditions, the question of whether school choice “works” has already been answered. It’s working perfectly for you.

Deciding whether or not to permit parents to choose based on test-based evidence is presumptuous. It says, in effect, that one’s values, aspirations and priorities for one’s child amount to nothing. Worse, our evidence-based debate presumes that a single, uniform school structure is and ought to be the norm, and that every departure from that system must justify itself in terms of a narrow set of outcomes that may not reflect parents’ – or society’s – priorities. Academic outcomes matter, of course, but so do civic outcomes, character development, respect for diversity and faith and myriad others.

This isn’t to say that the research on the effect of school choice on test scores is meaningless. But it has to be read and understood in the broader context. Test scores are important, but they’re far from what’s most important about exercising educational choice. As Pondiscio concludes:

School choice proponents who seek to prove that vouchers, tax credits and scholarships “work” by citing test-score-based research have allowed themselves to be lured into argument that can never be completely won. They have tacitly agreed to a reductive frame and a debate over what evidence is acceptable (test scores) and what it means to “win” (better test scores). This is roughly akin to arguing whether to shop at your neighborhood grocery store vs. Wal-Mart based on price alone. Price is important, but you may have reasons for choosing the Main Street Grocery that matter more to you than the 50 cents per pound you’d save on ground beef. Perhaps Main Street’s fresh local produce and personal service are more important to you.

If we limit the frame of this debate to academic outputs alone, every new study provides ammunition, but never a conclusion. The real debate we should be having is, “What kind of system do we want?” Answer that question first, then use evidence to improve the school designs, policies and programs we have agreed deserve public support.

Amen, brother!


Kevin Carey Flashes Back to 2009 for a Wild West tax credit tale

March 3, 2017

(Guest Post by Matthew Ladner)

Kevin Carey is at it again- this time by flashing back to eight year old allegations about the Arizona tax credit program as a dire warning about the dangers of a federal tax credit. When these stories ran in 2009, here is what I had to say about it here on the Jayblog:

When presented with this type of information, the first instinct of some will be to deny it, to hunker down, to accuse our enemies of far greater misdeeds, or to otherwise try to put lipstick on a pig. Good luck with that.  It is blindingly obvious to me that Arizona’s tax credit is system is a good program overall that suffers from specific weaknesses that can and must be addressed.  Otherwise, writing articles like this one will become the journalistic equivalent of using a shot-gun to shoot fish in a bucket.

Since then, things have improved substantially, but Kevin did not get the memo. Here are a few items that Kevin left out:

Subsequent to 2009, the state enacted new legislation to require STOs to both consider financial need in the granting of scholarships, and to report to the Arizona Department of Revenue on the family income of recipients. When you examine the Arizona Department of Revenue Reports, you find that approximately 80 to 90 scholarship funds went to middle and low-income students. This not only is a more progressive distribution than many public schools and school districts, it beats the living daylights out of another Arizona tax credit for public school kids that overwhelmingly goes to advantaged public schools. Quite frankly it is likely that a large majority of private choice funds were going to middle and low-income children before the state required reporting. It’s just nice to have an Arizona Department of Revenue report that confirms it.

Carey wrote “Some states, like Alabama and Indiana, limit tax credit vouchers to low- and middle-income families, or to students who were previously enrolled in public school. But others, including Arizona, do not, subsidizing private education for the well-off.” Two of Arizona’s credits are means tested, and two are not. One of the two that is not means tested exclusively serves children with disabilities. I’ll be for completely means-testing private choice programs the very instant that Kevin gets means-testing passed for district schools. Until such time, let’s note for the record that the Arizona private tax credit programs serve provide far fewer dollars to “well off” kids than say, Scottsdale Unified. Someone please wake me up when the Times runs a breathless expose about rich kids getting exclusive access to fancy and abundantly funded public schools.

In addition to the state taking action, donors apparently expressed their displeasure with what they read about in the East Valley Tribune as well during the next donation cycle (see page 8.) If donors don’t like the way scholarship groups run their business, they have the option of not donating, or donating to other groups. 2010 was a rough year for scholarship groups. Decentralized accountability strikes again!

Reasonable people can disagree about the degree and extent of oversight and other devilish details in a program like this. Even we in the Wild West have to make adjustments on occasion, and the democratic process is ultimately pretty good at hashing these sort of things out. I’ll be happy to make my donation this April to help a low-income parent choose a school for their child.

 


Setting the Record Straight on Educational Choice

March 3, 2017

(Guest Post by Jason Bedrick)

In the last couple weeks, the New York Times has cranked their effort to discredit educational choice policies up to 11. The formula is simple: downplay the positive findings from all the previous gold-standard research and focus instead on more recent studies from Louisiana, Ohio, and Indiana — two of which are not random-assignment studies and one of which hasn’t even been released (not to mention the likelihood that overregulation is hampering Louisiana’s voucher program). Sadly, this distorted narrative is spreading, but some are pushing back. Yesterday, Paul DiPerna of EdChoice and Neal McCluskey of the Cato Institute each provided essential context for understanding the research on school choice.

In Education Next, DiPerna wrote:

Contrary to recent editorials in some major U.S newspapers, the empirical research on school choice programs is far more positive than not. Summaries of the effects of multiple programs generally show positive effects, as does a meta-analysis of gold-standard experimental research on school choice by Shakeel, Anderson, and Wolf (2016). Participating students usually show modest improvements in reading or math test scores, or both. Annual gains are relatively small but cumulative over time. Graduation and college attendance rates are substantially higher for choice students compared to peers. Programs are almost always associated with improved test scores in affected public schools. They also save money. Those savings can be used to increase per-pupil spending in local school districts. Studies also consistently show that programs increase parent satisfaction, racial integration and civic outcomes.

It’s true that recent studies have reported some initial negative effects on choice students’ test scores. The most sobering come from the rigorous, experimental evaluation of the Louisiana Scholarship Program (LSP). The LSP has a different, much more restrictive regulatory framework for private schools than other choice programs. The negative results in math should be monitored, but it’s important to note that the evaluation is only in its second of seven planned years.

Broad perspective and context are essential. Negative initial findings in one or two locations, based solely on one performance metric, should not halt the creation or expansion of school choice programs in other parts of the country. Generalizing those findings across states is problematic because education is sensitive to state and local cultural, political, governmental and economic conditions. The many places where we have observed significant positive results from choice programs swamp the few where we have seen negative findings. We need to consider the complete research base and not disproportionately emphasize the most recent studies.

McCluskey also turned a gimlet eye on the studies that found negative impacts on test scores:

First, the vast majority of random-assignment studies of private school voucher programs—the “gold-standard” research method that even controls for unobserved factors like parental motivation—have found choice producing equivalent or superior academic results, usually for a fraction of what is spent on public schools. Pointing at three, as we shall see, very limited studies, does not substantially change that track record.

Let’s look at the studies Carey highlighted: one on Louisiana’s voucher program, one on Ohio, and one on Indiana. Make that two studies: Carey cited Indiana findings without providing a link to, or title of, the research, and he did not identify the researchers. The Times did the same in their editorial. Why? Because the Indiana research has not been published. What Carey perhaps drew on was a piece by Mark Dynarski at the Brookings Institution. And what was that based on? Apparently, a 2015 academic conference presentation by R. Joseph Waddington and Mark Berends, who at the time were in the midst of analyzing Indiana’s program and who have not yet published their findings.

Next there is Ohio’s voucher program. The good news is that the research has been published, indeed by the choice-favoring Thomas B. Fordham Institute. And it does indicate that what the researchers were able to study revealed a negative effect on standardized tests. But Carey omitted two important aspects of the study. One, it found that choice had a modestly positive effect on public schools, spurring them to improve. Perhaps more important, because the research design was something called “regression discontinuity” it was limited in what it was able to reliably determine. Basically, that design looks at performance clustered around some eligibility cut-off—in this case, public schools that just made or missed the performance level below which students became eligible for vouchers—so the analysis could not tell us about a whole lot of kids. Wrote the researchers: “We can only identify with relative confidence the estimated effects…for those students who had been attending the highest-performing EdChoice-eligible public schools and not those who would have been attending lower-performing public schools.”

That is a big limit.

Finally, we come to the Louisiana study, which was random-assignment. Frankly, its negative findings are not new information. The report came out over a year ago, and we at Cato have written and talked about it extensively. And there are huge caveats to the findings, including that the program’s heavy regulations—e.g., participating schools must give state tests to voucher recipients and become part of a state accountability system—likely encouraged many of the better private schools to stay out. There are also competing private choice programs in the Pelican State. In addition, the rules requiring participating private schools to administer state tests are new, and there is a good chance that participating institutions were still transitioning. Indeed, as Carey noted, the study showed private school outcomes improving from the first year to the second. That could well indicate that the schools are adjusting to the change. And as in Ohio, there was evidence that the program spurred some improvements in public schools.

Both blog posts are worth reading in full, but the main point is this: the research literature is generally positive. The few negative findings are disconcerting and should cause education reformers to think critically about policy design, but the literature still generally finds that students exercising school choice tend to perform as well or better than their district school peers, they’re more likely to graduate high school and enroll in college, they’re less likely to be involved in crime, and all these positive effects come at a much lower cost per pupil to the taxpayer. Additionally, the overwhelming majority of studies find that choice programs have a modest but statistically significant positive effect of the performance of district schools.

Educational choice remains a win-win solution.

 


Keep School Choice in the States

March 2, 2017

img_1358

If D.C. wants to push school choice, it should make the D.C. voucher universal. Let D.C. clean its own long-neglected house and leave the states in charge of their own education systems.

As I told the New York Times in a story that ran today:

Greg Forster, a fellow at EdChoice, a research and advocacy organization that promotes school choice options, said that while he welcomes more support for the idea of school choice, he wants the issue to remain a state responsibility. “We have achieved a lot of victories at the state level by building bridges,” Mr. Forster said. Having Mr. Trump as an advocate “is a bigger problem for the school choice movement than it is a blessing, in my book,” he said.

He added that there is “no need for a federal push for school choice” because the options are increasingly gaining ground, leading to 61 private school choice programs in 30 states and the District of Columbia.

See also.


Why Arizona Charter Leaders Should Feel Confident About the 2017 NAEP

March 2, 2017

azmeritnaep

(Guest Post by Matthew Ladner)

So the 2017 NAEP is underway as we speak. As the NAEP released all six 2015 exams a cornucopia of good news spilled out for Arizona as a whole and (especially) for Arizona charters. Statewide Arizona has been leading the nation in 4th to 8th grade cohort gains since 2009. That’s a pretty good measure of overall school quality for those grades, as the demographic profile of the cohort isn’t likely to change much for a cohort between (for instance) the time when they were 4th graders in 2011 to when they were 8th graders in 2015. Kids will come and go of course, but absent a DC level of gentrification the academic ability of those coming and going should not skew heavily in a particular direction. You also have a measured amount of sampling error in both the 4th and 8th grade measures, but these are quite modest.

Arizona’s charter schools rocked all 2015 NAEP exams in the fashion of a New England state. This is quite impressive given the very modest level of per-pupil spending in Arizona charters, the majority minority student population etc. Sampling error is a bigger issue with regards to charter schools, but as you can see in the figure above, the state’s AZMerit exam tells us a story very similar to NAEP, and does not involve sampling. AZMerit in short provides backup to the findings in NAEP.

The figure above should lend confidence to Arizona charter leaders that they are likely to rock the NAEP again in 2017. The figure shows 4th and 8th grade NAEP scores from 2015 (proficiency rates) and then the proficiency rates for districts and charters from 2016 on AZMerit. These tests are not scaled exactly the same so you would not expect them to match up exactly, but the similarity in the pattern provides confidence that 2015 was not a fluke. Some of the differences between Arizona charters and districts can be attributed to differences in student demographics, although not nearly so much as sometimes are imagined, but New Hampshire doesn’t have any excuses for losing out to Arizona charter schools.

On the final set of columns on the right, for perspective, Arizona charters scored just a smidge below the highest performing state (MA) while Arizona districts would be right around the national average. Drawing random samples of students leaves room for goofy variation in subgroup scores, but there isn’t any reason to believe such goofiness would skew the same way in six different samples unless one wants to believe the NAEP is stacking the deck for charter schools in Arizona for some nefarious reason. While they were at it, the same conspirators would have to have infiltrated the AZMerit as well.

Absent goofiness and based on what we see in the AZMerit, I’m expecting Arizona charter students to CeleNAEP Good Times again in 2017. The statewide trend will be of even greater importance. Let’s see what happens next.

 


Jason Riley: The Next Step is Education Savings Accounts

March 1, 2017

 

(Guest Post by Matthew Ladner)

Jason Riley weighs in on ESAs, federalism and parental choice in his Wall Street Journal column today:

After years of federal overreach through No Child Left Behind, Common Core and Obama administration “guidance” on lavatory usage, the states—where Republicans now occupy 33 of the 50 governors’ offices—are not only reasserting local control of K-12 education but reimagining it.

In addition to this charter progress, education reformers see prospects for more private school choice in the form of education savings accounts, or ESAs, which they describe as the next step in school choice. Under an ESA system, money that would otherwise go to funding a child’s public-school education is instead placed into a restricted-use bank account, from which the family can withdraw to spend on a variety of education-related services. Like vouchers, ESAs allow the money to follow the child. But ESAs don’t limit education options in the way that vouchers do. Instead, families can use money in the account for tuition, textbooks, tutoring, test preparation, transportation, Advanced Placement courses, online learning and even college savings accounts.

Jason Bedrick, a policy analyst at the research organization EdChoice, told me that along with allowing families to tailor spending to the education needs of their children, ESAs can control costs. “Moving from a coupon or voucher model to a bank-account model helps guard against tuition inflation like we’ve seen in Pell Grants,” he said. With ESAs, “there’s no price floor. If you’ve got a $5,000 voucher, no school is going to charge less than $5,000. With an ESA, there’s a lot more competition because private schools are not just competing against each other and against public schools but also competing against other sorts of education opportunities.” In other words, a parent with an ESA has the ability to hold both public and private schools accountable.


Spontaneous Order, Foreign Aid and K-12

February 28, 2017

(Guest Post by Matthew Ladner)

Annie Lowrey’s NYT magazine article The Future of Not Working is well worth reading. The article describes a Silicon Valley funded experiment with a universal basic income in Africa. Personally I’m skeptical of the notion that human labor is going to become obsolete, and I am even more skeptical of the idea of a universal basic income when we currently stand tens of trillions short on previously made commitments. Nevertheless, this article is well worth reading to the very end, as it contains a powerful insight. With the benefit of modern cell phone banking account technology, this group has been giving aid in the form of small cash payments instead of whatever the aid organizations happen to want to give out. This allows people to work things out for themselves:

The residents of this village had received money in 2013, and it was visibly better off than the basic-income pilot village. Its clearings were filled with mango plantings, its cows sturdy. A small lake on the outskirts had been lined with nets for catching fish. “Could you imagine sitting in an office in London or New York trying to figure out what this village needs?” Bassin said as he admired a well-fed cow tied up by the lakeside. “It would just be impossible.”

Many popular forms of aid have been shown to work abysmally. PlayPumps — merry-go-round-type contraptions that let children pump water from underground wells as they play — did little to improve access to clean water. Buy-a-cow programs have saddled families with animals inappropriate to their environment. Skills training and microfinance, one 2015 World Bank study found, “have shown little impact on poverty or stability, especially relative to program cost.”

All across the villages of western Kenya, it was clear to me just how much aid money was wasted on unnecessary stuff. The villagers had too many jerrycans and water tanks, because a nongovernmental organization kept bringing them. There was a thriving trade in Toms canvas slip-ons: People received them free from NGO workers and then turned around and sold them in the market centers. And none of the aid groups that had visited the villages managed to help the very poorest families.

The article goes on to explain that cash payments have been abjured in aid programs in the past. It would deprive beneficiaries of the “benevolent guidance” of very well-meaning people, and would also require fewer such people. It however seems entirely obvious that the Kenyan villagers in this article know their own needs much better than the distant would-be do-gooder, and that they are far more capable of making good use of resources. All of this very much brings to mind the Douglas Carswell quote (via Matt Ridley):

The elite gets things wrong, says Douglas Carswell in The End of Politics and the Birth of iDemocracy, ‘because they endlessly seek to govern by design in a world that is best organized spontaneously from below.’ Public policy failures stem from planners excessive faith in deliberate design. ‘They constantly underrate the merits of spontaneous, organic arrangement, and fail to recognize that the best plan is often not to have one.’

Education Savings Accounts anyone?