J.K. Rowling: The Jeb Bush of the NEPC Florida Fantasy

September 13, 2011

(Guest Post by Matthew Ladner)

The Arizona School Boards Association had their annual law conference last week, and had William Mathis from the Think Tank Review Project present on “Are Things as Sunny as They Seem in Florida?”

I went first, and presented charts like this, showing the vast improvement in Florida’s 3rd grade reading scores:

I have repeatedly asked the Think Tank Review Project people to explain why Florida’s 4th Grade NAEP scores continued to rise in 2007 and 2009 even as 3rd grade retention fell substantially. Or for that matter, why their 3rd grade scores have improved so strongly. Dr. Mathis made no attempt to address the issue.

I also presented charts like these:

Now, call me crazy, but when you are the state called “Arizona” in above chart, you might want to make a careful study of what the other state did to get their English Language Learners to read. This phenomenon  of course is not limited to ELL. Another chart I used showed the combined learning gains on all four NAEP tests for children with disabilities for the entire period we have data from all 50 states (2003-2009).

Just in case you are squinting that’s Florida in red with a gain of 69 points and Arizona in green with a decline of two points.

Dr. Mathis proceeded with his presentation unperturbed. He complained about the 3rd grade retention policy without any effort to explain why Florida’s 3rd grade scores had so profoundly improved, and why Florida’s 4th grade NAEP scores continue to increase even as retention rates have significantly declined.

To give Dr. Mathis’ presentation the fairest possible reading, I would say that he was trying to make the following points: that correlation is not causation, and that to use the terminology of Campbell and Stanley, I had not “controlled for history.” That is to say, there could be other possible explanations for Florida’s gains other than the reforms.

Now it is of course the case that correlation can lead us very much astray, and it is the case that “history” has a nasty habit of bedevilling our theories of causality. As I have noted in the past, however, the Florida reforms unfolded in the real world, rather than in a random assignment study. A great many things unfolded all at once. This is called “life” and there is nothing to be done about this but to gather as much data as possible to draw the best informed decisions we can.

Both Chatteriji and Mathis ignored the Education Next piece in which Dan Lips and I examined other possible explanations for Florida’s gains. Huge spending increases (nope), decline in the percentage of low-income or minority students (nope-increases in both), preschool voucher program (nope- students too young to have aged into the NAEP sample) and class size amendment (nope- implemented very slowly, gains already well under way, formal evaluations negative) and retention law (scores continued to rise even as retention fell). This sort of information might be unhelpful if you are simply trying to get the idea in that something other than a set of hated reforms drove the gains.

Mathis however posited other types of “history” and noted other ways that the world had changed after 1998. On his list of other parts of uncontrolled “history” with regards to Florida’s gains were Harry Potter books (kids reading more fiction) and the more widespread availability of personal computers at home.

Sadly, the format of the panel did not provide time for rebuttal. We had two other people on with us, and took questions from the audience. Had there been such time, however, I would have noted that while Arizona may seem backwards to outsiders (Dr. Mathis lives in Colorado) that we do in fact have Harry Potter books and even personal computers in our humble little patch of cactus. In fact, I am rather confident that Harry Potter books and personal computers became increasingly pervasive in all 50 states.

You never know, Harry Potter books could have powerful educational properties that only manifest themselves on massive peninsulas with high rates of humidity and large concentrations of alligators. The children of Arizona, landlocked in an arid climate, and with not much more in the large lizard department than the occasional Gila monster, may have been left behind. I can’t prove that this isn’t the cause after all.

Nevertheless I’m going to stick with my theory that Governor Bush’s success in implementing a varied and comprehensive set of K-12 reforms in 1999 served as the driver for the large increases in academic attainment seen in Florida’s NAEP scores since 1998. Dr. Mathis and his compatriots can continue to play their stategic nihlism game if they wish, ignoring the problems with their arguments and the studies most on point for the subject at hand (like the regression discontinuity studies of Florida’s retention policy).

Until they put forward a plausible explanation for Florida’s gains, I cannot for the life of me find any reason to take them seriously.

8 Comments | research reports | Tagged: , , , | Permalink
Posted by matthewladner

Florida Tax Credit Analysis find Participant Gains

August 31, 2011

(Guest Post by Matthew Ladner)

A careful analysis of test score gains by David Figlio of Northwestern University has found a modest but statistically significant gains for Florida tax credit students. The data in this study are messy, and Dr. Figlio admirably goes about sorting through the various issues in an even-handed fashion.

Figlio employs a regression discontinuity design to analyze the data, and his finding of a small but statistically significant academic gain fits quite comfortably with the larger random assignment literature, which find small year to year gains which accumulate over time.

One of the under-appreciated features of the random assignment literature: the studies usually fall apart after three or four years due to attrition in the control group. Our window into the academic benefits of choice is therefore limited. Figlio’s employment of a different analytical technique provides confirms previous findings, and may (?) open the door to longer term assessment. The challenges with the data described in this paper, however, suggest that it may not be easy.

Money quote from the study, with a definite echo of previous random assignment studies:

These differences, while not large in magnitude, are larger and more statistically significant than in the past year’s results, suggesting that successive cohorts of participating students may be gaining ground over time.

Good discussion of the results over at RedefinED, including a discussion of the baseline results (tax credit students are poorer and less Anglo). Emerson also puts this study in the context (Figlio also found positive public school effects associated with the Step Up for Students program).

So, the Step Up for Students program has now been found to help improve public school results, help improve participant academic gains, generates high levels of parental satisfaction. Sounds like a rock solid justification for expansion to me.

 

7 Comments | research reports, tax-credit scholarships | Tagged: , , , , | Permalink
Posted by matthewladner

Carr Makes It 19-0

August 17, 2011

This finding’s been replicated more often than Picard’s Earl Grey.

(Guest post by Greg Forster)

Still clearing the backlog: I haven’t had a chance yet to tout this new empirical study of Ohio’s EdChoice voucher program, by my old colleague Matt Carr, finding that – guess what, you’ll never believe this – vouchers improve outcomes at public schools!

Building on a large body of previous studies, this makes it nineteen (19) high-quality empirical studies finding school choice improves public schools and zero (0) studies finding it harms public schools.

Interestingly, Carr finds the positive impact is concentrated among the highest and lowest performing students. Since EdChoice is a failing schools voucher, you might expect schools to respond by improving service to those “bubble” students who are near the state proficiency cutoff. However, Carr finds the opposite.

Matt hypothesizes – plausibly enough – that schools are responding by improving services to the students who are most likely to use the voucher to leave. Low-performing students have the most obvious motivation to seek better services, while high-performing students are the most likely to have actively involved parents.

I do have one quibble with the study. Matt writes that his study “provides an analysis of a voucher program that has not yet been rigorously studied for its competitive effects on traditional public schools.”

Oh, really?

2 Comments | research reports, vouchers | Tagged: , , , | Permalink
Posted by Greg Forster

Podcast on Power

August 16, 2011

(Guest post by Greg Forster)

Still catching up on a travel backlog – in case you missed it, here’s my latest pod-type casting module over the inter-net system of tubes via Heartland. Main topic: the delicate balance between truth and power, and how the late unpleasantness shows the dangers to which some education reformers are already succumbing as they displace the minions of the blob as gatekeepers to the center of the conversation. Our old friend Jack Jennings, aka the human torch, also makes an appearance!

9 Comments | instructional reform, politics, research reports, tax-credit scholarships, vouchers | Tagged: , | Permalink
Posted by Greg Forster

NEPC’s attempt at strategic nihilism

August 4, 2011

(Guest Post by Matthew Ladner)

In the American film classic Animal House there is a scene where students smoke marijuana with their English professor, played by Donald Sutherland, and speculate that it could be the case that the molecules in your fingernails each contain a microscopic universe.

You can’t prove that there aren’t microscopic universes in your fingernails, after all, so they might be in there!

A nice post from Mike Petrilli on the Florida NAEP score gains prompted a response from Kevin Welner from NEPC that shows that the spirit of Sutherland’s Professor Dave Jennings is alive and well at the University of Colorado.

Again there is no attempt to address any of the gaping holes in retention theory. These holes include the fact that Florida’s 4th grade reading scores had improved substantially before the retention policy went into effect, and that they have continued to rise even as retention has fallen off substantially, and that they have fallen off substantially because of a very large improvement in 3rd grade scores.

Welner attempts to tiptoe around this by noting that our EdNext article addressing these points were addressed to a previous Walter Haney paper on the subject rather than the NEPC stuff, which is a distinction without much of a difference. The Chatterji paper contains a carbon copy of the Haney analysis. Amazingly, Chatterji dinged Burke and I for not doing a literature review (not the norm in our tribe) and then cites neither the Education Next paper nor Haney’s analysis. At best, she employed a double standard and at worst, she owes Professor Haney an apology.

Welner’s broader project is to attempt to use the causation problem as a shield. We don’t know, after all, exactly what caused Florida’s remarkable learning gains. Florida’s reformers had to implement their reforms in the real world rather than in a petri dish or in an Intention to Treat Random Assignment study. Welner believes that this allows him the opportunity for strategic nihilism:

The truth might be: (a) there are not actual improvements (the current study is too weak to say whether or not there are), (b) there are improvements, and they’re caused by a combination of all these things, (c) there are improvements, and they’re caused by something none of us pointed to (perhaps the green shirts??), or (d) there are improvements, and they’re caused by some of the things we’re pointing to BUT some of the other things we’re pointing to are actually harming students (just not enough harm to overcome the benefits of the other things).

In other words, when it comes to understanding the FL package of reforms, we are flying blind.

Welner is flying blind all right, but it is by choice. Let’s take each of these little gems on one at a time:

A. The NAEP results show very substantial improvements, as do other indicators.

B. I have always held that the exact cause for the improvement is impossible to know, because Florida’s reformers enacted multiple reforms simultaneously. The logical response to this is not to do none of the Florida reforms, but to do all of them.

C. Florida lurked near the bottom on NAEP for many years, enacted reforms in 1999, and then enjoyed sustained gains over time. While it could be the case that some mysterious X-factor caused the improvement, I’ve yet to hear a plausible theory regarding this. Dan Lips and I addressed multiple possibilities in the Education Next article, including demographic change, spending, etc, and found no evidence to support them.

D. This could be the case, but I haven’t seen a single scrap of evidence to suggest that it is actually the case- return to B above.

Welner is of course correct that there is a correlation and causation problem to consider. As a practical matter, there is nothing else to do but to carefully examine the evidence and history and draw the best conclusions that we can. Dan Lips and I did this in the Education Next article. Florida’s reforms coincided with the student population becoming poorer and less Anglo. State lawmakers increased funding per pupil, but it wasn’t by much and is still below the national average. NEPC complains about a lack of mention of the preschool voucher program when those kids have yet to age into the 4th grade NAEP sample. The class size amendment was implemented very slowly, long after Florida’s scores had begun to rise.

If Dr. Welner would like to provide a plausible explanation for why Florida’s NAEP scores increased so much after 1998, I’d be very interested to read it.

If he prefers to attempt to continue to play games, NEPC’s credibility will go on double secret probation.

 

 

 

 

4 Comments | research reports | Tagged: , , | Permalink
Posted by matthewladner

Speaking of Intellectual Corruption!

July 27, 2011

(Guest post by Greg Forster)

We now interrupt this serialized political philosophy dissertation on how power corrupts knowledge systems to bring you something completely unrelated – the latest nonsense from the increasingly irrelevant, lifelong Democratic party hack Jack Jennings!

Talk about weasel words. Studies clearly and conclusively finding modest positive impacts from vouchers magically become studies finding that voucher outcomes are “similar” to outcomes at public schools, which then magically become studies provding no “clear” evidence that vouchers have any impact. Never mind that most studies of vouchers have been conducted at Harvard, Princeton, Cornell, Stanford, the Fed, etc. – they’ve been conducted by “pro-voucher” researchers, which is technically true if you define any researcher who does a study finding positive impacts from vouchers as a pro-voucher researcher. (By the same method, you can prove that all studies finding cigarettes cause cancer were conducted by “anti-smoking researchers.”) Extremely minute scrutiny of studies by think tanks have found no scientific flaws in their work, which proves that even more minute scrutiny is obviously needed. And for some reason, a shift in the political winds away from supporting smaller voucher programs toward supporting larger voucher programs is evidence that vouchers are failing! Right, sure they are.

For the record, the empirical evidence supporting school choice is overwhelming – stronger than the evidence supporting any other reform policy. Impact sizes are sometimes large but usually modest, not surprising given that existing programs are tiny, underfunded and overregulated. But the evidence is consistent that school choice produces benefits even under these disadvantageous conditions.

Oh, and more good news: everyone’s onto Jennings’ game, so no one cares about his nonsense. Mickey Kaus:

At this point if Jack Jennings doused himself in gasoline and set himself ablaze in front of the NEA, would anyone notice?

I have Jennings’ professional reptuation hanging on the wall in my trophy room, right next to my Jay Mathews meal ticket.

4 Comments | research reports, tax-credit scholarships, vouchers | Tagged: , | Permalink
Posted by Greg Forster

MPS Takes “Standing in the Schoolhouse Door” to a Whole New Level

May 31, 2011

(Guest post by Greg Forster)

Over the weekend, John Witte and Pat Wolf had a compelling article in the Milwaukee Journal Sentinel summarizing the real (as opposed to media-reported) results of the Milwaukee voucher program research being conducted by the School Choice Demonstration Project.

And then they dropped a bomb:

Recently, our research team conducted site visits to high schools in Milwaukee to examine any innovative things they are doing to educate disadvantaged children. The private high schools of the choice program graciously opened their doors to us and allowed us full access to their schools. Although several MPS principals urged us to come see their schools as well, the central administration at MPS prohibited us having any further contact with those schools as they considered our request for visits. We have not heard from them in weeks.

Our report on the private schools we visited, which will offer a series of best practices regarding student dropout prevention, will be released this fall. Should MPS choose to open the doors of their high schools to us, we will be able to learn from their approaches as well. [ea]

MPS opposition to vouchers takes standing in the schoolhouse door to a whole new level.

Leave a Comment » | journalism, research reports, vouchers | Tagged: , , , , | Permalink
Posted by Greg Forster

Vouchers Help But Obama Opposes

March 30, 2011

In a new study released today by a team of researchers led by Josh Cowen at the University of Kentucky, we learn that voucher students in Milwaukee are more likely to graduate high school and go to a four year college than their counterparts in the Milwaukee Public Schools.  The report concludes:

MPCP [voucher] students were more likely to have enrolled in a four year college, even after accounting for race, gender and prior achievement.  They were less likely to have dropped out of high school or remained enrolled after four years.  These differences may be partially explained by family background characteristics such as parental education and income.  They do not appear to be related to private school “creamskimming” of students into or out of MPCP between 8th and 9th grade.

Attending a private school with a voucher resulted in about a 7 percentage point improvement in the probability of attending a four year college.  Considering that is a move from about 32% to 39% attending 4 year college, it is a big effect.

And this attainment benefit is consistent with the findings of the U.S. Department of Education’s official evaluation of the DC Opportunity Scholarship voucher program, led by my colleague Patrick Wolf, which found:

The offer of an OSP scholarship raised students’ probability of completing high school by 12 percentage points overall (figure ES-3). The graduation rate based on parent-provided information was 82 percent for the treatment group compared to 70 percent for the control group. The offer of a scholarship improved the graduation prospects by 13 percentage points for the high priority group of students from schools designated SINI in 2003-05 (79 percent graduation rate for the treatment group versus 66 percent for the control group).

Despite these positive results, the Obama Administration issued a statement opposing the continuation and expansion of the DC voucher program, on which the U.S. House is scheduled to vote today.  They boldly (and falsely) declared:

Private school vouchers are not an effective way to improve student achievement. The Administration strongly opposes expanding the D.C. Opportunity Scholarship Program and opening it to new students.  Rigorous evaluation over several years demonstrates that the D.C. program has not yielded improved student achievement by its scholarship recipients compared to other students in D.C.

Given the lack of intellectual honesty on the part of the Obama Administration in declaring that vouchers have no benefits for students even after rigorous research (including the official evaluation they released!) finds otherwise, confirms the danger of entrusting any additional authority over eduction policy in the national government.  They will lie, cheat, and crush their opponents, so why would we want to give these folks control over a nationalized set of standards, curriculum, and testing.

6 Comments | politics, research reports, vouchers | Permalink
Posted by Jay P. Greene

Fear the Win-Win!

March 25, 2011

(Guest post by Greg Forster)

At the beginning of a very kind column praising my new report on the empirical evidence on vouchers, Jay Mathews indicates that for some strange reason, he’s afraid of me and my school-choice posse:

Do I really want to get beaten bloody again by school vouchers devotees?

Come on, Jay. I’m not a dangerous man. I would never beat anyone bloody. I’m soft and harmless. I’m a perfectly ordinary bunny rabbit. A cute, fluffy, harmless bunny rabbit.

Well, okay, I have been known to bite. With big, sharp, pointy teeth. But just to stretch my repertoire, I’ll take the soft approach this time.

Jay acknowledges the evidence:

Greg Forster, a talented and often engagingly contrarian senior fellow at the Foundation for Educational Choice, has expanded a previous study to show that nearly all the research on vouchers, including some using the gold standard of random assignment, has good news for those who believe in giving parents funds that can be used to put their children in private schools. Students given that chance do better in private schools than similar students do in public schools, the research shows. Public schools who are threatened by the loss of students to private schools because of voucher programs improve more than schools that do not have to worry about that competition, the research also shows.

Yet he thinks we shouldn’t support vouchers because . . . well, I’ll let him explain:

I see nothing morally, economically or politically wrong with vouchers. I have never thought that they drained public schools of vital resources. I think a low-income family that gets the chance to choose a private school that suits their child should do so.

But I think such programs have limited growth potential because there are never going to be nearly enough empty spaces in private schools to help all the students who need them. Forster and other voucher advocates say this will change when voucher programs become universal. Then, entrepreneurs will be able to convince investors that they can create a new generation of private schools with the new wave of voucher students.

I think they are wrong about that. The young educators who have led the robust growth of charters prefer to work in public schools. Many voters will continue to resist sending their tax dollars to private schools, particularly with the pressures to cut back government spending that are likely to be with us for many years.

So that’s two arguments. Entrepreneurial startups won’t attract talented education refomers, and voters won’t support the programs.

It’s true that the leading-edge school reformers, the people Matt calls “the cool kids,” prefer to work in public schools. As I’ve written before, you can already see how that strategic choice is leading to dead end after dead end. The school choice movement needs to start building bridges to these people and showing them that in the long run, only school choice can provide the institutional support they need to sustain the kind of reforms they want.

As for politics, school choice has always polled well (for a discussion of the research and methodological issues, see here). The American people are not, in fact, uncomfortable with allowing religious institutions to participate in publicly funded programs on equal terms alongside other institutions. There was a time when they were (see “amendments, Blaine”) but that bigotry has receded.

Oh, and as for pressure to cut spending, school choice saves money. Tons and tons of it. That has always been one of our biggest assets in the political fight – that’s why the Foundation for Educational Choice produces state-focused fiscal studies year after year, to show each state how school choice would save taxpayer money while delivering better education.

The political obstacle to choice has never been the public at large. It has always been the blob, with its huge piles of cash fleeced indirectly from taxpayers, and (perhaps more important) its phalanx of highly disciplined volunteers and voters. A minority of the voters can control the outcome if they are single-issue voters when the rest of the public takes into account the whole panoply of problems confronting the body politic. And when you threaten to derail a gravy train, it tends to make the passengers into single-issue voters.

But the tide is changing. The cynical selfishness of the blob is more and more visible to more and more people. Reform has already won the war of ideas. That does not mean the ground war is won. The unions are still big, rich, and powerful. But they are no longer sacred. They have lost their mystique. No one thinks the unions speak for kids anymore; no one even thinks the unions speak for teachers anymore. And in the end, that’s what counts.

As Jay has put it, the unions are now the tobacco lobby. Or, as I have put it, they’re Bull Connor. That’s why school choice is now poised for a series of big political wins.

Jay is skeptical – pointing to the greater success of charters, he thinks vouchers won’t make big gains this cycle. As readers of JPGB know, the answer to the charter argument is that vouchers make the world safe for charters. As for whether vouchers make big gains this year, we’re about to find out.

Tell you what, Jay. Let’s make a bet. You say there won’t be “a wave of pro-voucher votes across the country.” Me and my posse at FEC will go back and count up the number of school choice bills (private choice, not charters) that passed state chambers in 2008-2010. Then we’ll set a mutually agreed on bar for the number of voucher bills passing chambers this year. If we hit the bar, you have to buy me dinner at a Milwaukee restaurant of my choice. But if we don’t hit the bar, I buy you dinner at a DC restaurant of your choice. That’s pretty lopsided in your favor, dollar-wise. How about it?

HT

11 Comments | competitive effects, instructional reform, politics, research reports, vouchers | Tagged: , , | Permalink
Posted by Greg Forster

Vouchers Are a Win-Win Solution – Updated Edition

March 23, 2011

(Guest post by Greg Forster)

The Foundation for Educational Choice has just released my new report, “A Win-Win Solution: The Empirical Evidence on School Vouchers.” It’s an updated edition of my 2009 report providing a comprehensive overview of, well, the empirical evidence on school vouchers. In addition to incorporating new studies, this edition also expands the scope. The first edition only looked at the evidence on how vouchers impact public schools; this new edition also includes how vouchers impact the students who use them.

As before, I summarize the research with one striking chart. Or, now, two charts – one on how vouchers impact participants, and the other on how they impact public schools:

 

(Note: This image has been corrected; an earlier version transposed Milwaukee and Florida in Table 2. I apologize for the error.)

I’m not sure I can improve on what I wrote here on JPGB when I posted the first edition two years ago:

Worth a thousand words, isn’t it? I mean, at what point are we allowed to say that people are either lying, or have been hoodwinked by other people’s lies, when they say that the research doesn’t support a positive impact from vouchers on public schools?

There’s always room for more research. What would we all do with our time if there weren’t? But on the question of what the research we now have says, the verdict is not in dispute.

The report surveys all the random-assignment research on participants, and all the research (using all methods) on public school impacts. Readers of JPGB are probably familiar with the reason for this difference: random assignment is so far superior to other methods that when a large body of random assignment research exists, it ought to be given priority. However, since it’s not possible to do random-assignment research on how vouchers impact public schools, we have to cast a wider net – and the research methods being used in this field have been improving over time. Yet the results have remained consistent – how about that?

Here’s the executive summary of the new report:

This report collects the results of all available empirical studies using the best available scientific methods to measure how school vouchers affect academic outcomes for participants, and all available studies on how vouchers affect outcomes in public schools. Contrary to the widespread claim that vouchers do not benefit participants and hurt public schools, the empirical evidence consistently shows that vouchers improve outcomes for both participants and public schools. In addition to helping the participants by giving them more options, there are a variety of explanations for why vouchers might improve public schools as well. The most important is that competition from vouchers introduces healthy incentives for public schools to improve.

Key findings include:

  • Ten empirical studies have used random assignment, the gold standard of social science, to examine how vouchers affect participants. Nine studies find that vouchers improve student outcomes, six that all students benefit and three that some benefit and some are not affected. One study finds no visible impact. None of these studies finds a negative impact.
  • Nineteen empirical studies have examined how vouchers affect outcomes in public schools. Of these studies, 18 find that vouchers improved public schools and one finds no visible impact. No empirical studies find that vouchers harm public schools.
  • Every empirical study ever conducted in Milwaukee, Florida, Ohio, Texas, Maine and Vermont finds that voucher programs in those places improved public schools.
  • Only one study, conducted in Washington D.C., found no visible impact from vouchers. This is not surprising, since the D.C. voucher program is the only one designed to shield public schools from the impact of competition. Thus, the D.C. study does not detract from the research consensus in favor of a positive effect from voucher competition.
  • The benefits provided by existing voucher programs are sometimes large, but are usually more modest in size. This is not surprising since the programs themselves are modest — curtailed by strict limits on the students they can serve, the resources they provide, and the freedom to innovate. Only a universal voucher program could deliver the kind of dramatic improvement our public schools so desperately need.

6 Comments | competitive effects, research reports, vouchers | Tagged: , | Permalink
Posted by Greg Forster

« Previous Entries
Next Entries »