No Jack Jennings Is Not on Fire

July 29, 2009

No two people are not on fire

(Guest post by Greg Forster)

Stop the press! How did I miss this on Eduwonk last week?

At this point if Jack Jennings doused himself in gasoline and set himself ablaze in front of the NEA, would anyone notice?

Hey, that’s what happens when you spend too long peddling political hackery trumped up as research. Sooner or later, people get wise to the con and stop taking you seriously.

Of course, Andy feels the need to call Jennings’ work “important.” But if all the empty, generic words of praise people rotely intone about Jennings doused themselves in gasoline and set themselves ablaze in front of the Merriam-Webster publishing comany, would anyone notice?

In other Eduwonk news, give Andy credit for not drinking too much of the yesterday’s new Race to the Top flavor Kool-Aid; he linked to this item, which helps illustrate just how deep the kabuki goes.

3 Comments | politics | Tagged: , , , , , | Permalink
Posted by Greg Forster

Ed Sector’s K-12 Incoherence Week

May 15, 2009

(Guest Post by Matthew Ladner)

I’ve been out and about this week, but our pals over at Education Sector have kept me entertained. On Carey’s too-cool for-school dissing of vouchers, I can’t help but wonder: what would stop an advocate of home-schooling from dismissing Kevin’s beloved charter schools on a broadly similar basis? After all, home-schooling is legal in all 50 states, charters in only 40. Many of those 40 laws, however, are dogs that will never produce more than a rounding error number of schools. They’re not bad, they’re just drawn that way:

Precise numbers are not available, but twice as many or more students may be home-schooling than attending charter schools, and the rate of growth has been faster. High quality outcome data is hard to come by, but anecdotally universities have come to view home-school students very positively. I haven’t seen the same said for charter schools yet, nor have we seen (yet) a charter school student crush the evil Sooners like the bugs they are after winning the Heisman Trophy.

Im way too cool for you Carey, google my girlfriend

I'm too cool for you Carey, google my girlfriend...

Does it follow then that home-school supporters should be completely dismissive of charter schools? No of course not. The truth is that we don’t know what is going to take hold in K-12 reform, only that it is going to change.

Meanwhile, Andy Rotherham has delivered a brilliant column on the limitations of transparency that all but screams out at the end for a decentralized, self-regulating mechanism to hold schools accountable for results.

Ummm….you know….like parental choice.

3 Comments | vouchers | Tagged: , , , | Permalink
Posted by matthewladner

Andy’s Just Plain Wrong

March 5, 2009

Andy Rotherham is a great guy.  And he’s often right.  But I’m afraid that on vouchers he’s just plain wrong.

Andy responded to my post, which was a response to an earlier post he wrote on vouchers.  Let me just run through his arguments:

First, Andy wants to argue that vouchers have stalled politically.  I pointed out that there are now 24 voucher or tax-credit programs in 15 states serving more than 100,000 students.  And two new programs were adopted last year and a third significantly expanded. 

No fair, Andy cries, including tax-credit programs “creates a false sense of scale for intentional choice plans.”  What’s false about counting tax credit programs, like the one in Florida which functions as the largest voucher program in the country?  The program gives vouchers — excuse me — “scholarships” to students from organizations that are funded with dollar for dollar tax credit donations from corporations.  This is virtually identical in financing and effect as the state simply giving vouchers to students.  The only difference is that the tax credit is treated better by the courts (don’t ask why) because the money never enters the state treasury before going right back out the door as a voucher.

But let’s say we grant Andy his odd position that tax-credit programs don’t count.  We still have 13 voucher programs in 10 states serving about 50,000 students.  And the two new programs adopted last year were both voucher programs.  Wish as he might, Andy still can’t show that vouchers have stalled politically.

Second, Andy rightly says, “Reasonable people can review the cumulative literature about choice plans and disagree on how substantively significant or transformative these effects are (or could be at scale) and what that means for vouchers as a policy. ”  While reasonable folks could disagree about the magnitude of the effect of expanded school choice on public school performance, no reasonable person could disagree with the observation that the research literature supports at least some positive impact.  Given how hard it is to find any policy intervention that raises student achievement, consistently finding a positive impact from the systemic effect of vouchers should be treated as a big deal.  It isn’t to Andy. 

Third, Andy concedes that the more frightening prospect of vouchers helped spread charters, at least in the early stages of the charter movement.  But now that charters have reached critical mass, they may well do just fine without the viable threat of new and expanded voucher programs.  Folks who are really sincere about charters shouldn’t get so comfortable.  Just look at the unionization of the KIPP charter in NY or the constant effort to regulate charters to death in many states.  Dropping vouchers from your arsenal would be like confronting a resurgent Russia after dismantling all of your nuclear weapons.  You may think your conventional forces are up to the task, but ask the Poles how they would feel about it.

I’ve never understood why people would support charters but oppose vouchers.  The theory that expanded choice is good for the participating student and helps spur improvement in traditional public schools is required for both reforms.  Yes, charters are more easily subject to regulation than private schools receiving vouchers, but healthy charter programs require light regulation and states have not been shy about applying similar light regulation to voucher programs. 

The only reason I can imagine that folks would support charters but oppose vouchers is for political gain since the theory and evidence for both are essentially the same.  And I understand why politicians invent these false distinctions to prove their moderation and good sense by opposing the one they artificially dub as radical.  But we aren’t politicians.  We don’t have to lie or invent false distinctions to please constituencies.  Universities, think tanks, and the blogosphere should be refuges for reasoned inquiry and dispute, not rhetoric for political advantage.  As it says on the great seal — Veritas.

UPDATE — Andy’s a nice guy.  I tried to make my post as hyperbolic as possible and he responds kindly and reasonably.  Damn, he’s good.

UPDATE TO UPDATE — Just to be clear, I still think Andy is just plain wrong.  The fig leaf that Andy uses to be pro-charter while anti-voucher is the concern that vouchers sever “the connection between avenues of democratic input into schooling decisions and those decisions.   In other words, for some people the issue isn’t choice, rather it’s accountability in a broad sense.”  The reality is that there is as much opportunity for democrat input in the  design and operation of voucher programs as charter schools or traditional public schools for that matter.  The public can place whatever regulations it deems necessary on voucher schools as a condition of receiving those funds, just as it does with charter and traditional public schools.  Of course, all of these systems would operate best with minimal regulation.  If regulation were the answer to school effectiveness our public schools would already be fantastic.

10 Comments | charter schools, competitive effects, vouchers | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Beltway Confusion

March 3, 2009

(Beltway edu-analysts discuss the world over brandy and cigars.  Note where they are headed.)

I feel sorry for my education colleagues within the DC Beltway.  I don’t know if it’s all those wine and cheese receptions or box lunch lectures that addle their brains, but they are clearly confused.  They confuse political analysis for research.  And they confuse their political preferences for political analysis. 

Look, for example, at the recent post by my friend Andy Rotherham at Eduwonk on vouchers, which states:

“Now, paradoxically, the school choice experience since the early 1990s has lessened the allure of vouchers as a scalable education reform but at the same time made these smaller “pilot” type initiatives like the one in D.C. seem less toxic and more harmless among an increasing number of players.   Opponents don’t even really have a slippery slope to point to in any of the early adopter sites for vouchers.  There’s not one in D.C.  There it’s the public charters not the vouchers that are taking over and not in the other cities/states, even Milwaukee, where vouchers have been tried and the effects have been modest.  In other words, vouchers are not destroying the public schools.  Rather, systemically, they’re not really doing much of anything at all.”

Andy concludes that “systemically” vouchers aren’t “much of anything at all” because they aren’t expanding very rapidly.  This is a political analysis on the appeal of larger voucher programs, not a summary of research on the effects of vouchers on public school achievement.  If Andy had wanted to talk about the research on the systemic effects of vouchers he would have referenced this literature, which clearly shows that expanding choice and competition through vouchers improves public school performance.  So, Andy substitutes political analysis for research.

But he also substitutes his political preferences for political analysis because he ignores the steady growth in vouchers over the last two decades.  There are now 24 voucher or tax-credit programs in 15 states serving over 100,000 students.  Just last year two new programs were adopted, in Georgia and Louisiana, and the tax credit program in Florida was significantly expanded.  Andy may wish the voucher movement to be stalled, but a clear political analysis would reveal that vouchers continue to move forward.

Now, it’s true that there have been setbacks in the voucher movement.  And it’s true that each new program encounters a blizzard of opposition, making each step forward seem inordinately difficult.  But vouchers are just the spearhead of a broader choice movement that includes the more rapidly expanding charter movement.  If not for the viability of voucher programs, charters would have been the target of this onslaught of opposition. 

Vouchers have made the world safe for charters.  And the moment that vouchers really do stall, the enemies of school choice will redirect their fire at charters, strangling them with regulation and repealing charter gains.  To say that vouchers haven’t really done much of anything politically because charters are really where the action is ignores how much charters owe their political strength to the credible threat of new and expanded voucher programs.

It may be fashionable at Beltway receptions to dismiss vouchers as everyone is eager to be seen as championing the latest DC fad proposal.  But real analysis of research and politics show that the expansion of school choice, led by vouchers, will have a greater impact on education reform than building new school buildings, expanding pre-school, adopting a 21st century curriculum or whatever folks there are now talking about.

(edited for typos)

2 Comments | charter schools, competitive effects, vouchers | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Educating Journalists about Education Science

July 16, 2008

(Guest post by Greg Forster)

Don’t worry, this post is definitely not a continuation of the recent big dustup about 1) whether it’s naughty for scholars to provide journalists with accurate information about their work; and 2) whether it’s naughty for anonymous bloggers to argue that scholars’ motives are relelvant to their credibility, but bloggers’ motives aren’t relevant to theirs (which reminds me of Pat Moynihan’s quip about the Supreme Court cases, since overturned, holding that government can’t subsidize private school books but can subsidize classroom equipment such as maps; Moynihan asked, “What about atlases?” – books of maps? What about scholars who are bloggers? Or bloggers who write about scholarly studies? Once you start legitimizing ad hominem arguments, where do you stop?).

But I would like to expand on a comment that Eduwonk made during said dustup, which deserves more attention and has significance well beyond the issues that were at stake in that squabble. The comment got lost in the exchange because it was somewhat tangential to the main points of contention.

He wrote:

Not infrequently newspapers get snookered on research and most consumers of this information lack the technical skills to evaluate much of the work for themselves.   As education research has become more quantitative — a good thing — it’s also become less accessible and there is, I’d argue, more an asymmetry to the information market out there than a fully functioning marketplace of ideas right now.  In terms of remedies there is no substitute for smart consumption of information and research, but we’re not there yet as a field.

We are living in the first golden age of education research, brought on by the advent of systematic data collection, which every other field of human endeavor began undertaking a long time ago but which education is only getting around to now because it has been shielded from pressure to improve thanks to its protected government monopoly. Given the explosion of new information that’s becoming available, educating journalists about quantitative research is a huge problem. Jay is right that there is a marketplace of ideas. There really can’t help but be one; the idea some people seem to have that we can forbid people who own information from spreading it around as much as they want is silly. But just because there’s a market doesn’t mean there’s a perfect market, and Eduwonk is right that markets require informed consumers to function well. The current state of methodological ignorance among journalists does hinder the market of ideas from functioning as well as it should. (I’ll bet Jay would agree.)

As it happens, the same subject came up this morning in a completely different context, as my co-workers and I struggled to figure out the best way to present the findings of an empirical study we’re coming out with so that journalists will be able to follow them. And I wasn’t there, but I hear this topic also came up at a bloggers’ panel at the recent conference of the Education Writers’ Association.

Here at the Friedman Foundation, this has been a topic of great importance to us for some time, since exposing the bad and even bogus research that’s used to justify the status quo is one of our perennial challenges. We took a stab at composing a journalist’s guide to research methods. It went over well when we first distributed it (at last year’s EWA, if memory serves). But it’s necessarily very basic stuff.

Eduwonk is also right about journalists having been snookered by lousy research, and I think that has had both good and bad effects. The good news is that I’ve noticed a clear trend toward greater care in reporting the results of studies (not at propaganda factories like the New York Times, of course, but at serious newspapers). In particular, we’re seeing journalists talk about studies in the context of previous studies that have looked at the same question. Of course, we have a long way to go. But we’re on the way up.

On the bad side, however, I have also noticed a greater reluctance to cover studies at all. Part of that is no doubt due to the increase in volume. I’m young, but even I can remember the heady days of 2003 when any serious empirical study on the effects of a controversial education policy (vouchers, charters, high-stakes testing) would get at least some coverage. Now it’s different, and (to echo Eduwonk) that’s a good thing. But I think it’s extremely unlikely that this is the only factor at work. Junk science has poisoned the well for serious research. No doubt that was part of its intended purpose (although of course the motives of those who produce it have no relevance to its scientific merts or lack thereof).

My hope is that journalists will soon realize they’re getting left behind if they don’t learn how to cover the research accurately. Their job is to go where the news is. If the news is in quantitative research – and that is in fact where a lot of it is – they’ll have to learn how to get there.

Also, the changing media landscape will help. The old idea that journalists must be neutral stenographers with Olympian detachment from all the issues they cover is an artifact of the mid-20th-century role of the media as oligarchic gatekeeper, and is rapidly dying out. As “news” increasingly includes coverage by people who are actively engaged in a field, even as advocates, we can expect the news to be increasingly provided by people with greater amounts of specialized knowledge. (By the way, the old idea of the scholar as detached Olympian stenographer is equally an artifact of vanished circumstances, and will probably be the next thing to go; see the Our Challenge to You statement on the inside cover of any empirical study published by the Friedman Foundation for our views on the relationship between advocacy and scholarship.)

An optimistic view, yes – but since my optimism on other subjects has been triumphantly vindicated over the past year, even when the conventional wisdom said to head for the hills, I think I’ll let it ride.

Leave a Comment » | journalism, methodology | Tagged: , , | Permalink
Posted by Greg Forster

Eduresponses to Edubloggers

July 10, 2008

My recent posts on the release of our new study on the effects of high-stakes testing in Florida and posts here and here on the appropriateness of releasing it before it has appeared in a scholarly journal, have produced a number of reactions.  Let me briefly note and respond to some of those reactions.

First, Eduwonkette, who started this all, has oddly not responded.  This is strange because I caught her in a glaring contradiction: she asserts that the credibility of the source of information is an important part of assessing the truth of a claim yet her anonymity prevents everyone from assessing her credibility.  I prefer that she resolve this contradiction by agreeing with my earlier defense of her anonymity that the truth of a claim is not dependent on who makes it.  But she has to resolve this one way or another — either she ends her anonymity or she drops the argument that we should assess the source when determining truth.

But apparently she doesn’t have to do anything.  Whose reputation suffers if she refuses to be consistent?  Her anonymity is producing just the sort of irresponsibility that Andy Rotherham warned about in the NY Sun and that I acknowledged even as I defended her.  The only reputation that is getting soiled is that of Education Week for agreeing to host her blog anonymously.  If she doesn’t resolve her double-standard by either revising her argument or dropping her anonymity, Education Week should stop hosting her.  They shouldn’t lend their reputation to someone who will tarnish it.

Mike Petrilli over at Flypaper praises our new study on high stakes testing but takes issue with referencing comments by Chester Finn and Diane Ravitch about how high stakes is narrowing the curriculum in the “pre-release spin.”  I agree with him that this study is not “the last word on the ‘narrowing of the curriculum.’”  But to the extent that it shows that another part of the curriculum (science) benefits when stakes are applied only to math and reading, it alleviates the concerns Checker and Diane have expressed. 

As we fully acknowledge in the study, we don’t have evidence on what happens to history, art, or other parts of the curriculum.  And we only have evidence from Florida, so we don’t know if there are different effects in other states.  But the evidence that high stakes in math and reading contribute to learning in science should make us less convinced that all low stakes subjects are harmed.  Perhaps school-wide reforms that flow from high stakes in math and reading produce improvements across the curriculum.  Perhaps improved basic skills in literacy and numeracy have spill-over benefits in history, art, and everything else as students can more effectively read their art texts and analyze data in history.

Andy Rotherham at Eduwonk laments that what I describe as our “caveat emptor market of ideas” doesn’t work very well.  I agree with him that people make plenty of mistakes.  But I also agree with him that “in terms of remedies there is no substitute for smart consumption of information and research…”  There is no Truth Committee that will figure everything out for us.  And any process of reviewing claims before release will make its own errors and will come at some expense of delay.  Think Tank West has added some useful points on this issue.

Sherman Dorn, who rarely has a kind word for me, says: “Jay Greene (one of the Manhattan Institute report’s authors and a key part of the think tank’s stable of writers) replied with probably the best argument against eduwonkette(or any blogger) in favor of using PR firms for unvetted research: as with blogs, publicizing unvetted reports involves a tradeoff between review and publishing speed, a tradeoff that reporters and other readers are aware of.”  He goes on to have a very lengthy discussion of the issue, but I was hypnotized by his rare praise, so I haven’t yet had a chance to take in everything else he said.

2 Comments | journalism, research reports, school accountability | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

Eduwonkette and Eduwonk Aren’t Edumarried?

July 8, 2008

The New York Sun had a nice profile yesterday of Eduwonkette.  Well, it’s not exactly a profile because Eduwonkette writes anonymously.  In the article some folks complain that her anonymity is a problem: “A co-director of the Education Sector think tank, Andrew Rotherham, suggested on his blog Eduwonk that Eduwonkette might be unfairly pretending to be unbiased because she has ‘skin in the game… It’s this issue of you got all this information to readers, without a vital piece of information for them to put it in context.'”

I think Andy’s mistaken on this. (Did they have some kind of edu-break-up?)  The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.  I wish we all spent a whole lot less time analyzing people’s motives and a whole lot more time on their evidence and arguments. 

The only major problem with anonymity is lack of responsibility for being wrong.  There is a reputational price for making bad arguments or getting the evidence wrong that Eduwonkette avoids paying professionally — although she does pay a reputational price to the name brand of Eduwonkette.

Speaking of being wrong, Eduwonkette knocks the study Marcus Winters, Julie Trivitt, and I released today through the Manhattan Institute.  She complains: “It may be an elegantly executed study, or it may be a terrible study. The trouble is that based on the embargoed version released to the press, on which many a news article will appear today, it’s impossible to tell. There is a technical appendix, but that wasn’t provided up front to the press with the glossy embargoed study. Though the embargo has been lifted now and the report is publicly available, the technical appendix is not.”

This isn’t correct.  Embargoed copies of the study were provided to reporters upon their request.  If they requested the technical report, they could get that.  Both were available well in advance to reporters so that they could take time to read it and circulate it to other experts before writing a story.  Both the study and the technical report were made publicly available today (although there seems to be a glitch with the link to the technical report that should be fixed within hours).  The technical report can be found here.

And while we are on the subject of Eduwonkette being wrong, her attacks on test-based promotion policies are overdone.  The Jacob and Lefgren paper does raise concerns, but there is more positive evidence from the experience in Florida.  As I wrote in a previous post: “In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.”

Our results may actually be consistent with what Jacob and Lefgren find.  We find academic benefits for students retained in third grade.  They find: “that grade retention leads to a modest increase in the probability of dropping out for older students, but has no significant effect on younger students.”  It could be that test-based promotion is more beneficial when done with younger students.  It could also be that the policy has positive effects on achievement with some cost to graduation. 

And particularly severe problems with the integrity of test results used for promotion decisions in Chicago may limit the ability to generalize from Chicago’s experience.  In Chicago it may have been easier to move retained students forward by cheating on the next test than actually teaching them the basic skills they need to succeed in the next grade.

Besides, I’m sure that Edwuonkette wouldn’t put too much stock in Jacob and Lefgren’s non-peer-reviewed paper released straight to the public.  I’m sure she would be consistent in her view that: “By the time the study’s main findings already have been widely disseminated, some sucker with expertise in regression discontinuity may find a mistake while combing through that appendix, one that could alter the results of the study. But the news cycle will have moved on by then. Good luck interesting a reporter in that story… So as much as I like to kvetch about peer review and the pain and suffering it inflicts, it makes educational research better. It catches many problems and errors before studies go prime time, even if it doesn’t always work perfectly.”  

Or do these standards only apply to studies whose findings she doesn’t like?   If Eduwonkette isn’t careful she might get a reputation.

6 Comments | social promotion | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

%d bloggers like this: