Eduwonkette Apologizes

July 8, 2008

I appreciate Eduwonkette’s apology posted on her blog and in a personal email to me.  It is a danger inherent in the rapid-fire nature of blogging that people will write things more strongly and more sweeping than they might upon further reflection.  I’ve already done this on a number of occasions in only a few months of blogging, so I am completely sympathetic and un-offended.

One could argue that these errors demonstrate why people shouldn’t write or read blogs.  In fact some people have argued that ideas need a process of review and editing before they should be shown to the public.  These people tend to be ink-stained employees of “dead-tree” industries or academia, but they have a point: there are costs to making information available to people faster and more easily.

Despite these costs the ranks of bloggers and web-readers have swelled.  There are even greater benefits to making more information available to more people, much faster than the costs of doing so.  People who read blogs and other material on the internet are generally aware of the greater potential for error, so they usually have a lower level of confidence in information obtained from these sources than from other sources with more elaborate review and editing processes.  Some material from blogs eventually finds its way into print and more traditional outlets, and readers increase their confidence level as that information receives further review.

Of course, the same exact dynamics are at work in the research arena.  Releasing research directly to the public and through the mass media and internet improves the speed and breadth of information available, but it also comes with greater potential for errors.  Consumers of this information are generally aware of these trade-offs and assign higher levels of confidence to research as it receives more review, but they appreciate being able to receive more of it sooner with less review.

In short, I see no problem with research initially becoming public with little or no review.  It would be especially odd for a blogger to see a problem with this speed/error trade-off without also objecting to the speed/error trade-offs that bloggers have made in displacing newspapers and magazines.  If bloggers really think ideas need review and editing processes before they are shown to the public, they should retire their laptops and cede the field to traditional print outlets. 

We have a caveat emptor market of ideas that generally works pretty well.

So it was disappointing that following Eduwonkette’s graceful apology, she attempted to draw new lines to justify her earlier negative judgment about our study released directly to the public.  She no longer believes that the problem is in public dissemination of non-peer-reviewed research.  She’s drawn a new line that non-peer-reviewed research is OK for public consumption if it contains all technical information, isn’t promoted by a “PR machine,” isn’t “trying to persuade anybody in particular of anything,” and is released by trustworthy institutions.

The last two criteria are especially bothersome because they involve an analysis of motives rather than an analysis of evidence.  I defended Eduwonkette’s anonymity on the grounds that it doesn’t matter who she is, only whether what she writes is true.  But if Eduwonkette believes that the credibility of the source is an important part of assessing the truth of a claim, then how can she continue to insist on her anonymity and still expect her readers to believe her.  How do we know that she isn’t trying to persuade us of something and isn’t affiliated with an untrustworthy institution if we don’t know who she is?  Eduwonkette can’t have it both ways.  Either she reveals who she is or she remains consistent with the view that the source is not an important factor in assessing the truth of a claim.

No sooner does Eduwonkette establish her new criteria for the appropriate public dissemination of research than we discover that she has not stuck to those criteria herself.  Kevin DeRosa asks her in the comments why she felt comfortable touting a non-peer-reviewed Fordham report on accountability testing. That report was released directly to the public without full technical information, was promoted by a PR machine, comes from an organization that is arguably trying to persuade people of something and whose trustworthiness at least some people question.

So, she articulates a new standard: releasing research directly to the public is OK if it is descriptive and straightforward.  I haven’t combed through her blog’s archives, but I am willing to bet that she cites more than a dozen studies that fail to meet any of these standards.  Her reasoning seems ad hoc to justify criticism of the release of a study whose findings she dislikes.

Diane Ravitch also chimes in with a comment on Eduwonkette’s post: “The study in this case was embargoed until the day it was released, like any news story. What typically happens is that the authors write a press release that contains findings, and journalists write about the press release. Not many journalists have the technical skill to probe behind the press release and to seek access to technical data. When research findings are released like news stories, it is impossible to find experts to react or offer ‘he other side,’ because other experts will not have seen the study and not have had an opportunity to review the data.”

Diane Ravitch is a board member of the Fordham Foundation, which releases numerous studies on an embargoed basis to reporters “like any news story.”  Is it her position that this Fordham practice is mistaken and needs to stop?

3 Comments | methodology, research reports | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Eduwonkette and Eduwonk Aren’t Edumarried?

July 8, 2008

The New York Sun had a nice profile yesterday of Eduwonkette.  Well, it’s not exactly a profile because Eduwonkette writes anonymously.  In the article some folks complain that her anonymity is a problem: “A co-director of the Education Sector think tank, Andrew Rotherham, suggested on his blog Eduwonk that Eduwonkette might be unfairly pretending to be unbiased because she has ‘skin in the game… It’s this issue of you got all this information to readers, without a vital piece of information for them to put it in context.'”

I think Andy’s mistaken on this. (Did they have some kind of edu-break-up?)  The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.  I wish we all spent a whole lot less time analyzing people’s motives and a whole lot more time on their evidence and arguments. 

The only major problem with anonymity is lack of responsibility for being wrong.  There is a reputational price for making bad arguments or getting the evidence wrong that Eduwonkette avoids paying professionally — although she does pay a reputational price to the name brand of Eduwonkette.

Speaking of being wrong, Eduwonkette knocks the study Marcus Winters, Julie Trivitt, and I released today through the Manhattan Institute.  She complains: “It may be an elegantly executed study, or it may be a terrible study. The trouble is that based on the embargoed version released to the press, on which many a news article will appear today, it’s impossible to tell. There is a technical appendix, but that wasn’t provided up front to the press with the glossy embargoed study. Though the embargo has been lifted now and the report is publicly available, the technical appendix is not.”

This isn’t correct.  Embargoed copies of the study were provided to reporters upon their request.  If they requested the technical report, they could get that.  Both were available well in advance to reporters so that they could take time to read it and circulate it to other experts before writing a story.  Both the study and the technical report were made publicly available today (although there seems to be a glitch with the link to the technical report that should be fixed within hours).  The technical report can be found here.

And while we are on the subject of Eduwonkette being wrong, her attacks on test-based promotion policies are overdone.  The Jacob and Lefgren paper does raise concerns, but there is more positive evidence from the experience in Florida.  As I wrote in a previous post: “In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.”

Our results may actually be consistent with what Jacob and Lefgren find.  We find academic benefits for students retained in third grade.  They find: “that grade retention leads to a modest increase in the probability of dropping out for older students, but has no significant effect on younger students.”  It could be that test-based promotion is more beneficial when done with younger students.  It could also be that the policy has positive effects on achievement with some cost to graduation. 

And particularly severe problems with the integrity of test results used for promotion decisions in Chicago may limit the ability to generalize from Chicago’s experience.  In Chicago it may have been easier to move retained students forward by cheating on the next test than actually teaching them the basic skills they need to succeed in the next grade.

Besides, I’m sure that Edwuonkette wouldn’t put too much stock in Jacob and Lefgren’s non-peer-reviewed paper released straight to the public.  I’m sure she would be consistent in her view that: “By the time the study’s main findings already have been widely disseminated, some sucker with expertise in regression discontinuity may find a mistake while combing through that appendix, one that could alter the results of the study. But the news cycle will have moved on by then. Good luck interesting a reporter in that story… So as much as I like to kvetch about peer review and the pain and suffering it inflicts, it makes educational research better. It catches many problems and errors before studies go prime time, even if it doesn’t always work perfectly.”  

Or do these standards only apply to studies whose findings she doesn’t like?   If Eduwonkette isn’t careful she might get a reputation.

6 Comments | social promotion | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

New Study Release Tomorrow

July 7, 2008

Keep your eyes peeled for the release tomorrow by the Manhattan Institute of a new study on the effect of high-stakes testing on achievement in low-stakes subjects. The study, led by Marcus Winters and co-authored by me and Julie Trivitt, examines whether achievement in math and reading comes at the expense of science on Florida standardized tests.  Because there are meaningful consequences for performance in math and reading, but not for the rest of the curriculum, many people have worried that schools would improve their math and reading results by skimping on science and other subjects.

These concerns are not just coming from the usual critics of school accountability.  Even accountability advocates have expressed second thoughts.  For example, Chester Finn writes in the National Review Online: “Do the likely benefits exceed the ever clearer costs? Boosting skill levels and closing learning gaps are praiseworthy societal goals. But even if we were surer that NCLB would attain them, plenty of people — parents, teachers, lawmakers, and interest groups — are alarmed by the price. I don’t refer primarily to dollars. (They’re in dispute, too, with most Democrats wrongly insisting that they’re insufficient.) I refer to things like a narrowing curriculum that sacrifices history, art, and literature on the altar of reading and math skills…”

Diane Ravtich has similarly stepped on the high-stakes brakes, expressing concern about the crowding out of other academic subjects and activities: “a new organization called Common Core was launched on February 26 at a press conference in Washington, D.C., to advocate on behalf of the subjects that are neglected by the federal No Child Left Behind legislation and by pending STEM legislation. These subjects include history, literature, the sciences, the arts, geography, civics, even recess (although recess is not a subject, it is a necessary break in the school day that seems to be shrinking or disappearing in some districts). I serve as co-chair of CC with Toni Cortese, executive vice-president of the American Federation of Teachers.”

To find out whether these concerns are supported by the empirical evidence from Florida, tune into the Manhattan Institute web site tomorrow to see the study.

Leave a Comment » | school accountability | Tagged: , , , , , , , , , | Permalink
Posted by Jay P. Greene

Another Special Ed Post on Pajamas Media

July 5, 2008

Greg Forster and I continue the discussion of financial incentives and special education over at Pajamas Media.  This piece responds to a PJM column by Laura McKenna, which was a response to an earlier PJM column we wrote.

All of this builds on special education posts on this blog here, here, here, and here.

2 Comments | special ed, vouchers | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Another Order of Florida Reforms, Please

July 3, 2008

(Guest Post by Matthew Ladner)

In 2007, a family of four needed to earn less than $20,650 to qualify for a free lunch. In Arizona, the median family income for a family of four is over $65,000.

Here’s the surprising news: Low-income students in Florida—namely, those who qualify for free lunches—outperform all students in Arizona. That’s the insight to be gleaned by sifting through the treasure trove of data generated by the National Assessment of Educational Progress (NAEP), known as the Nation’s Report Card.

Figure 1 shows fourth grade reading scores for Florida students whose family income qualifies them for free lunch compared to all students in Arizona.

Figure 2 shows the percentage of Florida’s low-income children scoring basic or above on fourth grade reading and all Arizona students scoring at the same level.

You don’t need to take my word on these scores. You can go to the National Center for Education Statistics website and see them for yourself.

The point here is not to bash the underperformance of Arizona schools. Sadly, they have plenty of company. Rather, these data point to the enormity of the opportunity for improvement which we can and must achieve. Florida has found a way to significantly boost the performance of low-income students. Others should examine how and borrow everything we can.

3 Comments | competitive effects, vouchers | Tagged: , | Permalink
Posted by matthewladner

Fortune Favors the Bold

July 2, 2008

(Guest Post by Matthew Ladner)

The Goldwater Institute released a new study today co-authored by yours truly titled Fortune Favors the Bold: Reforms for Results in K-12 Education.

The study makes the case for robust education reforms rather than incremental tinkering with a fundamentally broken system. Here in Arizona, for example, inflation adjusted spending per capita in the K-12 schools has tripled since 1960, and 44% of our 4th graders in public schools score “below basic” on the NAEP 2007 4th grade reading exam.

The kids are running around with their hair on fire, while we adults wonder whether to use a cup or a bucket of water to put out the flames. It’s time to use the fire hose.

The four reforms proposed:

  • A universal system of choice, either through vouchers or a universal tax credit. Despite having among the nation’s most robust system of private and public school choice, choice programs have simply absorbed about of the third of the increase in the public school population since the charter law passed in 1994. Arizona could enjoy substantial academic and financial benefits from passing a far bolder choice program than anything currently on the books, and we would still be building new public schools.
  • Reforming Public School Governance. Charter schools make up nine out of the top 10 high schools in the greater Phoenix area, and the 10th is a magnet school. Not a single district school cracks the top 10, with charters enormously overrepresented. The lesson is clear: good schools need a district bureaucracy like a fish needs a bicycle. Schools can and should govern themselves, freeing resources for instructional use.
  • Reform of Testing. Standardized testing has jumped the shark in Arizona and elsewhere. We put forward a proposal on how to preserve the transparency and diagnostic value of testing even when states respond to pressure to dummy down their tests.
  • Measure Teacher Effectiveness on a Value Added Basis. William Sanders’ path-breaking work on value added assessment shows that students learn 50% more over a three year period with highly effective teachers compared to those stuck in classes with the bottom 20%. Dr. Sanders finds that variation attributable to teacher performance is 10 to 20 times greater than that associated with class size. Further, high-quality teachers are too rare and clustered in the leafy suburbs. Value-added measurement serves as a pre-requisite for rational and just treatment of teachers as professionals and improving learning for children.

    • Jeb Bush put in what seemed like a radical set of reforms in Florida in 1999, and they have paid huge dividends. The time has come not only for other states to emulate Florida, but to reach further, and do more. Fortune favors the bold and American education reform dithers between being misguided, pointless, and too timid.

     

3 Comments | competitive effects, instructional reform, vouchers | Tagged: , , | Permalink
Posted by matthewladner

Have You Heard About the Latest Thing in Education? Fads!

July 2, 2008

(Guest post by Greg Forster)

In a number of recent discussions, we’ve tangentially touched on the subject of educational fads: some new idea will quickly gather a huge following, tons of money will pour in to support it, and then a few years later everyone will forget about it – not because anyone actually bothered to measure whether the hot idea worked, but because a new fad will come along, and the cycle will start over. As a result, the demand for improvement that ought to be producing political capital for real reforms instead gets dissipated in the mad rush for fad after fad. Meanwhile, as educators see fads come and go, they become less receptive to all proposed ideas – even ideas that would accomplish real reform and/or have solid science showing they work. It seems to be pretty widely agreed upon that this has been one of the major problems in education over the past century.

I’m tempted to say that complaining about the problem of fads has become a hot fad among reformers. But fads are things that don’t last! Complaints about the fad problem, by contrast, are perennial. Recently I’ve noticed a couple more contributions to the genre: Roger Frank Bass of Carthage College provided a sharp-edged overview of the educational fad phenomenon in the Milwaukee Journal Sentinel on Sunday. Meanwhile, last Monday America’s last education labor reporter noted the juxtoposition of two headlines – “The Next Big Thing: Small Schools” in Baltimore, and “Small School Experiment Doesn’t Live Up to Hopes” in Seattle. “Maybe they were just on the wrong coast,” quips Antonucci. (Come to think of it, I was shocked back in May to see Newsweek pimping small schools on the front cover – this, not months but years after the failure of the Gates Foundation’s extraordinary investment in small schools had become clear even to the Gates Foundation. There’s nothing sadder than people who desperately want to be cool but are always wearing last year’s outfit.)

What I don’t see very often, though, is reformers asking why we have this problem. There are no other socially urgent functions (from law and emergency services to the production and sale of consumer goods) that have remained basically unchanged and unimproved for a century because they’re crippled by an inability to 1) stick with one idea long enough to 2) objectively measure whether it works and 3) make a deliberate decision on whether to keep it. Only education spins its wheels this way; everywhere else in society, “fads” are either the province of children and adolescents (for whom they are probably beneficial, since they harmlessly dissipate dangerous youthful energies) or else are shunted off into the realm of “fashions,” which are not supposed to accomplish anything serious and thus do nobody any harm if they constantly change.

So what’s different about education? Well, if you’ve read your homework assignment, or if you’ve spent any time here on Jay P. Greene’s Blog (or “JPGB,” as all the kids are calling it when they text each other about it), you already know the answer: accountability for results. Educational decisionmaking exists in an “outcome vaccuum” to a greater extent than decisionmaking in just about any other field.

Of course the vast majority of people in the system are well-intentioned, but we all know where the road paved with good intentions leads. Flitting from one promising idea to the next promising idea is precisely the kind of thing well intentioned people can end up doing when they lack the disciplining force imposed by firm outcome-based incentives.

“Incentives again? What, are you proposing a unified field theorem of educational problems, where everything that’s wrong with our schools traces back to this single cause?” Well, maybe not – but if there’s an alternative explanation for the unique prevalence of the fad problem in education, I’d be open to hearing it.

2 Comments | instructional reform, politics | Tagged: , | Permalink
Posted by Greg Forster

Taking the Public out of Public Schools

July 1, 2008

One of the arguments regularly trotted out against school choice is that public schools, whatever their academic defects, are at least accountable to the public in what they teach and how they operate.  And while private schools may be accountable to the parents of students who attend, they are not broadly accountable to the public in what they teach and how they operate.  Vouchers are bad, the argument goes, because public money goes to schools that are not really accountable to the public.

In light of yesterday’s post on wide-spread non-compliance by Georgia public schools with the state’s social promotion law, people who like to make this argument might want to reconsider. 

In general, I am struck by how little public accountability there can be in public schools.  As mentioned yesterday and in past postings on this blog, getting public schools to implement policies that the public adopts, through their elected representatives, is an enormous challenge.  If public school officials or educators prefer not to implement a policy they have remarkable latitude not to do it or to do it in a way that severely undermines or negates the purpose of the policy. 

Monitoring what schools actually do is extremely difficult because what happens in schools and classrooms behind closed doors is rarely seen by anyone other than the staff and students.  And even when non-compliance is observed, imposing sanctions is next to impossible.  I’ll wager that no teacher, principal, or superintendent in Georgia will lose his or her job or suffer in any other way for disobeying the law that they retain certain students.

Even in the age of NCLB, with wide-spread testing, obtaining meaningful school data can be a nightmare.  If researchers want to assess school policies or approaches they need access to individual student test and demographic data stripped of identifying information.  Requests for individual student data (with no identifying information) are regularly rejected on the false grounds that student privacy prevents releasing those data.  If there is no identifying information, then there is no threat to student privacy.  States and districts should post identity-stripped individual student data on the web for anyone to analyze.  Instead, they require proposals, inquire about purposes, assess whether the research could be embarrassing, drag their feet, and regularly turn those proposals down.

I was once even refused aggregate data for schools in a district in Massachusetts.  They claimed that they would only release the average test scores for schools if I could prove that I was a resident of their district.  What if I wanted to move there?  Too bad.

On another occasion a school official in Arizona refused to provide information on how many student were enrolled in each grade broken out by race and ethnicity.  I just wanted a count of students.  When asked why he wouldn’t provide the data, the official said that it sounded like we were trying to estimate graduation rates and he wouldn’t help us do that because we might report grad rates that were different from those reported by the state.  He actually said that out loud.

If the public isn’t allowed to know information as basic as how many students are enrolled in school, how public are those schools really?  The Arizona official acted as if they were his schools, not public schools because he could decide whether to share information based on whether it would be flattering to him or not.  Really, it should be none of his business why I want public information.  They’re my schools, too.

4 Comments | politics | Tagged: , , | Permalink
Posted by Jay P. Greene

Next Entries »