More Charter Evidence

September 22, 2009

Diane Ravitch has declared that the Obama administration’s policy of expanding the number of charter schools has “no credible basis in research.”  This is just plain wrong.  And a new study coming out today from Stanford’s Caroline Hoxby demonstrates that she is even more wrong.

I’ve already noted that the highest quality studies — those that avoid bias from the self-selection of students into charter schools either with random-assignment or rigorous instrumental variable research designs — show significant academic benefits for students who attend charter schools instead of traditional public schools.  These studies examine the effect of charter schools in Massachusetts, Florida, Chicago, and New York City. 

And now add to that pile an updated study from Caroline Hoxby mentioned in today’s WSJ and NYT on New York City charter effects.  Students accepted by lottery into one of NYC’s charter schools in kindergarten and remained in a charter school through grade 8 closed the achievement gap with wealthy kids attending schools in Scarsdale entirely in math and two-thirds of the way in reading.

Critics are clinging to a study by Margaret Raymond at CREDO, which shows more mixed results.  While that study has the benefit of covering 15 states and DC, it can’t correct for the self-selection of students into charter schools like the highest quality studies linked above.  On average, students appear to be drawn to switching to charter schools because they are having trouble in their traditional public school.  Simply controlling for those students’ prior achievement and other observed demographic factors doesn’t quite correct for whatever negative factors may have caused students to switch to charters and that may continue to hinder their academic progress.  The CREDO study is as good as it can be given its approach, but I would have greater confidence in the consistent findings from several studies in different locations that do control for self-selection into charter schools.

8 Comments | charter schools | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

The TIMSS Rorschach Test

December 9, 2008

The Rorschach inkblot test is a psychology test that was used to assess personality and emotions.  The way in which people saw ambiguous images, like the one above, was supposed to say something about who they really were.

The same is true for the interpretations being applied to the results of the 2007 TIMSS (Trends in International Mathematics and Science Study) released today.

Over at Flypaper, Mike Petrilli interprets the gains the US has made in math but not science as suggesting that accountability testing is shifting resources toward math and away from science: “The lesson is that what gets tested gets taught. Under the No Child Left Behind act, and state accountability systems before that, elementary schools have been held accountable for boosting performance in math and reading. There is evidence that American elementary schools are spending less time teaching science, and this is showing up in the international testing data.”

And Mike interprets the relatively good results that Minnesota had (yes, MN took the test as if it were a country) as supporting rigorous standards: “There’s also good news out of Minnesota today, which has made dramatic gains since adopting new, more rigorous math standards.”

But also at Flypaper, Diane Ravtich offers different interpretations.  She sees the gains even in math results as “actually small, only four points.”  She also declines to credit NCLB for any of those gains, even as a perverse result of resource shifting away from science.  She notes that gains were at least as large in the US during the period prior to implementation of NCLB.  And on the topic of Minnesota she takes issue with Mikes explanation for success: “Minnesota showed dramatic gains on TIMSS not because of ‘new, more rigorous standards,’ but because of that state’s decision to implement a coherent grade-by-grade curriculum in mathematics.”  Umm, I would explain the difference but I got so bored trying to distinguish standards from curriculum that I dozed off for a bit.

Rather than focusing on the gains (or lack of gains) made by the US relative to itself in the past, Mark Schneider at Education Week focuses on the comparison between the US and other countries.  He notes that while the US looks relatively strong on the TIMSS, that is distorted by the large number of  “low-performing countries in the calculation of the international average [including Jordan, Romania, Morocco, and South Africa that] drives down that average, improving the relative performance of our students.”

He further notes that we fare worse on the PISA, which reports results from the 30 OECD countries who are our major trading partners and economic competitors: “We do better in TIMSS than we do on PISA, but this is a function of the countries that participate in each, and we should not let the relatively good TIMSS results lull us into a false sense of complacency. Even in the relatively easier playing field of TIMSS, we are lagging far too many countries in overall math performance and in the performance of our best students.”

And at Huffington Post Gerald Bracey was able to offer his reaction to the results last week, before they were released.  He wrote: “It might be good to keep a few things in mind when considering the data:

1. The Institute for Management Development rates the U. S. #1 in global competitiveness.

2. The World Economic Forum ranks the U. S. #1 in global competitiveness.

3. The U. S. has the most productive workforce in the world.

4. “The fact is that test-score comparisons tell us little about the quality of education in any country.” (Iris Rotberg, Education Week June 11, 2008).

5. ‘That the U. S., the world’s top economic performing country, was found to have schooling attainments that are only middling casts fundamental doubts on the value, and approach, of these surveys…'”

Bracey also said that our students could beat up the students in other countries with higher TIMSS scores.  (Actually, I made that last bit up.)

To summarize, Mike Petrilli sees evidence supporting his past concerns about the narrowing of the curriculum and the need for rigorous standards.  Diane Ravitch sees no evidence to alter her negative view of NCLB.  Mark Schneider, the former head of the National Center for Education Statistics, sees the need to review more testing.  And Gerald Bracey doesn’t even have to see the results to know that our education system is doing a great job.  And when I look at the inkblot I see a pudgy guy with a beard and male-patterned baldness laughing.

(edited for clarity)

7 Comments | research reports, school accountability | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

A Few Comments

September 9, 2008

It must be the back to school season because there are a lot of interesting education pieces on the web.  I thought I’d just mention and briefly comment on some:

  • On Matt Ladner’s Little Ramona’s Gone Hillbilly Nuts about Diane Ravtich’s new-found enthusiasm for teacher unions and hostility to charter schools and merit pay — I posted this comment on his piece: “I liked Left Back, Language Police, and much of her historical work. That’s why it’s so disappointing to read what she is writing these days. From her earlier work one would never have guessed that she would accuse people who favor merit pay, reduction in teacher tenure rights, and charter schools of plotting to destroy public education.  And for someone whose past work relied on rigorous scholarship, it is shocking to see these new claims made without any evidence that merit pay, weaker tenure, and charter schools harm public education, let alone destroy it.  Other than the fact that Bloomberg and Klein support these policies, it is not clear why Diane Ravitch opposes them.”
  • Marcus Winters has a great piece on National Review Online about how reforming the teacher compensation system is the key to improving teacher quality and, in turn, student achievement.
  • Thomas Hibbs has a not-so-great piece on National review Online about how “the true teacher cannot simply be an instrument of the wishes of the student’s family.”  He’s right that parents can sometimes try to shield their children from burdens by lowering academic expectations and that teachers need to strive for excellence regardless.  But it’s unrealistic to expect that we can build an educational system based on “the teacher’s love.”  Parents, whatever their shortcomings, are more likely to be effective advocates for a child’s progress than even well-intentioned and well-trained teachers because the parents have a love for children that we cannot realistically expect from teachers. 
  • I don’t have time to comment on them, but you should also check out the rest of the National Review Online pieces, including those by Checker Finn, Neal McCluskey, Mike Petrilli and Amber Winkler, and Susan Konig.

4 Comments | teacher quality, teacher union | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

Little Ramona’s Gone Hillbilly Nuts

September 7, 2008

(Guest Post by Matthew Ladner)

The Nashville y’alternative band BR549 wrote a great song about a woman who used to be a punk rocker, but changes into a hardcore country and western enthusiast. The lyrics describe the conversion:

She done traded in her Doc’s for kicker boots
Safety-pinned tee shirts for Manuel Suits
Her hair’s grown out and it’s piled up high
She only shows her tattoos one at a time
She ain’t ashamed of the way she was
She hears old Hank, she can’t get enough
Her punk rock records are gathering dust
‘Cos little Ramona’s gone hillbilly nuts

This song involuntarily comes to mind every time I read a blog post by Diane Ravitch like this one. It could just be me, but Ravitch’s dislike for NYC Chancellor Joel Klein seems to have gone beyond the pale.

Let’s assume that Klein has spent gobs more money without getting much in the way of results. That is a matter of dispute, and I don’t have a dog in that fight. But even if it were true, Klein would have plenty of company: spending more money with flat academic achievement is about par for the course of American education over the last thirty plus years. For instance, the NAEP long term reading scale score for 17 year olds was precisely the same in 1971 and 2004.

Nationally, real spending per pupil doubled during that same period. As sorry as that record is, you could be rightly dismissed as nuts if you tried to argue that the nation’s public school leaders were out to destroy public education. Klein doesn’t support vouchers-so there’s no story there, even for inhabitants of the anti-voucher fever swamps. He does support charter schools, but charter schools are public schools and support for charters is well within the mainstream of the Democratic Party. And yet Ms. Ravitch writes:

So this is the strange new era we are embarked upon, in which the mantle of “reformer” has passed to those who would dismantle public education, piece by piece.

What seems strange to me is making such a charge against Klein, Booker, Rhee and Fenty without presenting a scintilla of supporting evidence. Stranger still to see someone accused of spending too much money on public schools and of seeking to dismantle public education in a single post.

25 Comments | politics, teacher union | Tagged: , , , | Permalink
Posted by matthewladner

The Vitamin C of Education

August 20, 2008

Earlier this week I made my Modest Proposal for B.B. (Broader, Bolder or is it Buying Bananas?).  I noted that Randi Weingarten denounced vouchers as a waste of time despite considerable evidence supporting it, while she embraced the B.B. idea of community schools despite there being absolutely no evidence to support the claim that public schools could improve achievement by expanding their mission to include a host of social services.

Given the lack of evidence for B.B. I generously : ) offered to support a series of large pilot studies of the community schools approach, if Weingarten, Leo Casey, and the B.B. crowd would agree to a similar series of large pilot voucher programs as a way of learning more about both reform strategies.  No word yet but perhaps their internet is broken (just try unplugging it and plugging it back in).

Shital Shah from the Coalition for Community Schools, however, sent me a nice note with a link to a report claiming to contain the evidence supporting their approach.  After reviewing the report I still see virtually no evidence to give us confidence that public schools can increase student achievement by offering everything from legal assistance to health care.

In Appendix B the report lists 21 studies of the community school approach.  Seven of them have no student achievement outcomes.  Seven examine student test scores but only make pre/post comparisons without any control group.  And another seven have comparison groups but none employ random assignment, regression discontinuity, or another rigorous research design.  Four of those seven just compare achievement at schools using the B.B. approach to city or statewide averages.  And of the seven studies with some kind of control group, two find null effects, another finds null effects in math but not reading and even then only among schools with “high implementation” of the approach.  The quality (and quantity) of the evidence supporting community schools is no greater than what we could find to support the healing power of crystals. 

I understand why Randi Weingarten or Leo Casey would be pushing the educational equivalent of crystal healing.  Their job is to advocate for the interests of their union, not to make fair and reasonable assessments of research claims.  If schools expand their mission to include providing health care and other social services just think of all of the dues-paying nurses and social workers they could add to their rolls.

The greater mystery is why normally tough-minded and rigorous researchers, like Jim Heckman and Diane Ravitch, would sign on to this approach entirely lacking empirical support.  Heckman won the Nobel Prize for Economics for crying out loud.  But then again Linus Pauling won the Nobel Prize for Chemistry and later became a public advocate for mega doses of vitamin C to cure cancer, another intervention completely unsupported by rigorous evidence.

I’ll repeat that I am not against trying the B.B. community school approach with large pilot programs that are carefully studied.  I just can’t see why normally smart people would fully endorse untested approaches while ignoring other interventions, like expanding choice and competition in education, which have considerably more supporting evidence.

(edited for typos)

Leave a Comment » | myth of helplessness, vouchers | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Broader, Bolder — Bigger Budget

July 28, 2008

Check out Ken DeRosa’s critique of Broader, Bolder (the union backed plan to improve the struggling education system by doubling-down the bet and expanding the responsibilities of those struggling schools to include health care and other social services). 

Here’s my favorite bit of his post:

“Let’s take Ravitch’s defense first:

I care as much about academic achievement as Checker or anyone else in the world, but I don’t see any contradiction between caring about academic achievement and caring about children’s health and well-being.

The issue isn’t about who cares about children’s health and well-being. The issue is whether public schools, who are by and large failing at their primary task of education, should take on the additional responsibilities of caring about children’s health and well-being. You could care very much about the health and well-being of children and NOT think it’s a good idea to hand these services over to our public schools.

The argument seems to be that since children attend school every day (cough, cough) that social services could be easily provided at school. Then why not hand over these responsibilities to the post office. After all, they make house calls six days a week regardless of the rain, snow, heat, or gloom of night. They could give the kids a quick vision screen and drop off any drug prescriptions.”

4 Comments | myth of helplessness | Tagged: , | Permalink
Posted by Jay P. Greene

Blog Rankings

July 14, 2008

This blog is not yet three months old but I am pleased to report that it is off to a good start.  According to Technorati’s rankings, JayPGreene.com is attracting more readers than the American Federation of Teachers’ blog, Edwize, more than Diane Ravitch and Deborah Meier’s, Bridging Differences hosted by Education Week, more than the Reason Foundation’s Out of Control, and the Center for Education Reform’s Edspresso.  It significantly trails the educouple of Eduwonk and Eduwonkette as well as Cato at Liberty (although that’s not primarily an education blog).  Flypaper, which started about the same time as this blog, is also off to a good start.  The Queen of education blogs seems to be Joanne Jacobs.

Here are the Technorati rankings (as of this morning) of education sites that seem to share some of the same audience as this blog.  By no means is this a comprehensive list of education blogs.  And I have no idea how reliable or meaningful Technorati’s rankings really are.  I’d continue blogging no matter what the rankings were because it’s fun.  I imagine the same is true of most others.

  1. Cato at Liberty               3,662
  2. Joanne Jacobs                3,709
  3. Eduwonkette                27,419
  4. Eduwonk                      30,876
  5. Flypaper                       95,943
  6. Jay P. Greene               104,227
  7. Bridging Differences   107,924
  8. D-Ed Reckoning         107,924
  9. AFT’s Edwize              116,227
  10. Edspresso                  123,039
  11. Out of Control            123,039
  12. Core Knowledge         127,851
  13. Sherman Dorn            151,703
  14. EdBizBuzz                   184,730

8 Comments | blogging | Tagged: , , , , , , , , , , , | Permalink
Posted by Jay P. Greene

Eduwonkette Apologizes

July 8, 2008

I appreciate Eduwonkette’s apology posted on her blog and in a personal email to me.  It is a danger inherent in the rapid-fire nature of blogging that people will write things more strongly and more sweeping than they might upon further reflection.  I’ve already done this on a number of occasions in only a few months of blogging, so I am completely sympathetic and un-offended.

One could argue that these errors demonstrate why people shouldn’t write or read blogs.  In fact some people have argued that ideas need a process of review and editing before they should be shown to the public.  These people tend to be ink-stained employees of “dead-tree” industries or academia, but they have a point: there are costs to making information available to people faster and more easily.

Despite these costs the ranks of bloggers and web-readers have swelled.  There are even greater benefits to making more information available to more people, much faster than the costs of doing so.  People who read blogs and other material on the internet are generally aware of the greater potential for error, so they usually have a lower level of confidence in information obtained from these sources than from other sources with more elaborate review and editing processes.  Some material from blogs eventually finds its way into print and more traditional outlets, and readers increase their confidence level as that information receives further review.

Of course, the same exact dynamics are at work in the research arena.  Releasing research directly to the public and through the mass media and internet improves the speed and breadth of information available, but it also comes with greater potential for errors.  Consumers of this information are generally aware of these trade-offs and assign higher levels of confidence to research as it receives more review, but they appreciate being able to receive more of it sooner with less review.

In short, I see no problem with research initially becoming public with little or no review.  It would be especially odd for a blogger to see a problem with this speed/error trade-off without also objecting to the speed/error trade-offs that bloggers have made in displacing newspapers and magazines.  If bloggers really think ideas need review and editing processes before they are shown to the public, they should retire their laptops and cede the field to traditional print outlets. 

We have a caveat emptor market of ideas that generally works pretty well.

So it was disappointing that following Eduwonkette’s graceful apology, she attempted to draw new lines to justify her earlier negative judgment about our study released directly to the public.  She no longer believes that the problem is in public dissemination of non-peer-reviewed research.  She’s drawn a new line that non-peer-reviewed research is OK for public consumption if it contains all technical information, isn’t promoted by a “PR machine,” isn’t “trying to persuade anybody in particular of anything,” and is released by trustworthy institutions.

The last two criteria are especially bothersome because they involve an analysis of motives rather than an analysis of evidence.  I defended Eduwonkette’s anonymity on the grounds that it doesn’t matter who she is, only whether what she writes is true.  But if Eduwonkette believes that the credibility of the source is an important part of assessing the truth of a claim, then how can she continue to insist on her anonymity and still expect her readers to believe her.  How do we know that she isn’t trying to persuade us of something and isn’t affiliated with an untrustworthy institution if we don’t know who she is?  Eduwonkette can’t have it both ways.  Either she reveals who she is or she remains consistent with the view that the source is not an important factor in assessing the truth of a claim.

No sooner does Eduwonkette establish her new criteria for the appropriate public dissemination of research than we discover that she has not stuck to those criteria herself.  Kevin DeRosa asks her in the comments why she felt comfortable touting a non-peer-reviewed Fordham report on accountability testing. That report was released directly to the public without full technical information, was promoted by a PR machine, comes from an organization that is arguably trying to persuade people of something and whose trustworthiness at least some people question.

So, she articulates a new standard: releasing research directly to the public is OK if it is descriptive and straightforward.  I haven’t combed through her blog’s archives, but I am willing to bet that she cites more than a dozen studies that fail to meet any of these standards.  Her reasoning seems ad hoc to justify criticism of the release of a study whose findings she dislikes.

Diane Ravitch also chimes in with a comment on Eduwonkette’s post: “The study in this case was embargoed until the day it was released, like any news story. What typically happens is that the authors write a press release that contains findings, and journalists write about the press release. Not many journalists have the technical skill to probe behind the press release and to seek access to technical data. When research findings are released like news stories, it is impossible to find experts to react or offer ‘he other side,’ because other experts will not have seen the study and not have had an opportunity to review the data.”

Diane Ravitch is a board member of the Fordham Foundation, which releases numerous studies on an embargoed basis to reporters “like any news story.”  Is it her position that this Fordham practice is mistaken and needs to stop?

3 Comments | methodology, research reports | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

New Study Release Tomorrow

July 7, 2008

Keep your eyes peeled for the release tomorrow by the Manhattan Institute of a new study on the effect of high-stakes testing on achievement in low-stakes subjects. The study, led by Marcus Winters and co-authored by me and Julie Trivitt, examines whether achievement in math and reading comes at the expense of science on Florida standardized tests.  Because there are meaningful consequences for performance in math and reading, but not for the rest of the curriculum, many people have worried that schools would improve their math and reading results by skimping on science and other subjects.

These concerns are not just coming from the usual critics of school accountability.  Even accountability advocates have expressed second thoughts.  For example, Chester Finn writes in the National Review Online: “Do the likely benefits exceed the ever clearer costs? Boosting skill levels and closing learning gaps are praiseworthy societal goals. But even if we were surer that NCLB would attain them, plenty of people — parents, teachers, lawmakers, and interest groups — are alarmed by the price. I don’t refer primarily to dollars. (They’re in dispute, too, with most Democrats wrongly insisting that they’re insufficient.) I refer to things like a narrowing curriculum that sacrifices history, art, and literature on the altar of reading and math skills…”

Diane Ravtich has similarly stepped on the high-stakes brakes, expressing concern about the crowding out of other academic subjects and activities: “a new organization called Common Core was launched on February 26 at a press conference in Washington, D.C., to advocate on behalf of the subjects that are neglected by the federal No Child Left Behind legislation and by pending STEM legislation. These subjects include history, literature, the sciences, the arts, geography, civics, even recess (although recess is not a subject, it is a necessary break in the school day that seems to be shrinking or disappearing in some districts). I serve as co-chair of CC with Toni Cortese, executive vice-president of the American Federation of Teachers.”

To find out whether these concerns are supported by the empirical evidence from Florida, tune into the Manhattan Institute web site tomorrow to see the study.

Leave a Comment » | school accountability | Tagged: , , , , , , , , , | Permalink
Posted by Jay P. Greene

Next Entries »