Why Random Assignment is Important

July 2, 2009

Bill Evers has an excellent post over on his Ed Policy blog about how unreliable observational studies can be and how important it is to test claims with random-assignment research designs. 

Observational studies (sometimes called epidemiological or quasi-experimental studies) do not randomly assign subjects to treatment or control conditions or use a technique that approximates random-assignment (like regression discontinuity).  Instead they simply compare people who have self-selected or otherwise been assigned to receive a treatment to people who haven’t received that treatment, controlling statistically for observed differences between the two groups.  The problem is that unobserved factors may really be causing any differences between the two groups, not the treatment.  This is especially a problem when these unobserved factors are strongly related to whatever led to some people getting the treatment and others not. 

The solution to this problem is random assignment.  If subjects are assigned by lottery to receive a treatment or not, then the only difference between the two groups, on average, is whether they received the treatment.  The two groups should otherwise be identical because only chance distinguishes them.  Any differences between the two groups over time can be attributed to the treatment with high confidence.

If you don’t believe that research design makes a big difference, consider this table that Bill Evers provides on how much results change in the field of nutrition when random assignment (or clinical) studies are done to check on claims made by observational studies:

If we want to avoid the educational equivalent of quack medicine, we really need more random-assignment studies and we need to give the random-assignment studies we already have significantly greater weight when forming policy conclusions.

As I’ve written before, we have 10 random-assignment studies on the effects of vouchers on students who participate in those programs. Six of those ten studies show significant academic benefits for the average student receiving a vouchers and three studies show significant academic benefits for at least one major sub-group of students.  One study finds no significant effects.  

I believe that there are more random-assignment studies on vouchers than on any other educational policy and there are certainly more studies with positive results.  The depth of positive, rigorous studies on voucher participant effects is worth keeping in mind each time some new observational or (even descriptive) study comes out on school choice, including the most recent report from Florida.  Our opinion shouldn’t be based entirely on the latest study, especially if it lacks the rigorous design of several earlier studies.

3 Comments | methodology, vouchers | Tagged: , | Permalink
Posted by Jay P. Greene

The Rush to Judgment

June 30, 2009

David Figlio’s latest report on Florida’s Corporate Tax Credit (CTC) Scholarship program was released yesterday.  I can’t find the report online but Ron Matus of the St. Pete Times sent it to me and you can read about it in his article. 

(UPDATE:  Here’s a link to the study.)

I agree with almost everything said in the article.  I even agree with Mark Pudlow, the spokesperson for the teacher union when he said: “There is no quick fix for struggling students.” 

The problem is that the standard for success when it comes to school choice is that it has to produce a quick fix or critics deem it a failure and declare: “we really ought to reconsider why we’re doing it.”  No one demands that every other education policy produce huge gains in a single year or they should be “reconsidered.”  Yes, promoters of policies may make unrealistic promises to get them adopted, but the standard for success should be long-term progress, not promises made by politicians.

So let’s slow the rush to judgment and review what we really know about the CTC program.  First, Figlio finds that 92.7% of all CTC students in private schools provided a usable standardized test to the evaluation.  This shows widespread compliance with the legal requirements for those students to be tested to satisfy political concerns for accountability.

Second, Figlio finds that the CTC program has largely targeted students who are significantly more disadvantaged than students remaining in Florida’s public schools — even significantly more disadvantaged than public school students receiving subsidized or free lunch.  So, concerns that the program would cream off the best students appear unfounded.

Third, and most importantly, Figlio’s report does not make any claims about whether students benefited academically from participating in the CTC program.  He simply provides descriptive information on the academic achievement of CTC students as well as subsidized lunch students in Florida public schools.  But we know that CTC students are even more disadvantaged than those public school students and Figlio makes no attempt in this report to control fully for those disadvantages.

Figlio makes these points explicitly and repeatedly in the report: “it is important to recognize that they are not causal estimates of the effect of program participation on student outcomes. Causal comparisons require more complete modeling of the selection decisions into the scholarship program and fuller data from a baseline than is afforded using the 2006-07 school year test score collection. More compelling causal estimates of program participation will be possible following the collection of the 2008-09 school year’s test score data. The comparisons in this subsection should be interpreted as purely descriptive in nature.”

Unfortunately, most people never pay attention to these warnings and rush ahead as if descriptive information is causal.  Folks wrongly conclude that if CTC students make year to year test score gains that are about the same as subsidized lunch public school students, then they must not be benefiting from the program.  Nothing in Figlio’s report supports that conclusion. 

To know whether CTC students are benefiting we would have to know how they would be doing had they remained in public schools.  The best way to judge that is with a random-assignment study where students admitted to the CTC program by lottery are compared with students who lose the lottery and remain in public schools.  Unfortunately, that research design is not possible because there was no lottery.  The next best thing would be to use a research design that approximated random-assignment (like a regression discontinuity) or a rigorous quasi-experimental design that controlled for all observed differences between the two groups.  But Figlio didn’t do that in this report.  He just provided descriptive statistics while promising a more rigorous research design next year.

Of course, we might wonder why Figlio bothered reporting this descriptive information without a more rigorous analysis.  I suspect that he was required to produce a report each year by the legislature, so he complied even though he didn’t have the information he needed for a causal analysis. 

And the descriptive information is useful.  It suggests that choice was no miracle cure since the raw differences between CTC and public students in academic progress were not huge.  Again, miracle cure is the wrong standard for judging a program’s success. 

The CTC program may well have attracted students who had been on a downward trajectory before they switched to a private school.  And the CTC program may well helped those students level-off and may, over time, enable them to make significantly greater progress than they would have made had they remained in public schools.  This is what we’ve seen from rigorous evaluations of other choice programs, including the most recent evaluation of the DC voucher program.  But these things require careful research designs and time to show themselves.  Let’s give David Figlio more time to use a better research design so that we can actually say something about the academic effects of the CTC program.

(edited for typos)

8 Comments | vouchers | Tagged: , , , | Permalink
Posted by Jay P. Greene

Reaction To SC Decision on Special Ed

June 29, 2009

Reactions are beginning to pour in on the Forest Grove School District v. T.A. Supreme Court decision, which Greg and I wrote about last week.  Predictably and unfortunately, those reactions are informed by concerns for the financial burdens of the decision that are lacking in both facts and perspective. 

Even the dissenting opinion, written by Souter and joined by Scalia and Thomas, made a point of worrying about the costs:

The majority’s suggestion overlooks the terms of the IDEA process, the substantial procedures protecting a child’s substantive rights under the IDEA, and the significant costs of its rule. To start with the costs, special education can be immensely expensive, amounting to tens of billions of dollars annually and as much as 20% of public schools’ general operating budgets. See Brief for Council of the Great City Schools as Amicus Curiae 22–23. The more private placement there is, the higher the special education bill, a fact that lends urgency to the IDEA’s mandate of a collaborative process in which an IEP is “developed jointly by a school official qualified in special education, the child’s teacher, the parents or guardian, and, where appropriate,the child.”Burlington, supra, at 368.

Just how much private placement is there?  How much does it really cost?  How big is this relative to total enrollment and expenditure in public schools?

As of 2007 there were 67,729 disabled students in private school at public expense who were there at the initiative of their parents.  That is 1.1% of the 5,978,081 students in special education and 0.14% of the 49,610,000 students in public education.  Theses percentages were not significantly different before 1997 when Congress amended the special education law in a way that the dissent believed would constrain burdensome private placements.  And Marcus Winters and I estimated that the total financial cost of private placement is less than a billion dollars and amounts to less than one-quarter of one percent of total public school spending. 

I understand that a billion dollars is a lot of money, but in a public education system spending more than $500 billion it is almost rounding error.  Souter, Scalia, and Thomas violate the Denominator Law, where it is required that all claims of “big” problems have to be put in perspective by including a denominator to show how large the problem really is given the full context.  As officers of the Court they should know that ignorance of the law is no excuse!

It’s also strange that Scalia and Thomas would join in a dissent that is based at least partially on concerns for the financial implications of their decisions.  I thought Scalia and Thomas believed in finding the original intent of the law.  A court ruling based on (false) fears of financial burdens of the law sounds like policymaking from the bench.

However, Debra Saunders, in a column on the decision at the San Francisco Chronicle, seems confused about what constitutes policymaking from the bench.  She writes: “the court arguably engages in policy-making when it tells districts how they must spend valuable education dollars.”  There is a federal law, IDEA, that tells schools how they must spend their money (along with the money they receive from the feds).  It says that all disabled students are entitled to a free appropriate public education.  It isn’t policymaking from the bench to say that students unreasonably denied appropriate services shouldn’t have to wait 2-3 years for the Courts to order the schools to provide those services.  The Courts can’t also provide a time machine, so we have to have a mechanism that handles what happens to kids while their parents fight with schools in the courts. 

In Forest Grove the Supreme Court said that parents should be able to take the risk of placing their children in private school and seeking reimbursement.  If the courts agree that the schools unreasonably denied services, then they can get reimbursed for their costs while they were waiting.  If the public schools were reasonable, then the parents are out the money. 

I agree with Debra Saunders that the facts in this specific case make it hard to understand how a lower court found that the public schools behaved unreasonably.  But the law isn’t about one set of facts; it applies to all instances.  If we pretend that the lower courts find that public schools denied a special education classification unreasonably, then obviously students would be denied their rights under IDEA if they had to wait 2-3 years for those services.

Debra goes on to violate the Denominator Law, writing: “It’s one of those nice people things. The government has expanded the notion of disability to the point of absurdity. But nice people refuse to look at the impending drain on public school budgets, or how one child’s boarding school tuition can mean that much less funding for all the other students’ educational needs.”  This was especially frustrating because I pleaded with her to report claims of financial burdens in context.  Besides quoting me, she chose to ignore my point and repeat her claims of burden with no basis to support it.

My colleague, Walter Olson, also has a post on Forest Groveon his blog, Overlawyerd.com.  While I disagree with Wally on this issue, I am sympathetic with his concerns.  Specifically, he notes that private placement is a remedy much more available to wealthy families than poor ones.  And he doubts the justice of disabled students having federally protected rights to an appropriate education while no one else does. 

I agree that rich kids have better access to this remedy than poor kids. That’s why I favor vouchers for special education, both to democratize this remedy and to better control costs. Vouchers control costs because the voucher is worth no more than would have been spent in public schools or private school tuition, whichever is less. Special ed vouchers also discourage over-identification of disabilities because schools would risk losing students when they classify students as disabled.

And I also agree that it is unclear why non-disabled students should have to wait in schools that fail to serve them appropriately while disabled students are entitled to find an appropriate education. But the solution is not to strip disabled students of that right.  The solution is to extend it to all students. Give vouchers to all students worth the amount that would be spent on them in public school (the amount would vary based on the cost of educating different kinds of students). If any student is unable to find what his family believes is an appropriate education, give them the resources to find it somewhere else.

1 Comment | special ed, vouchers | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Choice Victory in Special Ed SC Case

June 22, 2009
The Supreme Court ruled 6-3 today in Forest Grove School District v. T.A. that disabled students that the public schools unreasonably failed to identify as disabled don’t have to wait to seek placement in a private school and reimbursement for those costs. 
This ruling seems to give families with disabled children unilateral access to vouchers for private school if they can later prove that the public schools failed to provide adequate services or unreasonably failed to identify the disability.  The families assume the financial risk if they act unilaterally, but they can be fully reimbursed for their expenses if they are proven right.  The majority reasoned that delays were so long in adjudicating these disputes, that children would be denied their right to a free appropriate public education (FAPE) if they had to wait:
 
“Our decision rested in part on the fact that administrative and judicial reviewof a parent’s complaint often takes years. We concluded that, having mandated that participating States provide a FAPE for every student, Congress could not have intended to require parents to either accept an inadequate public-school education pending adjudication of their claim or bear the cost of a private education if the court ultimately determined that the private placement was proper under the Act.” (see p. 7 )
 
Now all we need to do is to grant to all children what we have given to disabled children.  Why should any child, disabled or not, be made to wait for an appropriate education?  Why can’t all parents seek a unilateral private placement and sue to be reimbursed if they can demonstrate that the public schools were failing to provide an appropriate education? 
 
Even better, why should we make parents prove to a court that the education in the public schools was not appropriate?  Why not let the parents be the judge of the appropriateness of the education being offered?
Updated:  I just noticed that Matt made a similar argument a while back.

2 Comments | special ed, vouchers | Tagged: , , | Permalink
Posted by Jay P. Greene

Teacher Unions = The Tobacco Institute

June 22, 2009

I want to add a little to my post the other day about how the teacher unions lie and so should not be treated as credible players in policy discussions. 

The unions don’t have to lie.  The NEA didn’t have to falsely claim that the DC voucher program “yielded no evidence of positive impact on student achievement.”  They could have said something about the effects not being large or that there are other harms to vouchers that are greater than the benefits.  A pattern of lying fundamentally undermines the credibility of the teacher unions so that they will increasingly be shunned in policy discussions and lose in policy debates.

You may think that the unions are so powerful that they can just lie and get away with it, but you’d be wrong.  Remember the fate of the tobacco industry.  They created the Tobacco Institute, which produced “research” claiming to be unable to find links between smoking and cancer. 

The tobacco companies didn’t have to do this.  They could have just said that people should be free to choose whether they smoke or not regardless of health risks.  They didn’t have to lie about health effects, they could have just said that it was none of the public’s business whether people chose to smoke or not.

At the time it was conventional political wisdom that the Tobacco Institute could get away with lying because the tobacco lobby was so powerful and rich that they could do almost anything.  But eventually lying destroys one’s credibility in a way that no amount of money can restore.  And the teacher unions may suffer the same fate as the Tobacco Institute.  They may seem all-powerful right now, but over time it is hard to sustain dumb ideas, especially when lying.

4 Comments | teacher union | Permalink
Posted by Jay P. Greene

Lieberman in WashPo

June 22, 2009

Sen. Lieberman has another great piece in the Washington Post on DC vouchers.  The issue just won’t go away as much as the cool crowd wishes it would.

Leave a Comment » | vouchers | Tagged: , | Permalink
Posted by Jay P. Greene

No News — NEA Lies

June 20, 2009

You read it here on JPGB first.  The NEA sent a letter to members of Congress containing bald-faced lies about the DC voucher program.  Now the WSJ has picked up the story.  The WSJ wrote:

Public school teachers are supposed to teach kids to read, so it would be nice if their unions could master the same skill. In a recent letter to Senators, the National Education Association claims Washington, D.C.’s Opportunity Scholarships aren’t working, ignoring a recent evaluation showing the opposite.

“The DC voucher pilot program, which is set to expire this year, has been a failure,” the NEA’s letter fibs. “Over its five year span, the pilot program has yielded no evidence of positive impact on student achievement.”

That must be news to the voucher students who are reading almost a half-grade level ahead of their peers. Or to the study’s earliest participants, who are 19 months ahead after three years. Parents were also more satisfied with their children’s schools and more confident about their safety. Those were among the findings of the Department of Education’s own Institute of Education Sciences, which used rigorous standards to measure statistically significant improvement.

It should be no news that the NEA lies.  They do not have a commitment to the truth; their only commitment is to the interests of their members and leadership.  If that requires lying, they show no restraint.

The only news is that people, including the news media, public intellectuals, and policymakers, continue to treat the teacher unions as if they were credible actors in education policy discussions.  It is a mystery to me why they are ever contacted for comment by reporters or invited to serve on panels.  People who feel obliged to lie should be shunned and their opinions should never be solicited because their opinion can never be trusted as serving the truth.

I understand that the teacher unions have a right to exist, to represent their members in negotiations, and to attempt to influence policy.  But I don’t know why anyone should help them influence policy since they have shown such a callous disregard for truth and obsessive concern with self-interest.

Now I know that Leo Casey or one of his sock puppets might accuse me of being untrustworthy.  Here’s the difference:  While I might be mistaken, I am unlike the union folks in that my continued employment is not dependent on my holding particular opinions.  If I woke up tomorrow and decided that vouchers made no sense, I would be perfectly free to do so without penalty.  My position as a tenured professor does not depend at all on my believing that something  is true.

The same cannot be said for Leo Casey or other unions flacks.  If they woke up one morning and decided that vouchers were the key to improving the education system, they could not say so and expect to continue to be employed.  If they cannot change their mind without severe penalty, why would we believe that they are telling us their honest opinion now?  And if we can’t be sure that they are telling us their honest opinions, why would we ever ask them for their opinions?

I also know that some might accuse Matt or Greg of lacking the freedom to change their minds since they don’t have tenure like I do.  Actually, there is a remarkable amount of latitude at think tanks for people to say what they really think.  If you don’t believe that, think about Sol Stern or Diane Ravtich.  Besides, if Matt or Greg suddenly changed their minds they could pretty easily find work at another think tank that held a different view.  Where would all of the union people work if they changed their minds?

I say what I say because I believe it is the truth.  The teacher unions say what they say because they want something.

2 Comments | teacher union, vouchers | Tagged: , , | Permalink
Posted by Jay P. Greene

TV News

June 15, 2009

How can anyone watch TV news?  First we had the media fawning over Susan Boyle, the British Idol singing star.  Sure she has a nice voice, but the not so subtle subtext of the coverage was: “How can anyone so homely have such a beautiful voice?!”  Wow, we always thought that beauty was an essential ingredient to a good singing voice.

And now we have the sensational coverage of Amanda Knox’s murder trial in Italy.  The not so subtle subtext of the coverage is: “How can anyone so pretty be a murderer?”

I know that all sentient beings understand the shallowness of TV news, but it is worth remarking on these egregious examples.

5 Comments | random pop culture | Permalink
Posted by Jay P. Greene

Do You Know What Else Rises to the Top?

June 10, 2009

If Arne Duncan did half of what he talks about, we’d be making huge progress toward education reform.  It would be great if  he actually followed the evidence regardless of ideology, only funded what works, made strides to end the lifetime-guaranteed employment of ineffective teachers, provided financial rewards to the most effective teachers, etc… 

We’d be lucky if Duncan manages to do one-tenth of what he talks about.  But I’m amazed at how many people confuse words with action.  Mike Petrilli is right that we should praise this new rhetoric and Greg has persuasively argued that rhetoric is politically important, but people really get carried away in their praise of a bunch of mostly empty words. 

Perhaps it is natural for people to suck up to whoever is in power.  Perhaps it is the triumph of hope over experience.  But I have to say that I am deeply skeptical of what Duncan will accomplish.

Let’s take as an example the Race to the Top money.  How does anyone really believe that a one-time expenditure of less than $5 billion is going to have any significant influence on the nature of $550 billion in annual expenditures?  This isn’t the tail wagging the dog.  This is the tail of the flea on the dog wagging the dog. 

What’s more, everyone except the most politically naive understands that there is enormous political pressure on Duncan to distribute the $5 billion roughly equally so that it provides absolutely no incentive to race to the top.  Andy Rotherham has dubbed this the peanut butter meme because people are guessing “how many states the Department of Education will have to include in the ‘Race to the Top’ funds to make the initiative politically palatable without spreading the money like ‘peanut butter’ across the states”

For those who still somehow believe that the Race to the Top money is going to have a big effect (and may also believe in the tooth fairy), I’d like to make a little wager.  I’m willing to bet that every state will receive at least some money from the Race to the Top fund and that the distribution of money will be roughly proportionate. If you think I’m wrong, would you be willing to bet that fewer than 30 states get the money? 

Like with much else that Duncan says, the Race to the Top fund is just a bunch of empty words.  You can’t have 30 and certainly not 50 states at the top.  Unfortunately, cream isn’t the only thing that rises to the top.

Edited to fix the link to Greg’s post; see also Matt’s post and to clarify Andy’s quotation.

2 Comments | politics | Tagged: , , , , | Permalink
Posted by Jay P. Greene

The National Standards Sausage-Making

June 9, 2009

Every decade or so we have to debate the desirability of adopting national standards for education.  People tend to be in favor of them when they imagine that they are the ones writing the standards.  But when everyone gets into the sausage-making that characterizes policy formulation, it generally becomes clear that no one is going to get what they want out of national standards.  What’s worse is that the resulting mess would be imposed on everyone.  There’d be no more laboratory of the states, just uniform banality.

Of course, some people always hope that they’ll somehow manage to sneak their preferred vision into place without having to go through the meat grinder.  That’s what is happening now with the National Governor’s Association effort at “voluntary” national standards.  In a process completely lacking in transparency and open-debate, some are rushing to announce a national standards fait accompli.

My colleague Sandra Stotsky tells us what’s what:

“If another country wanted other countries to respect its educational system and the reforms it was trying to make, who would it choose to lead such an important professional project as the development of its national standards in mathematics and in the language of its educational system itself?  In any other country in the world, one would expect a distinguished mathematician at the college level to be asked to chair the mathematics standards-writing committee–someone who commands the respect of the mathematics profession (and obviously is an expert on mathematics).   For the language standards-writing committee, one would likewise expect an eminent scholar in a college-level department–someone whose command of the language and understanding of the texts that inform the development of this language could not be questioned.   If the National Governors Association and the Council of Chief State School Officers had thought about national pride (and national need) as well as academic/educational expertise, then all of us would respect the Common Core Initiative and look forward with eagerness to the drafts the NGA and CCSSO have promised to make public in July.

 These two organizations could have followed, for example, the exemplary procedures followed by the National Mathematics Advisory Panel, on which I had the privilege to serve.  The Panel was chaired by the former president of one of the major universities in the country, all Panel members were identified at the outset, their qualifications were made known to the pubic, their procedures were open to the public and taped as well, and the final product was hammered out in public, after dozens of reviewers provided critical comments. 

 But instead of choosing nationally known scholars to chair and staff these committees–to assure us of the integrity and quality of the product–the NGA and the CCSSO have, for reasons best known to themselves,  treated the initiative as a private game of their own.  The NGA and the CCSSO haven’t even bothered to inform the public who is chairing these committees, who is on them, why they were chosen, what their credentials are, and why we should have any confidence whatsoever in what they come up with. 

 One person has announced on his own to the press and to a state department of education that he is chairing the mathematics standards-writing committee. He has not been contradicted by anyone at NGA or CCSSO, so we must assume he’s for real.  It turns out he is an English major with no academic degrees in mathematics whatsoever.  No one has yet announced on his/her own that he/she is chairing the English standards-writing committee.   One wag has already wondered whether this person might be a mathematics major with no academic degrees in English.  But it’s possible the sad joke in mathematics is not being repeated in English. 

 This country deserved better for a project of such national importance.”

Sandy Kress added these words of wisdom (pardon the capitalization since this was a comment on a post at Eduwonk):

“i suspect after the good feelings wear off, other governors and chiefs will begin to ask whether they can or should consider new standards at this time. once they learn about how hard it is to write new standards, they will ask even more questions. when we get to the controversies around whole language vs. phonics, they will ask more questions still. then comes computation vs. concepts. then comes all the many questions that arise once you get below the level of 30,000 feet. then – God forbid – you might even get to the place where you might possibly find the new standards under consideration to be no better than (or even possibly worse than) the standards you have! could it be that the tradeoffs that happen nationally will be the same as those that occur in the states? could the same interest groups intervene? could this nice dream be interrupted by the demons that bedevil state standard setting? could these interests be the problem as much as variation? oh no, could it be there’s no santa… no, i won’t go there.

and, oh yes, what about performance standards? if we ever get to detailed precise standards in each grade for reading and math, do the participants agree to common performance standards? if they don’t, who’s kidding whom? the real problem today is not so much that some states have vastly higher standards than other states; it’s more that their performance standards are greatly different. have the states, or will the states, commit to making those the same? if not, this will be utterly fruitless.

listen – DO NOT GET ME WRONG – i’m all for higher, fewer, clearer standards. i’ve spent a lot of time working on improving texas’ standards over the past 20 years. i’ve spent a lot of time with the hunt institute pushing more common standards. this is indeed the right thing to do.

but this process is going to be much more difficult than some think. it won’t happen overnight, nor should it. and there will remain great variation at the end of the day. it is utterly naiive and/or foolish to expect states to jump track from their current gameplans, particularly where they’re reasonably well thought out.

be prepared for states to recognize this “the morning after.” texas just recognized it before “the drinking began.”

also be prepared to realize that a better approach might be for one or more of these organizations to begin by recruiting the best and the brightest and actually doing the hard work of developing a few sets of model standards and then shopping them to the states, with the political support of those who rightly want high, common standards as well as perhaps some incentives from the feds to take these steps.” 

(edited for typos)

12 Comments | instructional reform, school accountability | Tagged: , , , , | Permalink
Posted by Jay P. Greene

« Previous Entries
Next Entries »