Bill Evers has an excellent post over on his Ed Policy blog about how unreliable observational studies can be and how important it is to test claims with random-assignment research designs.
Observational studies (sometimes called epidemiological or quasi-experimental studies) do not randomly assign subjects to treatment or control conditions or use a technique that approximates random-assignment (like regression discontinuity). Instead they simply compare people who have self-selected or otherwise been assigned to receive a treatment to people who haven’t received that treatment, controlling statistically for observed differences between the two groups. The problem is that unobserved factors may really be causing any differences between the two groups, not the treatment. This is especially a problem when these unobserved factors are strongly related to whatever led to some people getting the treatment and others not.
The solution to this problem is random assignment. If subjects are assigned by lottery to receive a treatment or not, then the only difference between the two groups, on average, is whether they received the treatment. The two groups should otherwise be identical because only chance distinguishes them. Any differences between the two groups over time can be attributed to the treatment with high confidence.
If you don’t believe that research design makes a big difference, consider this table that Bill Evers provides on how much results change in the field of nutrition when random assignment (or clinical) studies are done to check on claims made by observational studies:
If we want to avoid the educational equivalent of quack medicine, we really need more random-assignment studies and we need to give the random-assignment studies we already have significantly greater weight when forming policy conclusions.
As I’ve written before, we have 10 random-assignment studies on the effects of vouchers on students who participate in those programs. Six of those ten studies show significant academic benefits for the average student receiving a vouchers and three studies show significant academic benefits for at least one major sub-group of students. One study finds no significant effects.
I believe that there are more random-assignment studies on vouchers than on any other educational policy and there are certainly more studies with positive results. The depth of positive, rigorous studies on voucher participant effects is worth keeping in mind each time some new observational or (even descriptive) study comes out on school choice, including the most recent report from Florida. Our opinion shouldn’t be based entirely on the latest study, especially if it lacks the rigorous design of several earlier studies.