Comments on: Why Random Assignment is Important

By: The Minnesota Kid

The Minnesota Kid — Thu, 02 Jul 2009 20:39:30 +0000

Quantitative random assignment studies use hypothesis tests to generate conclusions. The logic of the hypothesis test is that one tests to determine if the null hypothesis of “no effect” can be conclusively rejected and, therefore, the alternative hypothesis of “an effect” can be affirmed. It is a serious scientific and logical fallacy to treat the failure to reject the null hypothesis as proving no effect. It doesn’t prove anything except that the study didn’t confirm the alternative hypothesis of an effect. The next study might, however, and your study might not have simply because it had too few observations or some other limitation.

Science never proves that a relationship doesn’t exist, only that a hypothesized one likely does or we can’t be sure. Just because you don’t see something doesn’t mean it’s not there — only that you don’t see it. By your logic, not seeing something would be definitive proof that it is not there.

By: Greg Forster

Greg Forster — Thu, 02 Jul 2009 18:02:19 +0000

What the table is showing is that the results of the epidemiological studies were later found to be false when compared with the results of random-assignment research. So each time the table says “False,” this indicates that the epidemiological studies found one thing, but the random-assignment studies found the opposite.

If you really think that nothing can be proven false, how do you know anything is true?

If you want to get really technical – as it seems you do – then at the very least, instead of “unconfirmed” you should at least say something like “refuted” or “disconfirmed” – since “unconfirmed” implies that the random-assignment study was inconclusive, when in fact it was not.

By: The Minnesota Kid

The Minnesota Kid — Thu, 02 Jul 2009 16:19:39 +0000

While I’m a big fan of random assignment studies, especially in education, the table from Bill Evers’ blog is technically incorrect. A clinical trial, or any hypothesis test for that matter, never proves a claim to be “false.” All hypothesis tests can do is confirm a hypothesis or not confirm it, thereby resulting in a judgment of uncertainty. You can’t prove a negative. That’s not how science works. Instead of “false,” the Clinical Trial column in the Evers table should say “unconfirmed” for most entries. That still would be an indictment of too heavy a reliance on observational studies, but it would be a scientifically defensible one.