Jay P. Greene's Blog

 (Guest Post by Matthew Ladner)

Last summer, I wrote a study for the Goldwater Institute reporting the results of a survey in which we gave 10 questions from the United States citizenship exam to Arizona high school students. The results were terrible, with only about 3.5% of the district students scoring 6 or better correct (passing). A few months later, I replicated the study for the Oklahoma Council of Public Affairs (OCPA) with Oklahoma students, and the results were even worse.

Since then the survey firm I used, Strategic Vision, has come under attack from a group of critics. I believe that it started over a poll from the New Hampshire primary last year. I am in no position to judge the merits of the case involved. Over the past few weeks, the critics have turned their fire on the civics surveys. Strategic Vision’s most aggressive critic claims that someone has replicated the Oklahoma survey and found them far more cognizant of civics than the Strategic Vision reported, concluding that SV simply fabricated the data. Smelling blood, my old friend Leo Casey is waving the bloody shirt. Some are wondering how anyone ever bought into the results in the first place or are feeling buyer’s remorse.

A few comments seem in order.

First, both myself and OCPA are investigating the validity of the survey. We have asked for and received call logs for the surveys, and we are awaiting receipts for the marketing lists employed in the survey. If, it stands to reason, a polling firm were simply fabricating data it seems terribly unlikely that they would pay thousands of dollars for marketing lists. If there has been a fraud, myself, the Goldwater Institute and OCPA were all victims of it.

Regarding the question of “how could anyone have ever believed these results” people should keep an open mind and examine the available evidence.

For the Arizona study, we purchased a poll of both public and private school students. The public results were terrible, but so too were the private school results. If memory serves, 3.5% of the public school students scored six or better and 13.8% of private school students scored six or better. We reported the private results in the study, and essentially characterized them as simultaneously better than the public results and still catastrophic.

If SV were simply manufacturing data, it seems at a minimum odd to make the private school results so terrible.

Second, results from other exams of civic knowledge should be considered. The Intercollegiate Studies Institute has run a series of tests on the civic knowledge of university students. You can look the results of the 2006 study here for both Freshmen and Seniors.

The average score for a freshman at Yale: 69.8%.

Harvard: 67.8%

Princeton: 66%

University of Texas at Austin: 53.8%

University of Michigan: 52.1%

These are all highly to uber selective universities, but strangely enough their students are arriving (and leaving btw-check out the Senior results) profoundly ignorant of American civics. In fact, the Ivy League kids, even the Seniors, score significantly worse than the alleged replication scores (no details provided btw)  in Oklahoma, where the kids supposedly got almost 80 percent correct.

Oklahoma high schools > Ivy League. Things that make you go hmmmmmm…

Next, it is worth considering the differences in testing method. The ISI results were given as multiple choice exams. If they ask you to name the first President of the United States, George Washington is jumping up and down right behind the letter “B.” You’ve really had to have been living on another planet not to get that one right.

The Arizona and Oklahoma surveys, however, emulated the methodology of the United States citizenship exams, directly lifting the questions from INS item bank, and employing their open-answer format. When you are asked “who wrote the Declaration of Independence” it is necessary to answer “Jefferson” without the benefit of four names, one of which is “Thomas Jefferson” literally staring you in the face.

In short, we have no reason to believe that the average high school student to be anywhere near as well-informed as the average student at highly selective universities. As it turns out, students at even highly selective universities know embarrassingly little about civics. Moreover the open answer format of the United States Citizenship Exam represents a higher hurdle of knowledge than a multiple choice exam. You are either carrying around the knowledge of how many Supreme Court Justices there are around in your head, or you aren’t. With a multiple choice exam, you’ve got a shot, but with an open answer format…good luck.

So my request to everyone is to stay calm and give us time to run the traps on this. If I got snookered, I’ll own up to it, but the jury is still out.