Civic Knowledge Polling Controversy

 (Guest Post by Matthew Ladner)

Last summer, I wrote a study for the Goldwater Institute reporting the results of a survey in which we gave 10 questions from the United States citizenship exam to Arizona high school students. The results were terrible, with only about 3.5% of the district students scoring 6 or better correct (passing). A few months later, I replicated the study for the Oklahoma Council of Public Affairs (OCPA) with Oklahoma students, and the results were even worse.

Since then the survey firm I used, Strategic Vision, has come under attack from a group of critics. I believe that it started over a poll from the New Hampshire primary last year. I am in no position to judge the merits of the case involved. Over the past few weeks, the critics have turned their fire on the civics surveys. Strategic Vision’s most aggressive critic claims that someone has replicated the Oklahoma survey and found them far more cognizant of civics than the Strategic Vision reported, concluding that SV simply fabricated the data. Smelling blood, my old friend Leo Casey is waving the bloody shirt. Some are wondering how anyone ever bought into the results in the first place or are feeling buyer’s remorse.

A few comments seem in order.

First, both myself and OCPA are investigating the validity of the survey. We have asked for and received call logs for the surveys, and we are awaiting receipts for the marketing lists employed in the survey. If, it stands to reason, a polling firm were simply fabricating data it seems terribly unlikely that they would pay thousands of dollars for marketing lists. If there has been a fraud, myself, the Goldwater Institute and OCPA were all victims of it.

Regarding the question of “how could anyone have ever believed these results” people should keep an open mind and examine the available evidence.

For the Arizona study, we purchased a poll of both public and private school students. The public results were terrible, but so too were the private school results. If memory serves, 3.5% of the public school students scored six or better and 13.8% of private school students scored six or better. We reported the private results in the study, and essentially characterized them as simultaneously better than the public results and still catastrophic.

If SV were simply manufacturing data, it seems at a minimum odd to make the private school results so terrible.

Second, results from other exams of civic knowledge should be considered. The Intercollegiate Studies Institute has run a series of tests on the civic knowledge of university students. You can look the results of the 2006 study here for both Freshmen and Seniors.

The average score for a freshman at Yale: 69.8%.

Harvard: 67.8%

Princeton: 66%

University of Texas at Austin: 53.8%

University of Michigan: 52.1%

These are all highly to uber selective universities, but strangely enough their students are arriving (and leaving btw-check out the Senior results) profoundly ignorant of American civics. In fact, the Ivy League kids, even the Seniors, score significantly worse than the alleged replication scores (no details provided btw)  in Oklahoma, where the kids supposedly got almost 80 percent correct.

Oklahoma high schools > Ivy League. Things that make you go hmmmmmm…

Next, it is worth considering the differences in testing method. The ISI results were given as multiple choice exams. If they ask you to name the first President of the United States, George Washington is jumping up and down right behind the letter “B.” You’ve really had to have been living on another planet not to get that one right.

The Arizona and Oklahoma surveys, however, emulated the methodology of the United States citizenship exams, directly lifting the questions from INS item bank, and employing their open-answer format. When you are asked “who wrote the Declaration of Independence” it is necessary to answer “Jefferson” without the benefit of four names, one of which is “Thomas Jefferson” literally staring you in the face.

In short, we have no reason to believe that the average high school student to be anywhere near as well-informed as the average student at highly selective universities. As it turns out, students at even highly selective universities know embarrassingly little about civics. Moreover the open answer format of the United States Citizenship Exam represents a higher hurdle of knowledge than a multiple choice exam. You are either carrying around the knowledge of how many Supreme Court Justices there are around in your head, or you aren’t. With a multiple choice exam, you’ve got a shot, but with an open answer format…good luck.

So my request to everyone is to stay calm and give us time to run the traps on this. If I got snookered, I’ll own up to it, but the jury is still out.



7 Responses to Civic Knowledge Polling Controversy

  1. Greg Forster says:

    You could also mention that extremely low civics knowledge is at least a century-old problem. Has everyone who’s studied this since 1917 been faking all their results?

    In a 1943 survey of civic knowledge among 6,000 incoming freshmen at the nation’s most selective colleges, half did not know the dates of the Civil War and could not locate St. Louis on a map. Nearly two-thirds mixed up Walt Whitman and jazz band leader Paul Whiteman.

    Next to that, Matt’s results seem almost unremarkable.

    I’ve written about this a few times.

    Leo needs to get himself some new cue cards.

  2. Patrick says:

    Does Leo Casey even believe there is anything wrong with public education?

  3. Greg Forster says:

    Sure he does. He thinks we don’t throw nearly enough money at it. Oh, and the blindfold needs to be on tighter.

  4. Michael Weissman says:

    The college survey you describe was qualitatively harder. It included questions on Plato, John Locke, etc. So the comparison (“Oklahoma high schools > Ivy League”) with the baby questions in the OK and AZ survey and purported surveys is bogus.

  5. Matthewladner says:


    I could not find a list of the items they used, but if you did please post it here. The format employed by the U.S. citizenship exam (open answer) is fundamentally more difficult than a multiple choice exam.

    Someone who knows more about testing than I do will have to evaluate whether a multiple choice question on Locke is more difficult than an open format question on the number of Supreme Court justices. I’m guessing it would depend on the questions.

    Some of what Silver has done, however, is to apply knowledge about multiple choice exams to this open answer format. For example, if we gave a multiple choice exam on quantum physics to high school students, we could expect a low passing rate but a somewhat normal distribution of answers with the bell curve centered around the average guesser. Lucky guessers and students with knowledge would be on one side of the bell curve, unlucky guessers on the other.

    Give those questions in an open answer format, and there is no telling what distribution of answers might look like.

  6. Adam Molnar says:

    For the ISI survey, descriptions of the 2006 questions are on the last page:

    You can get all 60 questions from the 2007 report by clicking on “Take the 2007 Survey” at their website:

    It’s not exactly the same as the 2006 survey you quoted above, but it’s highly similar. The proportions correct are close. My personal opinion is that the 2007 ISI test (I got 56 of 60) is much more difficult than the citizenship exam.

  7. Michael Weissman says:

    Matthew- Whoops, sorry to just get back here and see that you has a question for me. Fortunately Adam answered it.

    I do know that on our Physics tests (multiple choice, it’s true) the inter-item correlations are reasonably high (say r=0.15) despite the sample population being very homogeneous (almost all students in top %5 or maybe 10%). Getting SV’s r of 0.016 in a fully heterogeneous student population on homogeneous items looks way off-scale. And of course some of the results (none of the 600 wrong numbers given for how many SCOTUS justices turned out to be 7)were so statistically crazy that it’s pointless to try to describe the odds. Hundreds of students said the non-Republican party is “Communist” and not one said “Socialist”?

    I’m not blaming you. I know from experience that it’s extremely hard to think “complete fraud” when it’s just not part of your starting framework. However, in retrospect these things can be embarrassingly obvious.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s