Civic Knowledge Polling Controversy

November 12, 2009

 (Guest Post by Matthew Ladner)

Last summer, I wrote a study for the Goldwater Institute reporting the results of a survey in which we gave 10 questions from the United States citizenship exam to Arizona high school students. The results were terrible, with only about 3.5% of the district students scoring 6 or better correct (passing). A few months later, I replicated the study for the Oklahoma Council of Public Affairs (OCPA) with Oklahoma students, and the results were even worse.

Since then the survey firm I used, Strategic Vision, has come under attack from a group of critics. I believe that it started over a poll from the New Hampshire primary last year. I am in no position to judge the merits of the case involved. Over the past few weeks, the critics have turned their fire on the civics surveys. Strategic Vision’s most aggressive critic claims that someone has replicated the Oklahoma survey and found them far more cognizant of civics than the Strategic Vision reported, concluding that SV simply fabricated the data. Smelling blood, my old friend Leo Casey is waving the bloody shirt. Some are wondering how anyone ever bought into the results in the first place or are feeling buyer’s remorse.

A few comments seem in order.

First, both myself and OCPA are investigating the validity of the survey. We have asked for and received call logs for the surveys, and we are awaiting receipts for the marketing lists employed in the survey. If, it stands to reason, a polling firm were simply fabricating data it seems terribly unlikely that they would pay thousands of dollars for marketing lists. If there has been a fraud, myself, the Goldwater Institute and OCPA were all victims of it.

Regarding the question of “how could anyone have ever believed these results” people should keep an open mind and examine the available evidence.

For the Arizona study, we purchased a poll of both public and private school students. The public results were terrible, but so too were the private school results. If memory serves, 3.5% of the public school students scored six or better and 13.8% of private school students scored six or better. We reported the private results in the study, and essentially characterized them as simultaneously better than the public results and still catastrophic.

If SV were simply manufacturing data, it seems at a minimum odd to make the private school results so terrible.

Second, results from other exams of civic knowledge should be considered. The Intercollegiate Studies Institute has run a series of tests on the civic knowledge of university students. You can look the results of the 2006 study here for both Freshmen and Seniors.

The average score for a freshman at Yale: 69.8%.

Harvard: 67.8%

Princeton: 66%

University of Texas at Austin: 53.8%

University of Michigan: 52.1%

These are all highly to uber selective universities, but strangely enough their students are arriving (and leaving btw-check out the Senior results) profoundly ignorant of American civics. In fact, the Ivy League kids, even the Seniors, score significantly worse than the alleged replication scores (no details provided btw)  in Oklahoma, where the kids supposedly got almost 80 percent correct.

Oklahoma high schools > Ivy League. Things that make you go hmmmmmm…

Next, it is worth considering the differences in testing method. The ISI results were given as multiple choice exams. If they ask you to name the first President of the United States, George Washington is jumping up and down right behind the letter “B.” You’ve really had to have been living on another planet not to get that one right.

The Arizona and Oklahoma surveys, however, emulated the methodology of the United States citizenship exams, directly lifting the questions from INS item bank, and employing their open-answer format. When you are asked “who wrote the Declaration of Independence” it is necessary to answer “Jefferson” without the benefit of four names, one of which is “Thomas Jefferson” literally staring you in the face.

In short, we have no reason to believe that the average high school student to be anywhere near as well-informed as the average student at highly selective universities. As it turns out, students at even highly selective universities know embarrassingly little about civics. Moreover the open answer format of the United States Citizenship Exam represents a higher hurdle of knowledge than a multiple choice exam. You are either carrying around the knowledge of how many Supreme Court Justices there are around in your head, or you aren’t. With a multiple choice exam, you’ve got a shot, but with an open answer format…good luck.

So my request to everyone is to stay calm and give us time to run the traps on this. If I got snookered, I’ll own up to it, but the jury is still out.


Freedom from Responsibility Preview part Deux

June 25, 2009

 (Guest Post by Matthew Ladner)

In the end more than they wanted freedom, they wanted security. When the Athenians finally wanted not to give to society but for society to give to them, when the freedom they wished was freedom from responsibility, then Athens ceased to be free.

Edward Gibbon

Last week we had our first sneak peak at Freedom from Responsibility.

Today, more details about the results. The Goldwater Institute randomly drew 10 questions from the United States citzenship exam item bank. We hired a survey firm to interview a sample of both Arizona public and private school high school students.

The questions for neither the citizenship test nor our survey were multiple choice. When you are asked “Who was the first President?” you must answer “Washington” in order to receive credit. Applicants for citizenship must get six out of the ten questions correct to pass. A recent trial of a slightly reformatted exam found that 92.4% of citizenship applicants passed the test on the first try.

Charles N. Quigley, writing for the Progressive Policy Institute, explained the critical nature of civic knowledge:

From this nation’s earliest days, leaders such as Thomas Jefferson, James Madison, and John Adams recognized that even the well-designed institutions are not sufficient to maintain a free society. Ultimately, a vibrant democracy must rely on the knowledge, skill, and virtues of its citizens and their elected officials. Education that imparts that knowledge and skill and fosters those virtues is essential to the preservation andimprovement of American constitutional democracy and civic life.

Paul D. Houston, the executive director of the American Association of School Administrators, also put the issue in focus:

If you look back in history, you will find the core mission of public education in America was to create places of civic virtue for our children and for our society. As education undergoes the rigors of re-examination and the need for reinvention, it is crucial to remember that the key role of public schools is to preserve democracy and, that as battered as we might be, our mission is central to the future of this country.

Here are the 10 questions randomly selected, and their answers:

1.What is the supreme law of the land?Answer: The Constitution

2. What do we call the first 10 amendments to the Constitution?

Answer: the Bill of Rights

3. What are the two parts of the U.S. Congress?

Answer: Senate and House

4. How many Justices are on the Supreme Court?

Answer: Nine

5. Who wrote the Declaration of Independence?

Answer: Jefferson

6. What ocean is on the East Coast of the United States?

Answer: Atlantic

7. What are the two major political parties in the United States?

Answer: Democratic and Republican

8. We elect a U.S. Senator for how many years?Answer: Six

9 . Who was the first President of the United States?Answer: Washington

10. Who is in charge of the Executive Branch?Answer: The President

A majority of Arizona public high school students got only one of these questions correct, with 58% correctly identifying the Atlantic Ocean as being off the east coast of the United States, with 42% unable to do so. It was all downhill from there. 29.5% of students identified the Constitution as the supreme law of the land, 25% of students identified the Bill or Rights as the first 10 amendments to the Constitution (12% said they were called “The Constitution” and 16% “The Declaration of Independence.”)

Twenty three percent of Arizona public high schoolers identified the House and Senate as the chambers of Congress. Nine point four percent that the Supreme Court has nine justices. Only 25% of students correctly identified Thomas Jefferson as the author of the Declaration of Independence. An almost majority of 49.6 percent identified the two major political parties, only 14.5% answered that Senators are elected for six year terms. Finally, only 26.5% of students correctly identified George Washington was the first President. Other guesses included John F. Kennedy, Ronald Reagan, George Bush and Barack Obama.

Only 26% could identify the President as being in charge of the Executive Branch. All in all, only 3.5% of public school students passed the test by getting six or more items correct. That’s 40 students out of a sample of 1,134 district students.

There were no major differences in performance based on grade (Seniors did approximately as poorly as Freshmen) nor by ethnicity. Profound ignorance is quite equally distributed in large measure across students in the public school system.

Two obvious questions to ask: is it fair to give this test? In order to answer, I examined the Arizona state standards for 8th grade social studies, which all or nearly all of these students will have taken. These standards are included as an Appendix in the study. What they show is that students are supposed to have learned about John Locke, the Mayflower Compact, the Articles of Confederation, the Constitution, Checks and Balances, Seperation of Powers, etc. etc. etc.

Everything they ought to have needed, in other words, to have passed this test. If, that is, they had actually learned any of that material in practice, which they obviously did not.

Second, I gave the test to my own 1st and 2nd grade sons. They both got 3 answers correct. We’ll be working on that. In so doing, they outscored about 40% of the Arizona high school sample, and tied or exceeded about 60 percent.

Charter school kids performed far better but still terribly- with a passing rate about twice as high as the public school kids. Private school students passed at a rate four times higher, which ultimately is both much better and still pathetic.

I had a very difficult time writing a conclusion to this study. More on that for the next post, but you tell me: if you were an Arizona lawmaker what would you do about this?