When we examine the results of standardized test scores we typically think we are seeing evidence of what students know. As it turns out, that is only partially right. Test scores capture both what students know as well as their willingness to exert effort to show us what they know.
A new paper by my colleagues, Gema Zamarro, Collin Hitt, and Ildefonso Mendez uses multiple, novel techniques to demonstrate that between 32% and 38% of the variation in PISA test performance across countries can be explained by how much effort students are willing to exert rather than what they know. The implications of this finding for ed reform are huge. When we see low test score performance we are often misdiagnosing the problem as poor content instruction when it may in fact be insufficient development of student character skills. If we focus all of our energy on the former without addressing the later, we’ll fail to make as much progress.
So, how do Gema, Collin and Ildefonso know that between 32% and 38% of variation in PISA test performance across countries is explained by effort? They used three different methods to measure the influence of effort. First, they took advantage of the fact that the order of questions without the PISA was randomly ordered. They then compared how well students performed on the first set of items relative to the last. Because the order of items was randomized the first and last questions were, on average, of equal difficulty. The decline in getting items correct from the start to the end of the exam is therefore a function of the decline in effort students are willing to exert, not the difficulty of the items.
If you compare performance in the US and Greece (as can be seen in the figure above), students in the two countries do about as well at the beginning of the test. That means that students in Greece and the US know about the same amount of stuff. But students in Greece decline much more rapidly across the test, which means that those students are less willing to exert consistent effort. When we compare PISA results from the US and Greece we wrongly conclude that content instruction in Greece must be much worse. In reality, Greek students know as much as students in the US but simply exert a lot less effort.
A second way the paper measures effort is by examining responses students gave to a survey that was administered at the same time as the PISA. Using novel techniques that have been validated in previous research, they measure the extent to which students skip answers (or say “don’t know”) as well as the extent to which students give careless answers as proxies for their effort. Both skipped answers and careless answers yield very similar results to what they find from the decline across the test.
Some people have expressed skepticism about the focus on “non-cog” or character in education research because they believe that these capture personality traits that are largely inherited and immutable. This research contradicts that claim. Unless we think there are big and important genetic differences across countries, the variation in effort across countries has to be explained by factors that are social constructed and, at least in theory, could be changed. In addition, great work by Gema and Albert Cheng has found that student effort can actually be changed when students are randomly assigned to different teachers who themselves possess different character skills.
The evidence is becoming clear that character matters and is subject to influence by the education system.
Jay P. Greene's Blog 
The researchers believe that the “decline in getting items correct from the start to the end of the exam is therefore a function of the decline in effort students are willing to exert”?
Seems to me another cause of the decline in getting items correct from the start to the end of the exam is a phenomenon called running out of time.
Running out of time is also an indication of effort. Imagine that two people are equally able to answer the first 10 questions correctly, but one answers the last 10 at a higher rate than the other. Because the order of items is randomized, the first 10 and the last 10 items are of equal difficulty. So answering fewer of the last 10 is not a function of how much stuff one knows. It’s a function of how much effort one is exerting. Even if one person answers fewer of the last 10 because they run out of time, how quickly one moves through the test given equal ability to answer the first 10 is a matter of effort, not knowledge.
Shouldn’t any study on PISA results indicate why anyone should care what students do on it? Finnish mathematicians had to get Finland to use TIMSS instead of PISA (after Finland hit the top on PISA) to find out how their students were really doing in math.
Click to access Stars%20by%20Which%20to%20Navigate%20-%20October%202009_8.pdf
What if the second person takes twice as long to answer the first 10 questions as the first person. They get the same score on the first 10, but contestant number two finds himself desperately guessing at the end as time runs out.
The findings on non-answers and careless answers don’t align with the theory that the difference in late questions is speed rather than diligence. People with more non-answers or careless answers should have more time to complete the test. That’s not to say it’s impossible speed plays some role, but it’s not going to fully explain the difference and isn’t likely to explain most of it.
We should come up with another term to replace character “skills.”