My student, Collin Hitt, and colleague, Julie Trivitt, have an amazing paper on how we can efficiently measure an important non-cognitive skill that is strongly predictive of later life outcomes. A growing number of researchers have come to realize that lifetime success is partially a function of traditional academic achievement (cognitive skills) and partially a function of what are called non-cognitive skills, such as hard work, self-discipline, determination, etc… Schools may play a central role in conveying both types of skills, but for the most part we have only been collecting information on cognitive skills in the form of standardized test results. The main difficulty in expanding the types of measures we collect to include non-cognitive skills is that we have not developed efficient mechanisms for doing so.
Hitt and Trivitt have taken an enormous step forward to solve this problem. They have discovered that student non-response on surveys (not answering questions or saying they don’t know) is an excellent measure of non-cognitive skills that are strongly predictive of later life outcomes. In particular they examined survey response rates from the National Longitudinal Study of Youth (NLSY) given to students ages 13 to 17 in 1997. The number of items that students answered was predictive of the highest level of education students attained by 2010, controlling for a host of factors including measures of their cognitive ability. If students care enough to answer questions on a survey they are more likely to care enough to pursue their education further.
They then examined another data set to see if they found the same relationship. They did. The number of items that students in Milwaukee answered in a survey when they were in 9th grade was predictive of whether they graduated high school and went to college later, controlling for their academic achievement and other factors.
If this holds up when examined with multiple data sets, it will be an amazing breakthrough for researchers. We will finally have a fairly easy to obtain measure of an important non-cognitive skill that is predictive of later life success.
When studying voucher or other school choice programs, for example, we have observed modest test score benefits for participants, but fairly large attainment benefits. This suggests that school choice has larger effects on non-cognitive skills, but up until now we haven’t been able to observe these non-cognitive benefits without waiting nearly a decade to see if students graduate high school and go on to college. With the Hitt and Trivitt measure, we will have an early warning indicator of whether students are acquiring non-cognitive skills and are more likely to have higher attainment later.
I am not suggesting that the Hitt and Trivitt measure can be used in an accountability system, since it is certain not to work once high stakes are attached. But for research purposes it could be incredibly useful.
Developing an accurate and efficient measure of non-cognitive skills is especially important because one commonly considered measure, the self-reported “grit scale” developed by Angela Duckworth, may not be holding up very well. In the recent Dobbie and Fryer evaluation of the Harlem Promise Academy, it actually appears that the Duckworth scale was a contrary indicator of later life success. That is, students who rated themselves higher on the grit scale were less likely to succeed. We have also tried the Duckworth scale in an experiment and found that it was uncorrelated with other, behavioral measures of non-cognitive skills, such as time devoted to a challenging task and delayed gratification. But the self-reported grit scale was related to a student self-assessment of honesty, suggesting that the Duckworth scale may really measure how highly students will rate themselves more than actual grit or other non-cognitive skills.
Of course, the Hitt and Trivitt measure requires a lot more testing and field research, but it is one of the more exciting recent developments in education research.
Jay P. Greene's Blog 
I’m eager for the feedback brought on by Jay’s generous take on our paper. Those that read the paper will see that we owe an intellectual debt to David Hedengren and Thomas Stratman at George Mason University. Their working paper, “The Dog That Didn’t Bark,” provided the first compelling evidence that survey item response rates might be a measure of effort and conscientiousness. Our paper was the first to test whether, over time, item response rates were predictive of educational outcomes, as we would expect. Indeed they are – a result that we’re excited about and which I hope brings greater attention to Hedengren and Stratman’s work.
You still get a BOOOOOOOOOOOOM!!!!!!!!
This is very, very interesting.
Is it standard practice to refer to these moral character traits as “skills”? That way of framing them is going to bite us in the can down the road. Remember that Daniel Willingham paper on how much damage it does when we try to teach things that are not skills as if they were skills? The sooner we can start labeling these as character traits, the sooner we can get serious about figuring out how schools can help students form them.
Greg: “traits” and “skills” are used more or less interchangeably in the human capital literature. James Heckman is the leading researcher in the field. Here are two recent entries from his C.V.:“Achievement Tests and the Role of Character in American Life,” and “Formulating, Identifying and Estimating the Technology of Cognitive and Noncognitive
Skill Formation.” You tell me which is better.
The primary focus of my paper is conscientiousness – a person’s combination of persistence, attention to detail, decisiveness, focus, grit, punctuality. “Conscientiousness” is the term chosen by psychologists to describe these related behaviors. These are driven by a moral trait that I loosely refer to as “giving a s#!t.” But that’s not a technical term…yet.
You tell me which is better.
I already did!
Fascinating discovery by Hitt and Trivitt that the number of survey items that 13-17 year-olds choose not to answer is a predictor of their future educational attainment and lifetime success. When kids don’t care, why would we expect them to complete items in surveys, tests, applications or anything else connected with school? Getting them to care is the challenge. Caring is a pre-condition for “grit.”
When students care they tend to have developed a compelling vision of success. It’s harder for kids in poverty and in single-parent homes, especially those attending large, impersonal middle and high schools where students change subjects, teachers and work groups every 50-90 minutes in response to a bell (the proverbial “factory model”).
It’s likely easier to develop a compelling vision of success in small schools and charter schools because of the trust students tend to develop in their teachers and classmates. This advantage, however, is diminished when charters duplicate what the large “factory model” high schools do. Over 85% of American high schoolers attend the factory model. Many get lost in this chaos. Half, more in cities, have never benefited from this century-old design. Many countries are replacing it. U.S. reluctance to do so is a likely reason for stagnant achievement scores over 30 years (NAEP, ACT, SAT) despite more than doubling real spending per student.
Getting students to care would also seem connected with instruction that stimulates rather than anaesthetizes and a schedule keeps together students with the same group of teachers for enough time (most/all of the school day) to establish the necessary trust and bonding. In such learning environments teachers know that relationships and relevance are pre-conditions for achieving rigor/academic success, and developing such a family-like team-oriented environment takes time.
More time together is pursuit of excellence is often afforded by academies, magnet schools, or schools within schools that are not so wedded to the factory model. After school activities such as sports, robotics and debate teams and clubs are indeed the model for developing the intense focused relationships that get students to care.
The high school of the future, I’ll bet, will replace the one-teacher-for-one-class model with variations of the following: curricula that are highly experiential, team-taught, cross-disciplinary, project- and competency-based, computer-assisted, and which integrate training in emotional intelligence, thinking styles and career development with academics. In other words, just the opposite of the factory model.
Following are links to articles that describe a program that could serve as a template for putting together the pieces:
:
http://educationviews.org/fast-break-a-program-to-get-ready-for-the-season.
http://educationviews.org/why-college-developmental-education-is-failing-america/ – same issues in high school
A Program to Handle the Crisis of Competence – describes program from a student perspective.
So there we have it: Great teachers + great leadership + great curricula + great parent involvement = students who care with a great future! It’s unlikely to happen with the factory model that has bored the last 2-3 generations of students. Let’s train our research guns as Jay sugggests on the non-cognitive as well as the cognitive domains. This holistic approach will contribute mightily to strategies we should pursue to reinvent the American high school.
Here’s the URL for that last article on a program that gets teenagers and young adults to care and succeed.
http://www.educationnews.org/articles/9435/1/A-Program-to-Handle-the-Crisis-of-Competence/Page1.html
Researchers find that students who don’t answer all the questions are precisely the students who tend to do poorly in school?
Almost — if there are two students with the same achievement scores, the one who tends to answer more questions on surveys tends to go farther in school (and eventually earn more income) than the student who tends to answer fewer questions. Answering questions seems to be a proxy for whether someone cares enough to do something that is asked of them. All else being equal, people who care more, tend to succeed more.
This seems like a valuable study.
Thanks to Collin Hitt for clarifying the concept in his above response. It seems like these “skills” are a mix of cultural, psychological and richness of personal experience factors.
It touches me deeply that this “Give a Shxx” index of “skills” are so neglected. My most recent experience has allowed me to work closely with people who are low on the “index.” It is sad because so many have promise but are clearly not going to “make it.”
I’ve seen job interviews where the person was not hired because the asked, “Like if someone comes down do I have to come in to work.” What they meant was that that they wondered if it was OK to not come to work as scheduled if they received an unexpected visitor.” Clearly they did not understand the basic concept of a job.
The employer was not willing to take the risks associated with hiring someone who did not understand the norms of the dominant commercial culture (DCC.) The employer knew from experience that due to non-cognitive-skill reasons, this bright young person would bring cross cultural chaos that would overwhelm the management.
I’m not sure that Mr. Hitt would disagree entirely but I think what looks like “don’t give a shxx” is partially a common misconstruction by ordinary people in the DCC. It is not a matter of not caring in general as it is being in a subculture that does not recognize, and by extension, does not value or follow the norms of the DCC.
In some cases, if they acted according to the norms of the DCC be they might be chided by their piers. The pattern of ignoring these skills is thus reenforced.
The challenge seems to bring change while avoiding conflict with the current Values Free ideal in education.
To further reflect on this interesting and valuable discussion, see this article by the late Dr. Martin Haberman, 2009 recipient of AERA’s legacy award for lifetime achievement::
http://educationviews.org/unemployment-training-the-ideology-of-non-work-learned-in-urban-schools/
[…] Greene (himself brilliant) introduced me to this “brilliant new measure of non-cognitive skills.” Demonstrating proficiency in math and language arts certainly is important, but what about other […]