Marty West and colleagues have an incredibly important study described in Education Next this week. It’s based on a piece published earlier this year in Psychological Science, a leading psychology journal, but the Ed Next version is probably easier for ed reform folks to access and grasp.
At its heart, the study applies well-established concepts from cognitive psychology to the field of education policy, with potentially unsettling results. Intelligence, or cognitive ability, can be divided into two types: crystallized knowledge and fluid cognitive skills. Crystallized knowledge is all of the stuff you know — facts, math formulae, vocabulary, etc… Fluid cognitive skills are the ability to think quickly, keep things in memory, and solve new problems. The two are closely connected, but there are important distinctions between the two types.
West and his colleagues collected data from more than 1,300 8th graders in Boston, including some of the city’s famously high-performing charter schools to see how these schools affected both types of cognitive ability. The bottom line is that schools believed to be high-performing are dramatically improving students’ crystallized knowledge, as measured by standardized tests, but have basically no effect on fluid cognitive skills. That is, Boston’s successful charter schools appear to be able to get students to know more stuff but do not improve their ability to think quickly, keep things in memory, or solve new problems.
Perhaps we should be happy with the test score gains and untroubled by the lack of improvement in fluid cognitive skills. Chetty et al suggest that test score gains are predictive of later success in life, so who cares about those other skills? Maybe. But maybe the students in Chetty experienced improvements in both crystallized knowledge and fluid skills, but he only has measures of the former. It could still be the case that both types were essential for success.
There are worrisome signs that graduates from schools like KIPP are struggling in college despite impressive test score improvement in K-12. Perhaps the mis-match between improved crystallized knowledge and stagnant fluid skills cannot produce sustained success. Perhaps these products of successful ed reform know more of the high school curriculum but are unable to do things, like think quickly and solve new problems, that are important for later life accomplishment. E.D Hirsch and his followers have been convinced that gains in crystallized knowledge would translate into improved fluid skills, but that appears not to be the case — at least not in these model charter schools in Boston.
If fluid skills really matter, ed reform is in a serious pickle. First, we almost exclusively measure crystallized knowledge with our reliance on standardized tests. If anything, we appear to be increasingly emphasizing (and measuring) crystallized knowledge to the exclusion of fluid skills. So even when we manage to produce test score gains, we are more likely to neglect fluid skills.
Second, no one really knows how to improve fluid skills in a school setting. There are some laboratory experiments that have successfully altered fluid skills, but those effects are often fleeting and have never been replicated in a school environment. It’s taken us decades to devise some effective strategies for improving test scores. It may take us decades more to devise strategies for schools to affect fluid skills, even if we start caring about and measuring those outcomes.
Educational success probably requires addressing student needs and abilities on multiple dimensions. Large, technocratic systems built around standardized test results have a hard time focusing on more than the one dimension of test scores. The research by West, et al suggests that not all dimensions of academic progress necessarily move in sync. The unattended dimension of fluid skills may spoil the progress on the attended dimension of crystallized knowledge by undermining later-life success.
Jay P. Greene's Blog 
I think this is a really important, original study, Jay but I’m conflicted about what to take away from it. It’s impossible to interpret the findings without knowing the answer to the question, is fluid intelligence impervious to training? If it is, then what Marty et al. have shown is that charter schools can help students to achieve at levels beyond what we might expect for them, given their fixed intelligence, which is really something to celebrate. If not though, then you are dead right that the education policy world will find itself in a real pickle- by structuring systems (eg. testing and accountability systems) and policies (eg. charter renewal decisions) that measure success primarily on the basis of a metric (ie. test score gains) that only partially accounts for success.
Either way, your point about technocratic accountability systems is well-taken. The authors call for educators to experiment with Lumosity-type games and training programs to see if they can nudge students’ fluid intelligence levels but why on earth would schools devote time to this if they are caught up in an accountability system that isn’t structured to reward such gains?
These are excellent points, Anna. My guess is that the test-score progress achieved by teachers in the Chetty, et al study might have been in better sync with unobserved progress on fluid cognitive skills. Those test scores were from before the current era of high stakes testing. A good teacher back then might have been good on multiple dimensions.
But with the current technocratic focus on improving test scores, we have probably facilitated the disconnection between raising test scores and making progress on other dimensions, like fluid cognitive skills. Teachers and their students are being narrowly prepped for how they can improve test scores to the neglect of other issues.
So I am worried that the current mismatch may prevent current test score gains from yielding the same later life success that Chetty et al observed.
You are right that we do not know whether or how schools can alter fluid cognitive skills, but I think it is important that we try. And important that we keep in mind that there are probably more dimensions of effective instruction that can easily be measured and centrally managed.
Disclaimer up front – I founded one of those Boston charters, and created a teacher residency that supplies a bunch of the other charters.
Also have supported work by Marty West and Josh Angrist et al to enable studies of Boston charters.
1. It’s totally legit to be concerned that “Teachers and their students are being narrowly prepped for how they can improve test scores to the neglect of other issues.”
2. However, I don’t think that is actually what is happening. Not sure what to blurt out — it’s just not. When you’re in a math class, I think you’d say “Hmm, this seems like a pretty good math class compared to X and Y what I’ve seen elsewhere.”
3. I’d invite you to visit and make up your own mind.
4. Our internal data on our alums’ college graduation rates seem to show large gains on that indicator too.
5. Now these are imperfect. And we don’t have labor market outcome data.
What we need is a follow-up study to the Kane/Angrist lottery study where charter lottery winners and losers are tracked for their college outcomes and (per Chetty) their labor market outcomes. I’m actually not sure if that study is already underway. But if it is, I’ll put down a friendly wager on “large effect” on both and even give you some odds.
6. I agree with Anna, you, and Marty — that the mechanism of exactly how these outlier 2% of charters produce unusually large test score gains is kind of cloudy.
7. After 15 years of doing this, my sense of that mechanism —
a. the top charters get kids to put in much more effort compared to what they’d done before;
b. sometimes succeed in getting that additional effort to yield a lot of additional learning/knowledge and sometimes only a little;
c. that additional effort also forms some habits that lead to later work “on their own” (college and beyond);
d. a weakness is that just like most other public schools, we do a bit too much english and math, and not enough science and social studies, but that doesn’t explain why no excuses charters get large gains and others do not
e. the increased knowledge and effort leads (based on our internal data, but not established by a lottery study to my knowledge) to greater college success for the alums; greater high school graduation rates for even those who attend but attrit;
f. whether that leads to earning gains that are smaller per “test score gain” than Chetty’s data, or in fact larger than Chetty’s data (which i think is possible), i have no idea. but I’d bet on same or larger.
g. we haven’t explicitly tried to drive up “fluid intelligence” — and to my knowledge, i’m not aware of other public schools (or private ones) trying to do that.
okay, that’s my 2 cents.
my question jay –
essentially, does your view on the merits of no excuses schools (separate from the benefits of parent choice, which apply to all charters, not just those with high test scores)….boil down to “If Chetty later life earnings and college grad rate gains are there, count me in; if they’re not there, it suggests these schools just did meaningless test prep?”
Hi Mike,
These are wonderful comments. I appreciate your ability to bring up-close observation of these schools into the discussion. And you may well be right that the lack of movement on fluid cognitive skills is generally not a problem. I certainly hope you are right.
But I am a worrier, so I worry about whether the mismatch between progress on crystallized knowledge and fluid intelligence is a warning flag. It certainly undermines my confidence in the ED Hirsch assumption that all of these things should move in sync.
I also agree that the next step is to get later life outcomes for these kids and I believe those efforts are underway. We’ll see.
As to my view of the merits of no excuses schools — I support them regardless. I am no better at being a technocrat judging school quality from test scores than anyone else. Besides, even crystallized knowledge seems good even if it doesn’t yield money. As Animal House said, “Knowledge is good.”
1. Knowledge is good. Belushi too:
“Christ. Seven years of college down the drain. Might as well join the f’ing Peace Corps.”
Little known: was actually that quote which motivated Wendy Kopp to create TFA.
2. Thanks for reply. I’ll read up on Hirsch’s views here on fluid intel.
The science of intelligence testing is over 100 years old, but, in the wake of brouhahas over “The Bell Curve,” Larry Summers, and James D. Watson, the ed reform movement has been trying to ignore its many valuable lessons. You’ll notice that the authors of the Education Next article don’t mention the dread letters IQ and write, “Psychologists now consider cognitive ability (few dare say “intelligence” anymore) to have two primary components: crystallized knowledge and fluid cognitive skills.”
So, old lessons that were known to scientists like Raymond Cattell (1905-1998) and Arthur Jensen (1923-2012) are slowly, haltingly being relearned.
What Mr. Greene worries about is a very serious problem if the sole goal of Ed Reform is to Close the Gap of approaching one standard deviation in intelligence and achievement seen on average between blacks and Hispanics versus Asians and whites.
A more feasible and scientifically reasonable national goal, however, would be to try to boost the achievement of _each_ group by half of a standard deviation.
First, it’s a lot less hard to boost performance by half a standard deviation than by a full standard deviation.
Second, flexible cognitive ability or intelligence or IQ or the g Factor or whatever is least politically incorrect to call it puts a leash on how much achievement can be increased. Nothing we can do in K-12 can make people with 2-digit IQs able to be hired as Google software engineers.
On the other hand, we ought to be able to teach people with 90 IQs enough to be respectable citizens with productive jobs. (75 IQ is always going to be a problem, though).
But, if reformers don’t allow themselves to think in terms of the scientifically battle-tested concepts developed the last 100+ years by IQ researchers, they’ll be fighting the battle with one arm tied behind their backs.
The goal of the education reform movement should be to improve outcomes rather than in closing achievement gaps. There is ultimately only one good way to close an achievement gap (everyone improving but the bottom improving faster than average). There are multiple bad ways for gaps to close (top coming down, everyone going down but top declining faster).
Unfortunately, a very large fraction of public statements by key ed reform leaders focus upon Closing the Racial Gap. In turn, that leads to self-defeating policies like less disciplining of black youths.
Nothing to add here – just wanted to say I enjoyed reading the comments. Thanks, Neerav