Jay P. Greene's Blog

When it is pointed out that NAEP scores for 17 year-olds or graduation rates have remained flat for roughly three decades despite a doubling in per pupil spending (adjusted for inflation), I always brace myself for the Simpson’s Paradox response.  I particularly brace for it because its most active (and grating) purveyor is Gerald Bracey — D’oh!

As Bracey explains it, “Simpson’s Paradox occurs whenever the whole group shows one pattern but subgroups show a different pattern. ”  Test scores may rise over time for every ethnic/racial subgroup but the overall average may still decline or remain flat.  “The explanation lies,” Bracey argues, “in the changing makeup of the test taking groups. At Time 1, only 20% of the test takers were minorities. At Time 2, they make up 40% of the group. Their scores are improving, but they are still lower than whites’ so as they become a larger and larger proportion of the total sample of test takers, their improving-but-lower test scores attenuate the overall average or, in this case, actually cause it to fall.”

On the surface this story sounds very appealing.  Even sensible-sounding people like JPGB commentator, Parry, repeat the argument.  But on closer examination, Simpson’s Paradox does not explain away the frustrating lack of education productivity over the last few decades.

If we want to know whether we are receiving returns on our enormous additional investment in education, we want to see progress in the overall picture.  It would provide us with little comfort to see that our investments benefited some students but did not produce an aggregate gain — unless holding steady was actually a victory in the face of significantly more difficult to educate students.

And that is the unstated argument behind the use of Simpson’s Paradox to explain the lack of educational progress: minority students are more difficult to educate and we have more of them, so holding steady is really a gain.

The problem with this is that it only considers one dimension by which students may be more or less difficult to educate — race.  And it assumes that race has the same educational implications over time.  Unless one believes that minority students are more challenging because they are genetically different, which I do not imagine Bracey or Parry believe, we have to think about race/ethnicity differently over time as the host of social and economic factors that race represents changes.  Being African-American in 1975 is very different from being African-American in 2008.  (Was a black president even imaginable back then?)  So, the challenges associated with educating minority students three decades ago were almost certainly different from the challenges today.

If we want to see whether students are more difficult to educate over time, we’d have to consider more than just how many minority students we have.  We’d have to consider a large set of social and economic variables, many of which are associated with race.  Greg Forster and I did this in a report for the Manhattan Institute in which we tracked changes in 16 variables that are generally held to be related to the challenges that students bring to school.  We found that 10 of those 16 factors have improved, so that we would expect students generally to be less difficult to educate.  For example, we observed that students are significantly more likely to attend pre-school and come to the K-12 system with greater academic preparation.  Expansions in higher educational opportunities have significantly improved the average level of parental education, which should contribute to student readiness for K-12.  Median family incomes (adjusted for inflation) have improved and a smaller percentage of children live in poverty.  Children are more likely to come to school with better health and there are fewer teen moms.

Yes, some factors have made things more difficult.  There are more students from homes in which English is not the first language and more children in single-parent households.

And yes, there are more minority students, but those minority students have better incomes, better educated parents, more pre-school, and lower rates of crime in their communities.  Unless one wants to make a genetic argument, it is obviously misleading to say that students in general are more difficult to educate because there are more minority students.

But that is exactly what the purveyors of Simpson’s Paradox are doing.  They focus only on race and act as if it were an immutable influence on academic performance.  Many things have changed over the last few decades and most of them tend to make students better prepared for K-12 school.  Even if you are not completely persuaded by the report that Greg and I produced (and we make no claim to having a definitive analysis), it would be very difficult to suggest that students have become twice as difficult to educate to completely off-set the doubling in resources we have devoted to their education.  Any reasonable examination of the evidence suggests that we have suffered from a serious decline in educational productivity, where we buy significantly less achievement for each additional dollar spent.