Simpson’s Paradox — D’oh!

January 12, 2009

When it is pointed out that NAEP scores for 17 year-olds or graduation rates have remained flat for roughly three decades despite a doubling in per pupil spending (adjusted for inflation), I always brace myself for the Simpson’s Paradox response.  I particularly brace for it because its most active (and grating) purveyor is Gerald Bracey — D’oh!

As Bracey explains it, “Simpson’s Paradox occurs whenever the whole group shows one pattern but subgroups show a different pattern. ”  Test scores may rise over time for every ethnic/racial subgroup but the overall average may still decline or remain flat.  “The explanation lies,” Bracey argues, “in the changing makeup of the test taking groups. At Time 1, only 20% of the test takers were minorities. At Time 2, they make up 40% of the group. Their scores are improving, but they are still lower than whites’ so as they become a larger and larger proportion of the total sample of test takers, their improving-but-lower test scores attenuate the overall average or, in this case, actually cause it to fall.”

On the surface this story sounds very appealing.  Even sensible-sounding people like JPGB commentator, Parry, repeat the argument.  But on closer examination, Simpson’s Paradox does not explain away the frustrating lack of education productivity over the last few decades.

If we want to know whether we are receiving returns on our enormous additional investment in education, we want to see progress in the overall picture.  It would provide us with little comfort to see that our investments benefited some students but did not produce an aggregate gain — unless holding steady was actually a victory in the face of significantly more difficult to educate students.

And that is the unstated argument behind the use of Simpson’s Paradox to explain the lack of educational progress: minority students are more difficult to educate and we have more of them, so holding steady is really a gain.

The problem with this is that it only considers one dimension by which students may be more or less difficult to educate — race.  And it assumes that race has the same educational implications over time.  Unless one believes that minority students are more challenging because they are genetically different, which I do not imagine Bracey or Parry believe, we have to think about race/ethnicity differently over time as the host of social and economic factors that race represents changes.  Being African-American in 1975 is very different from being African-American in 2008.  (Was a black president even imaginable back then?)  So, the challenges associated with educating minority students three decades ago were almost certainly different from the challenges today.

If we want to see whether students are more difficult to educate over time, we’d have to consider more than just how many minority students we have.  We’d have to consider a large set of social and economic variables, many of which are associated with race.  Greg Forster and I did this in a report for the Manhattan Institute in which we tracked changes in 16 variables that are generally held to be related to the challenges that students bring to school.  We found that 10 of those 16 factors have improved, so that we would expect students generally to be less difficult to educate.  For example, we observed that students are significantly more likely to attend pre-school and come to the K-12 system with greater academic preparation.  Expansions in higher educational opportunities have significantly improved the average level of parental education, which should contribute to student readiness for K-12.  Median family incomes (adjusted for inflation) have improved and a smaller percentage of children live in poverty.  Children are more likely to come to school with better health and there are fewer teen moms.

Yes, some factors have made things more difficult.  There are more students from homes in which English is not the first language and more children in single-parent households.

And yes, there are more minority students, but those minority students have better incomes, better educated parents, more pre-school, and lower rates of crime in their communities.  Unless one wants to make a genetic argument, it is obviously misleading to say that students in general are more difficult to educate because there are more minority students.

But that is exactly what the purveyors of Simpson’s Paradox are doing.  They focus only on race and act as if it were an immutable influence on academic performance.  Many things have changed over the last few decades and most of them tend to make students better prepared for K-12 school.  Even if you are not completely persuaded by the report that Greg and I produced (and we make no claim to having a definitive analysis), it would be very difficult to suggest that students have become twice as difficult to educate to completely off-set the doubling in resources we have devoted to their education.  Any reasonable examination of the evidence suggests that we have suffered from a serious decline in educational productivity, where we buy significantly less achievement for each additional dollar spent.

11 Comments | myth of helplessness | Tagged: , , , , | Permalink
Posted by Jay P. Greene

Rising Condemnation of US Incursion into Arizona

January 11, 2009

Missiles continued to fall in and near San Diego despite the US incursion into Arizona to halt the rocket-fire.  And rising condemnation from world leaders and street protests around the globe urged the US to end the humanitarian crisis.

China said it is shocked by the US attack on Arizona and has called for an immediate halt to the military campaign that has killed over 800 people.

Vice Premier Li Keqiang said in a statement on the Foreign Ministry’s Web site Sunday that “the Southwest peace process must continue and that realistic measures to ease the tension in Arizona should be carried out.”

The US unilaterally withdrew from Arizona in 2005 and Anglo settlements were dismantled.  Since that time around 7,200 rockets and mortars have struck parts of southern California.

In a recent op-ed former president Jimmy Carter explained the rationale for rocket-fire: “We knew that the 6.2 million inhabitants of Arizona were being starved, as the U.N. special rapporteur on the right to food had found that acute malnutrition in Arizona was on the same scale as in the poorest nations in the southern Sahara, with more than half of all Chicano families eating only one meal a day. Chicano leaders from Arizona were noncommittal on all issues, claiming that rockets were the only way to respond to their imprisonment and to dramatize their humanitarian plight.” 

Arizona is surrounded on three sides by US forces that restrict the flow of goods into the territory.  Commenting on the US two-week military offensive against Arizona, Cardinal Renato Martino, a former Vatican envoy to the United Nations and now Pope Benedict XVI’s top official on issues of peace and justice told the online newspaper Il Sussidiario.net that both sides were concerned only with their own interests.  “But the consequences of this selfishness is hatred, poverty, injustice. It is always the defenseless populations that pay,” he was quoted as saying. “Look at the conditions in Arizona: It looks more and more like a big concentration camp.”

Thousands of demonstrators in Madrid Sunday called for a halt to US attacks on Arizona, in a protest whose sponsors included Spain’s ruling Socialist Party.  Spain’s Socialist Prime Minister, Jose Luis Rodriguez Zapatero sharply criticized the US, calling its response to Chicano rockets fired at the US  “disproportionate.”  During this round of the conflict more than 800 Chicanos have been killed and more than 3,000 wounded compared to 10 US dead, 7 soldiers and three civilians.  Almost three million southern California residents are within the range of Chicano rockets, thousands of whom have been forced to seek shelter in bunkers or flee north.

“There is no military solution in Arizona,” the Los Angeles Times wrote in an editorial today.  “Weapons stockpiles and supply tunnels have been destroyed; leaders of the military wing and fighters have been killed. That may eventually buy short-term relief for the people of southern California who live under a rain of rocket fire, and whose government has every obligation to secure their safety. But rather than weaken the Chicano Resistance politically, it seems just as likely that the effect of the bloody siege will be to harden sentiment against the US on the Chicano street and drive new recruits into the arms of the Resistance’s military.”

Celebrities have joined human rights campaigners to call on British Prime Minister Gordon Brown to speak up against the US bombardment of Arizona.  Singer Annie Lennox and former model Bianca Jagger both made passionate pleas for an end to the bloodshed.

Student protestors at the University of California at Berkeley held aloft a photo of a Chicano man holding a key to his family’s  home in La Mesa, just east of San Diego.  “When Anglo militia groups violently took over the land that became Southwestern states in 1848,” said student leader Olin Tezcatlipoca, “They committed mass atrocities that led to the expulsion of thousands of  indigenous Chicanos from their homes. These people have never been allowed to return, and many continue to live difficult lives in refugee camps scattered throughout Latin America, as well as in temporary refugee camps in Arizona.  The only solution is to end the occupation.”

2 Comments | politics | Permalink
Posted by Jay P. Greene

Willingham Strikes Again

January 10, 2009

Psychologist Dan Willingham strikes again with another excellent video about educational psychology.  In this one he makes a strong case for the importance of teaching content for teaching reading.

2 Comments | instructional reform | Tagged: , , | Permalink
Posted by Jay P. Greene

Quality Counts Lacks Quality

January 9, 2009

(Guest post by Stuart Buck)

Education Week has released its annual report “Quality Counts,” which ranks all fifty states’ education systems along several dimensions, such as school finance, achievement, accountability, and the like.  You can find detailed statistics for any given state on an interactive map, and you can generate a table comparing the states of your choosing.

This Quality Counts report gets a huge amount of attention, as can be seen from the hundreds of results in a search of Google News.

  But the Quality Counts report suffers from two glaring flaws.  In fact, the report reminds me of the old joke (I can’t remember who to credit for this) of a beggar sitting on the streets of New York, with a sign reading, “Wars, 2; Legs Lost, 1; Wives Who Left Me, 2; Children, 3; Lost Jobs, 2.  TOTAL: 10.”  Well, obviously, the number “10” doesn’t represent ten of anything. 

 So what’s wrong with the Quality Counts report?

First, the “School Finance” measure has two basic components: equity and spending.  Equity refers to several measures that look at whether a state’s districts get relatively equal funding.  Fair enough, although there’s a decent argument that impoverished districts might need higher spending to attract better personnel.  But then part of the “School Finance” measure is based on per-pupil spending, as well as the percentage of a state’s taxable resources dedicated to education. 

The problem here is that it doesn’t make sense to reward a state with a higher grade just for spending more, in and of itself.  Indeed, the “spending” measure ends up getting averaged with the measure for “K-12 Achievement.”  This means that, in theory, a state with high spending and low achievement — thus combining incompetence and extravagance — could get an overall score equal to a state with low spending and high achievement.  But if a school manages to get high achievement with low spending, this means that, all else equal, that state has a more efficient and productive education system. 

Second, an even worse problem lies in the “Chance for Success” measure.  This ranking is supposed to tell us about the chances that people in a given state have of succeeding.  There are numerous components to the “Chances for Success” measure, including percent of students above 200% of the poverty line, percent of students with college-educated parents, percent of children whose parents speak English, and more.  Not surprisingly, the richer and more privileged states like Massachusetts, New Jersey, and Connecticut do quite well on this measure, while states like Arkansas, Mississippi, and New Mexico are near the bottom. 

What makes no sense whatsoever is that a high score on the “Chance for Success” measure is averaged together with all the other items — including K-12 Achievement — to produce each state’s final score.  You can see this for yourself: Pick your home state here, and then take the simple average of all six measures (Chances for Success; Standards, Assessment & Accountability; K-12 Achievement; Transitions & Alignment; School Finance; and Teaching Profession), and that average will be the state’s overall final score. 

In other words, imagine a state that managed to produce A-level achievement even though its population was poor and disadvantaged (and thus got a lower grade on the “Chances for Success” measure).  Under any rational grading system, we should give that state the highest possible rating.  But the Quality Counts method would actually downgrade the state for having too many poor children.  By the same token, Quality Counts would upgrade a poor-achieving state that happened to have a privileged and rich student population, even though that state’s education system would obviously be far more incompetent and inefficient.  If anything, the “Chances for Success” ranking should be counted inversely as compared to all the other measures of a state’s education system. 

6 Comments | school accountability | Tagged: , | Permalink
Posted by Jay P. Greene

The Irony of Social Promotion

January 9, 2009

In the current issue of the Economics of Education Review, Marcus Winters and I have an article about the use of exemptions to Florida’s test-based promotion policy.  Under Florida’s policy students need to perform above a certain level on the 3rd grade reading test to automatically be promoted to 4th grade.  If  students score below that level they can still be promoted if they are granted one of various exemptions.  Some of those exemptions are objectively measured, like scoring well on an alternative test or having certain special ed or English Language Learning classifications.  But other exemptions are more subjectively determined, like having a portfolio of work worthy of being promoted.

Marcus and I looked at who received those exemptions and whether being exempted was beneficial.  We found that African-American and Hispanic students were less likely to receive exemptions and get promoted, controlling for other factors.  That is, minority students with the same test scores and economic status were less likely to be exempted from retention if they fell below the testing threshold.  The test-based policy is not racially biased, since all students who lack the academic skills to pass the test may be retained.  The bias is introduced in who gets exempted from that test-based policy.

And the irony of it all is that failing to receive an exemption actually benefited those minority students academically.  That is, students who were denied the exemption and repeated third grade outperformed their promoted colleagues on achievement tests two years later.  The retained students had more academic skill at the end of 4th grade than their comparable promoted peers at the end of 5th grade — despite being exposed to one less grade of curriculum. 

Minority students denied the exemptions may have been the vicitms of discrimination, but they ended-up making greater academic progress as a result.  Receiving those exemptions wasn’t doing many of the other students any favors.

The St. Pete Times has an article on the study today and had a blog post recently.

2 Comments | social promotion | Tagged: , , , | Permalink
Posted by Jay P. Greene

Randi Weingarten Can’t Get No Respect

January 5, 2009

In what the AFT web site described as “her first major speech since being elected AFT president in July,” Randi Weingarten “decried the widespread scapegoating of teachers and teachers unions for public education’s shortcomings.”  Her comments have generated numerous reactions, including from NYT columnist Bob Herbert, Andy Rotherham, Joanne Jacobs, and our own Greg Forster.  They all raised interesting points, but none addressed one of the most curious aspects of Weingarten’s speech:  Why do teachers, perhaps more than other professionals, seek praise for their work (or are particularly sensitive to blame)? 

I don’t think other occupations have produced bumper-stickers that are the equivalent of “If you can read this thank a teacher.”  I can’t imagine plumbers distributing bumper-stickers that said: “If you flushed your toilet thank a plumber.”  Nor can I imagine: “If you still have your teeth thank a dentist.” 

Teachers particularly demand respect — and of course they deserve respect.  But why do they give speeches, print bumper-stickers, write letters, hold rallies, etc… decrying their social status when I am hard pressed to think of other occupations that do the same?

Of course, one important factor is that almost all teachers are public employees.  The demand for respect can be understood as part of the demand for resources.  My plumber doesn’t have to demand my respect to get my resources.  He just has to do a good job to get me to continue paying him for his services. 

But the resources devoted to education are largely unrelated to how well teachers serve their students.  Political popularity largely determines the level of resources available for teachers, so not surprisingly, teachers actively lobby the public to enhance their image.

The problem is that it is hard to sustain political popularity and community respect as results continue to disappoint despite huge increases in resources.  Teachers interpret this disappointment as a lack of respect, when it is really just frustration at being forced to pay for services that are chronically inadequate.  If people could hire teachers like they hire plumbers or dentists, teachers wouldn’t need to demand respect to get resources.  They would earn respect and resources by serving their voluntary customers well.

9 Comments | teacher pay, teacher union | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Blaming Special Ed

January 4, 2009

It’s all too common but also completely mistaken to blame special education for the shortcomings of the public k-12 system.  If you point out that per pupil spending has more than doubled in the last three decades (adjusting for inflation) while student outcomes have remained unchanged, people blame the rising costs of special education.  (See for example Richard Rothstein on this).  If you point out that the teaching workforce has increased by about 40% in the last three decades (adjusted for changes in student population), people blame special education (see below).  If budgets are tight and programs get cut, people blame special education for draining money from general education. 

Blaming special ed is easy.  Most attempts to blame special ed don’t even bother presenting data or make the most crude use of data to support their claims.  Reporters simply accept assertions from school and state officials without question.  Folks accept the blame-special-ed-story so easily because — well, to put it bluntly — it is a a widely held but unstated prejudice.  People quietly resent special education because they fear that it is short-changing their regular education students.  They assume that money spent on disabled kids is necessarily money taken away from general education.  They can’t imagine that resources for general education have also increased at a very rapid clip even as special ed costs have risen. 

School officials — people who should know better — play upon this popular prejudice to rationalize their failures.  They would never dare blame the programs that have been created or expanded in the last three decades for the education of poor and minority students.  Those programs also cost quite a lot of money.  No, school officials choose to blame special ed because it seems like blaming fate.  Fate has overwhelmed us with a rise in disabilities, the story goes, which have drained general education of money, teachers, and flexibility under tight budgets.  Never mind the considerable evidence that the rise in special education over the last few decades is almost certainly due to an increased classification of students as disabled rather than a true increase in the rate of disabilities in the world.  Fate had nothing to do with it.

I’ve rebutted the claims that special ed is largely responsible for rising per pupil spending in chapters 1 and 2 of the book Education Myths as well as in this Education Next article and in this paper that was published in the Peabody Journal of Education. 

My purpose in this post is to address the comment written by “Kevin” that attributed the increase in the teaching workforce to special education.  Kevin was responding to a post by Greg in which he wrote: “But teachers’ unions have pushed up costs – dramatically. In the past 40 years, the cost of the government school system per student has much more than doubled (even after inflation) while outcomes are flat across the board. And this has mainly been caused by a dramatic increase in the number of teachers hired per student – a policy that benefits only the unions.”  And Kevin replied: “Any comparison of staffing in schools 40 years ago and today typically ignores one group of staff that didn’t exist in 1968 – special education teachers and aides. Special education programs weren’t in most schools 40 years ago, hence there were no staff hired to work with those specific populations, particularly students with cognitive delay and autism who need a much higher staff ratio than is provided in the general education classroom.”

I’m bothering to rebut Kevin’s claims because 1) he appears to be a state employee (perhaps a school official, judging from his email address), and 2) his comments are typical of the blame special ed rhetoric.  Notice that Kevin doesn’t bother to present any evidence.  He just tells a plausible story, which because he and many others have “pre-judged” it to be true, they consider persuasive without need of any proof.  But let’s consider the evidence here.

In 1974, the year before federal legislation governing the education of disabled students was adopted, there were 2.165 million public school teachers and an average student to teacher ratio of 20.8.  In 2006 there were 3.177 million public school teachers, an increase of 1.012 million teachers.  And in 2006 there were a total of 404,577 teacher FTEs providing special education services.

But we have to adjust for the fact that that some of those 404,577 teachers assigned to special education have been shifted (or had their lines shifted) to special education as more students have been reclassified as disabled.  We also have to adjust for the fact that there are more students in 2006 than in 1974.  To make everything comparable, let’s assume that the student-teacher ratio had remained at 20.8 for all students.  Given that there were 49.370 million public school students in 2006, there would have been 2.374 million teachers if ratios had stayed the same for everybody instead of the 3.177 million teachers we have.  So there was really an increase of 803,442 teachers, adjusted for the change in student population.

But if the 6,081,890 students classified as disabled also had 20.8 students for each teacher, they would have 292,398 teachers.  Given that there are 404,575 teachers assigned to special education, the lower student-teacher ratios required for disabled students only results in a net increase of 112,179 teachers (404,575 minus 292,398).  So, of the 803,442 teachers added since 1974 only about 112,179 can be explained by the need to offer smaller student-teacher ratios to disabled students.  That is, special ed may only account for about 14% of the increase in the teaching workforce.

What people like Kevin forget is that while virtually “no staff” were hired specifically for special education several decades ago, there were also virtually no students classified as disabled (although most were in schools and under-served).  If we shift 6 million students into special education and maintained the same 1974 ratio of 20.8 students per teacher, we would have shifted 292,398 teachers with them.  It’s true that with an increase in federal and state subsidies along with a mandate to provide services, we’ve reduced student teacher ratios for disabled students.  But we’ve only added an additional 112,179 teachers to produce smaller ratios for disabled students.

Of course, Greg also makes an excellent point when he says in the comments to his post that resources devoted to special ed should also be expected to produce improvements in results.  Regardless of how resources have been allocated between regular and special education, the money should be yielding benefits for students.  The fact that we have observed virtually no change in student outcomes over the last four decades despite a huge increase in real expenditures (regardless of how it was allocated) is a source of chronic frustration with public education. 

The unstated assumption of these blame-special-ed stories is that money spent on special education is basically money flushed down the toilet.  They assume that nothing can help disabled kids, which fuels the quiet resentment of resources devoted to special education.  Rather than looking for scapegoats — special education, rising poverty, cosmic rays, etc… — folks should focus on the perverse incentives of a broken public education system.  The fault, dear reader, lies not in our stars but in ourselves.

22 Comments | myth of helplessness, special ed | Tagged: , , , , | Permalink
Posted by Jay P. Greene

Separated at Birth?

December 31, 2008

madonna

4 Comments | random pop culture | Permalink
Posted by Jay P. Greene

Arkansas Beats Oklahoma

December 31, 2008

Arkansas basketball was expected to finish last in the SEC this year.  But they are off to a 10-1 start and beat #4 ranked Oklahoma last night.  Keep an eye on Arkansas point guard Courtney Fortson.  Woo pig sooiiee!

Leave a Comment » | college sports | Permalink
Posted by Jay P. Greene

Samuel Huntington (1927-2008)

December 30, 2008

I returned from vacation to learn that one of my graduate advisors, Samuel Huntington, passed away on Christmas eve.  Huntington was the type of broad intellectual that has become a vanishing breed in academia.  He had a knack for identifying the big themes that were worthy of our attention and had the courage to make bold arguments while always remaining respectful of those with whom he disagreed. 

Now we are mostly left with academics who dwell on the latest methodological technique rather than what is substantively important.  Just pick up a recent copy of the American Political Science Review and you will search in vain for anything important, useful, and accessible. 

And the public intellectuals who still attempt to ask the big questions too often give answers that have all the depth of a self-help book.  Has Thomas Friedman ever made an argument that was not already the bland conventional wisdom of the Rotary Club in a small midwestern town? 

Josef Joffe said it best: “But who will embark on projects of this kind of sweep, breath and depth? Or write as elegantly as Sam has done?  That’s over in American academia, as is that fabulous confluence between America’s rise to world power and the influx of some of Europe’s greatest minds, courtesy of Adolf Hitler. Never before has there been such a perfect match between the demand for and the supply of great talent. One hates to think what would happen to a young Sam today. He might still graduate from Yale at age 18, but would he have become a Harvard professor at age 23? With that independence of mind, that contrarian spirit, that relentless search for conventional notions to be slain? Would a young Sam still be able to ask the Big Questions? And sin against so many idols demanding fealty to contemporary standards of correctness?”

Huntington’s passing isn’t just the personal loss of a wonderful man, teacher, and scholar.  It also marks the end of an era.

4 Comments | tributes | Tagged: , , , , | Permalink
Posted by Jay P. Greene

« Previous Entries
Next Entries »