The Other (More Important) Value-Added Measure

February 26, 2013

(Guest Post by Collin Hitt)

Some teachers are better than others when it comes to raising test scores, which in turn can raise students’ earnings in adulthood. But test scores aren’t everything. A new study looks at whether individual teachers can have similar impacts on suspension rates, school attendance, GPA and even graduation rates. It finds that they can, and do.

To put the non-test score estimates into perspective, having an Algebra or English teacher 20 at the 85th percentile of GPA quality versus one at the 15th percentile would be associated with 0.09 and 0.054 higher GPA, respectively. For both subjects, a teacher at the 85 percentile of ontime grade progression quality versus one at the 15th percentile would be associated with being 5 percentage points (0.14σ) more likely to enroll in 10th grade on time. Given that not enrolling in 10th grade is a strong predictor of dropout, this suggests significant teacher effects on dropout…

That’s from Northwestern’s Kirabo Jackson, His working paper uses state-of-the-art value-added methods to identify North Carolina high school teachers who have significant impacts on test scores. He then uses the same methods to see which teachers have an impact on “non-cognitive” behaviors. One would expect – at least, I expected – that the teachers who raise test scores also raise non-cognitive outcomes. Not so.

For all outcomes, Algebra teachers with higher test score value-added are associated with better non-test score outcomes, but the relationships are weak…This indicates that while teachers who raise test score may also be associated with better non-test-score outcomes, most of effects on non-test score outcomes are unrelated to effects on test scores. The results for cognitive ability are consistent with this…

Results for English teachers follow a similar pattern. English teacher effects on English test scores explain little of the estimated effects on non-test score outcomes…

Because variability in outcomes associated with individual teachers that is unexplained by test scores is not just noise, but is systematically associated with their ability to improve typically unmeasured non-cognitive skills, classifying teachers based on their test score value added will likely lead to large shares of excellent teachers being deemed poor and vice versa.

So teachers can have large effects on matters that are supposedly out of their hands. Suspensions, absence rates and GPA are functions of a student’s conscientiousness – or more generally, of his character. This study delivers another blow to the cop out lobby.

But it also presents a huge challenge to the proponents of test-based teacher policies. Read this line again: “classifying teachers based on their test score value added will likely lead to large shares of excellent teachers being deemed poor and vice versa.”  This is not a trivial matter. Jackson shows that non-cognitive outcomes are more strongly correlated with life outcomes than are test scores – especially for students with limited cognitive ability. This paper cannot be ignored.

Hat Tip: Joanne Jacobs

[Edited to correct formatting error and typos]


True to Her Traditions – At Last

November 11, 2011

(Guest post by Greg Forster)

On Veterans Day two years ago I posted a sharp condemnation of my grad school for its contempt of the military, even in defiance of its own traditions. In the comments, I made a promise that I would be prepared to post something more cheerful for Veterans Day “when the Ivies quit spitting on the people who fight and die to preserve their right to spit on them.”

Yale is bringing back ROTC, along with Harvard and Columbia. Princeton refuses to budge. Brown is still considering its position. Cornell, Dartmouth and Penn had already brought it back before this year.

I’m not sure at what point my stated obligation to “post warm fuzzies about mom and apple pie” kicks in, and I’ll admit that I don’t think the series of events leading up to these developments generally augers well for civil/military relations. But that war is over and now is a time for reconciliation. Six of the eight Ivies now offer military training within their for-credit educational curricula. That is progress.


Abraham Who?

June 21, 2011

(Guest post by Brian Kisida)

Last week the Feds released the latest NAEP assessment of students’ understanding of U.S. history.  It contained a mostly negative assessment of history knowledge, including some tidbits like only nine percent of fourth-graders could identify a photograph of Abraham Lincoln and give two reasons why he is an important historical figure.  You know the drill: First act shocked that our students did so poorly, wring your hands a bit, blame your favorite thing/organization/political movement for creating this travesty, and then finish by lamenting the eventual end of democracy and civilized society as we know it (and plenty of people will also tell you the end of civilization can be avoided, of course, if we give schools additional resources or adopt national standards).  Everyone’s doing it, from the folks over at Fordham to Diane Ravitch.  Diane says she’s worried because when it comes to our high school seniors, “all of these students will be voters in a year.”  Well, not if 200+ years of voter-turnout data has anything to say about it.

Another annoying thing about all of the hand-wringing coverage generated by these types of reports is the way people discuss NAEP’s outcome measures, such as “Basic,” “Proficient,” or “Advanced” as if they’re entirely objective.  Here’s an excerpt from Ravitch’s statement on the issue:

“It’s worth noting that of the seven school subjects tested by NAEP, history has the smallest proportion of students who score Proficient or above in the most recent results available. Among twelfth graders, for example, only 12 percent reach Proficient in U.S. history, compared to 21 percent in science, 24 percent in both civics and writing, 25 percent in geography, 26 percent in mathematics, and 38 percent in reading.”

Or take, for example, the Boston Globe, which concluded from the same data that:

 “In fact, American kids are weaker in history than in any of the other subjects tested by the NAEP — math, reading, science, writing, civics, geography, and economics.”

It’s as if they think NAEP’s outcome categories were set by the International Committee on Weights and Measures using specific gravity and atomic clocks.  They aren’t.  They are arbitrary categories determined by “experts,” and they certainly aren’t comparable across subjects.  We can’t conclude that students are doing worse in history than they are in math or english simply by looking at proficiency rates.

The results are, however, comparable across time.  When viewed longitudinally, there are a few positives in this latest report.  Scores for eighth-graders were up across the board, and scores for Black and Hispanic eighth-graders were especially positive, significantly narrowing the White-Black test score gap. 

However, like we’ve seen time and time again with NAEP results, twelfth-graders aren’t budging.  And at the end of the day, if twelfth graders are stagnating then gains for eighth-graders are largely irrelevant.

To be honest, I think it’s difficult to guage the state of history education based upon NAEP’s measures, or based upon shoddy attempts by others to interpret them.  I really don’t know, for example, exactly how many fourth graders should be able to tell me the importance of Abraham Lincoln.  What I do know, and what I find disturbing, is that we have continued to allocate more resources for high school history over this same time period that high school scores have remained flat.  As the NAEP report points out, the number of schools offering A.P. U.S. history courses has risen from 51 percent in 1990 to 80 percent in 2009.  And the percentage of students who have taken an A.P. history class has more than doubled since 1990.  You would think that would lead to some observable gains for high-schoolers.


Neal’s Eternal Platonic Beauty Queens

June 21, 2011

(Guest post by Greg Forster)

In a provocative (as in thought-provocative) post, Neal reviews what the Miss America USA [oops] contestants have to say about teaching creation and evolution in schools:

I didn’t tally their responses, but just listening to the contenders it seems their consensus answer represents America in microcosm: Most seem to have serious doubts about evolution, but support teaching it along with other viewpoints. It reflects both the overall split within the American public—40 to 50 percent of Americans are creationists, and roughly the same segment evolutionists—as well as the consensus view on teaching human origins: About 60 percent of Americans support teaching both evolution and creationism in public schools…

The extent of the evidence is that government schooling generally can’t handle controversy, but that is almost never even mentioned in the seemingly endless war between creationists and evolutionists. And the same is true for the aspiring Miss USAs. While a few appeared to conclude that the nation is too diverse for public schools to deal with this topic—see Miss Kentucky at the 5:07 mark, and Miss Utah at 12:36—the majority made no mention of the problem.

One contestant said “I think we should leave that up to the government.” Maybe she could get a job teaching in this school:

Reading Neal’s insightful post brought me back to a subject that I promised him a few months ago I would treat when I had the chance. Neal and I agree that a government school monopoly can’t handle controversial issues and harms the national culture by undermining pluralism – real pluralism, not the phony kind you get by indoctrinating everyone’s children in a uniform (and politicized) ideology of what pluralism means.

Back during the late unpleasantness over national standards, I wrote that government monopolization of schooling is one of the root causes of the culture war, but (I said) “that’s an argument for another day.” Neal objected:

Here’s where I think Greg is incorrect: Choice is not an argument for another day. It is the argument for this day.

Until all parents have real, full choice they will have no option but to demand that higher levels of government force intractable lower levels to provide good education. It won’t work — thanks to concentrated benefits and diffuse costs all levels of government are dominated by teachers’ unions and administrators’ associations that will never let tough accountability and high standards rein – but it is all that parents can do absent the ability to take their children, and tax dollars, somewhere else. That means choice is essential right now, because it is the only way to take power away from special-interest dominated government and give it to the people the schools are supposed to serve. In other words, it is the only option that will actually work, obliterating the special-interest hammerlock, imposing accountability to customers, and when coupled with freedom for educators unleashing competition, specialization, innovation, and constant upward pressure on standards. In other words, it will do all those things that national standardizers emptily and illogically promise that their reform will do, and much, much more.

I promised him a response and never got around to it. This looks like the time.

Time is actually the basis of my response. I’m more or less with Neal on all the substance here. But getting the reforms we want requires us to participate in a social process that takes place within institutions and relationships, and which therefore has to unfold over time.

It seems to me that some people in the school choice movement (Cateaux!) have fallen into the trap of thinking that a successful movement for reform looks like this:

  1. Map out the entire architecture of how the reform looks across all issues, institutions and areas.
  2. Distill this architecture into a single integrated blueprint that fits everything together.
  3. Fight to implement your blueprint. Treat anyone who doesn’t subscribe to your blueprint as an opponent.

This reflects, I think, too much emphasis on the cognitive – as though there were a Platonic Form of Education Policy and our job is to contemplate it until we grasp its unchanging eternal nature, then subdue the world of time and change until it matches the eternal unchanging Form. In this Platonic dualism, the changeless and eternal (in  this case, the perfect education policy blueprint) is the only thing we value for its own sake, and the world of time and change is valued only as raw material for constructing tangible manifestations of the intangible Forms.

Neal’s insistence that all issues must be settled right now reminds me of a scene from C.S. Lewis’s The Great Divorce, which is a book that really reflects the Platonic view (especially as channeled through Augustine, who took Plato mainstream). In the book, Lewis describes a dream in which he sees people’s entire lives distilled down to a single moment of choice between good and evil.

In one scene, a Bright Spirit offers a man eternal bliss if he will turn away from his evil desires. He keeps making excuses. The Bright Spirit offers to kill his evil desires and he says: “There’s time to discuss that later.” The Bright Spirit replies: “There is no time.” A little later he says, “Some other day, perhaps” and the Bright Spirit replies: “There is no other day. All days are present now.” He demurrs but offers to return later: “I’ll come again the first moment I can.” The Bright Spirit replies: “This moment contains all moments.” This is essentially the Platonic/Augustinian approach to life, death and the afterlife – your life “down below” in the world of time and change was really just the ephemeral expression of a single, trans-temporal choice between good and evil. This view feels transcendent and liberating to people who have a certain highly cognitive bent of mind, but it’s actually reductive and limiting.

Neal’s insistence that we can’t talk about national standards without also talking about school choice feels the same way. We’ll never make progress on any issues if we can’t discuss one issue without simultaneously discussing all the others. Or, to put it another way, we can’t win any battles if we think that they’re all really the same eternal trans-temporal battle, with little manifestations of it fought over and over again within time. Because that way of thinking forces you to isolate yourself from potential allies who may not be interested in your entire blueprint (ahem) but who may be useful coalition partners.


Speaking of Moynihan…

May 26, 2011

(Guest Post by Matthew Ladner)

I get another assist from DPM in responding to an editorial in the Minneapolis Star Tribune. Here’s the punchline:

Minnesota lies in the heart of Big Ten country, where people have long taken justifiable pride in their K-12 scores and the academic prowess of their universities.

The favorable demographics alluded to by Sen. Moynihan, however, have masked a growing problem: Minnesota suffers from the largest racial achievement gaps in the nation.

A system of schooling that gives the least to those starting with less is unworthy of the traditions and ideals of Minnesota.

Liberals and conservatives should work together with educators to fiercely pursue radical improvement in literacy skills. The students with the least have the most to gain.

While I am on a Moynihan kick, I may as well note that I love the DPM quote that RedefinED keeps as a permanent feature on their blog:

Diversity. Pluralism. Variety…We treasure these values, and I do not believe it excessive to ask that they be embodied in our national policies for education.

…and hopefully in our state policies as well!


Little Rock Deseg Decision Marks Generational Change

May 20, 2011

The remedies favored by the previous generation of civil rights activists are being rejected by a younger generation that has witnessed the ineffective nature and corrupting influence of pouring more and more money on school districts that fail to properly educate low-income African-American students.

In a boldly-worded decision yesterday, U.S. District Judge Brian S. Miller ordered the virtual end of $70 million in annual state deseg payments to the Little Rock, North Little Rock, and Pulaski County Special districts.  Here is a snippet from the Arkansas Democrat Gazette‘s coverage:

Miller wrote that, after listening to hours of testimony and reading thousands of pages of court submissions, it became clear to him that the state’s payments to the districts had become a reward for poor performance.

“The problem with this process is that it results in an absurd outcome in which the districts are rewarded with extra money from the state if they fail to comply with their desegregation plans and they face having their funds cut by the state if they act in good faith and comply,” Miller wrote.

“It seems that the State of Arkansas is using a carrot and stick approach with these districts but that the districts are wise mules that have learned how to eat the carrot and sit down on the job,” Miller continued. “The time has finally come for all carrots to be put away. These mules must now either pull their proverbial carts on their own or face a very heavy and punitive stick.”

It was also striking how the racial pandering that may have worked on the previous generation of African-Americans no longer works.  As Judge Miller explains, it is downright insulting.  Again, here is a longer clip from the Dem-Gaz’s coverage:

He said it appeared to him that few if any of the participants in the case “have any clue how to effectively educate underprivileged black children.” He said some participants in the desegregation case seem to believe there is a “magic spell that will do the trick, such as some special racially based formula or program.

“Even more concerning, however, is that it seems that some of the participants do not really care,” he said.

Miller criticized some of the witnesses and evidence presented to him last year, saying that some testimony was not relevant, or it was based on flawed logic, or that it was “appalling.”

He said some of the witnesses viewed the judge’s role as that of the innkeeper in The Canterbury Tales who had the job of determining which of the travelers at his inn told the best story.

Miller cited court testimony about seemingly unfair differences in pay for basketball and football coaches, which he said was an inaccurate analysis because football coaches are required to work during the summer and their contracts reflect that extra time.

“This truth, however, did not stop minute after grueling minute of mindless testimony about the injustice suffered by the basketball coach,” Miller wrote.

He also called attention to a witness who said teachers let their students “rap” in class as a way to promote reading and speaking English correctly.

“Although, at first blush, it might seem understandable for this witness to assume that a middle aged black judge would find this appealing, that presumption is simply untrue,” wrote Miller, who is black and said that doing rap does not necessarily lead to literacy or speaking correctly.

Of course, the three Little Rock area school districts have no idea how they will function without the extra $70 million they get from the state each year and have made no sensible contingency plans for the day when the deseg gravy train eventually had to end.  They’ve received over $1 billion in extra funds since 1989 and, according to the Court, have virtually nothing to show for it.

If you would like to see the entire ruling, click here.

(edited for typos)


Big Lunchlady Is Watching You

April 13, 2011

(Guest post by Greg Forster)

Theorists like Amy Gutmann argue that parental freedom needs to be compromised in the name of democracy because parents can’t be trusted as the default authority over the education of children. Jay has frequently responded by pointing out that this logic, applied consistently, would produce not just government control of formal schooling but government control of every aspect of child-rearing. One example I’ve seen him use to devastating effect is to point out that we don’t establish government control over children’s meals in order to ensure kids are getting proper nutrition. Jay suggests that this inconsistency indicates that these theories of democracy are really invented post facto to justify social institutions whose real existential principle is to provide unions with a gravy train.

Well, Jay, you should be careful what you ask for.

The Chicago Tribune reports that some Chicago schools – a government spokesperson declines to say how many – forbid students to bring any food from home unless they have a medical excuse.

Principal Elsa Carmona said her intention is to protect students from their own unhealthful food choices.

“Nutrition wise, it is better for the children to eat at the school,” Carmona said. “It’s about the nutrition and the excellent quality food that they are able to serve (in the lunchroom). It’s milk versus a Coke. But with allergies and any medical issue, of course, we would make an exception.”

Carmona said she created the policy six years ago after watching students bring “bottles of soda and flaming hot chips” on field trips for their lunch. Although she would not name any other schools that employ such practices, she said it was fairly common. [ea]

The Tribune headline writer makes an amusing attempt to soften the obvious implications here – the headline says the school forbids only “some lunches” from home. The actual policy described in the article is that all food from home is banned unless you challenge the ban and have a special medical reason.

Most readers of JPGB probably won’t need to have the real agenda spelled out here. Kudos to the Trib writers, Monica Eng and Joel Hood, for spelling it out to the paper’s readers:

Any school that bans homemade lunches also puts more money in the pockets of the district’s food provider, Chartwells-Thompson. The federal government pays the district for each free or reduced-price lunch taken, and the caterer receives a set fee from the district per lunch.

This lunchroom needs a better class of criminal.

It’s the same basic principle that has been driving the runaway overhiring of teachers for decades. It just involves the extension of the principle to a new sphere of social control.

HT Joe Carter at First Things


Who’s the Criminal?

January 26, 2011

In Akron, Ohio a woman who put her children in a better public school was sent to jail when private investigators hired by the school found that she did not live in the district.  Her father did and she sometimes stayed with him, but that was not enough to keep her out of prison for seeking a better education for her children.

Meanwhile, in Atlanta there is evidence of widespread cheating on standardized tests by teachers and administrators as well as a potential cover-up in the investigation of those accusations.  No one has gone to jail (and no one ever will) for robbing children of a quality education and then lying about their true achievement by cheating on the state test to hide that fact.

A few years ago Atlanta and other Georgia districts violated the state law to prohibit the social promotion of students who failed the 3rd grade reading test.  There was a procedure for exempting students if the schools and parents met and decided it was in the best interest of a student to be promoted, but many districts exempted virtually all of the students and did so without actually holding the required meetings.  They already knew what was best for children regardless of what the law said.

I could keep going with stories along these lines, but I think you get the idea.  So, who’s the real criminal here?


Acting White

November 22, 2010

Stuart Buck, the University of Arkansas graduate student and author of the well-reviewed book, Acting White, suggests that high academic achievement for African-American students is hindered by negative social pressure from peers.

Now Dan Willingham reviews a new study on the subject:

It used a sample of over 13,000 students, averaging about 15 years old. Social acceptance was measured with a simple 4 question interview that asked whether they felt socially accepted, and the frequency with which they felt lonely, felt disliked, or felt people were unfriendly to them.

The study took measures at two time points and examined the changein social acceptance across the year. The question of interest is whether students’ academic achievement (measured as grade point average) at Time 1 was related to the change in social acceptance over the course of the year.

For White, Latino, and Asian students, it was—positively. That is, the higher a student’s GPA was at Time 1, the more likely it was that his or her social acceptance would increase during the coming year. It was not a big effect, but it was present.

For African American and Native American students the opposite was true. A higher GPA predicted *lower* social acceptance during the following year. This effect was stronger than the positive effect for the other ethnic groups.

Thus, it seemed that the simpler version of the “acting white” hypothesis was supported.
But the story turned out to be a bit more complicated.

Further analyses showed that there was a social penalty for high achieving African Americans *only* at schools with a small percentage of black students. The cost was not present at high-achieving schools with mostly African-American students, or at any low-achieving schools.

At the same time, there was never a social benefit for academic achievement, as there was for White, Latino, and Asian students.

These more fine-grained analyses were not possible for the Native American students, because the sample was too small.

So what are we to make the of “acting white” phenomenon?

A single study is never definitive, but this study indicates that academic success is not universally taken by African American adolescents as a sign of rejecting African American culture. It is specific to particular contexts and is plausible a response to discrimination.

Sounds like this mostly supports Stuart’s argument but I’m curious to hear what he thinks.


The 21st Century Will Be HUGE!

October 20, 2010

‎(Guest post by Patrick Wolf)‎

Schools of the 21st Century need to do lots and lots of things.  That is the message from local experts impaneled by the teacher education sorority here at the University of Arkansas.  Summarizing the guidance from the panel, the Northwest Arkansas Times writes:

Students today need stronger foundations in foreign languages, physical education and the arts to participate and compete in a global economy, several panelists said Tuesday.

At the same time, schools need to be more responsive and tolerant to diversity, focus on higher educational standards and to be more effective in the use of data to better understand student learning, said Springdale School Superintendent Jim Rollins.

Schools also need a stronger foundation in multiculturalism because by 2030, research indicates more than half of all students will be non-white, Rollins said.

Later in the article, a professor is quoted as saying that our elementary and secondary students need a strong foundation in physical education, since they may end up working in a foreign country where people have to walk a lot.

I support foreign languages (though I speak none fluently), physical fitness (though I am a bit hefty), the arts (though I can’t draw to save my life), diversity (though I am a straight white male), high education standards (though I have a Ph.D. from Harvard), and data-based decision making (though I don’t always follow the projections when setting my fantasy football lineup).  These are all desirable things for students.  But are they all equally desirable?  Don’t we need to prioritize and make tradeoffs?  After all, children are only in school an average of 6.5 hours out of each school day.  If everything is important, isn’t nothing important?

The laundry list of supposedly required 21st Century skills articulated by the panel reminded me of the “Mountain of a Man” in Monty Python’s The Meaning of Life.  Upon entering a fancy restaurant, he refuses the offer of a menu, instead ordering plate after plate of everything.  What does the 21st Century diner need in a meal?  He needs steak, chicken, salmon, pork chops, tofu, arugula, tomatoes, yams, French fries, paella, gumbo, wienerschnitzel, lutefisk, roast turkey, pumpkin pie, carrot cake, and some more steak and salmon for good measure.  He also needs a larger restaurant staff to prepare all of these foods and serve them to him during his meal.  Can you say, “administrative bloat?”

The result of lacking discipline in selecting foods to eat is obesity and, in the movie, the Mountain of a Man explodes after a waiter insists that he cap off his bacchanal meal with “a wafer-thin mint.”  The clear message is that we have to make good choices and set priorities.  We can’t have everything.  As John Chubb and Terry Moe originally argued in Politics, Markets, and America’s Schools, a problem with education policy set by local experts through political bodies like school boards is that everyone has their own educational hobby-horse.  Each contributor to policy making will insist that their pet program get adopted.  The result is a curriculum like the Mountain of a Man — massive, bloated, and utterly lacking in discernment.  Eventually something has to give.

How do we set educational priorities if lots of things are worth learning?  One solution to the curricular and administrative bloat that comes from local experts and officials designing education is to allow the educational programs of schools to develop organically based on feedback from the choices of parents.  As parents we want our students to develop many skills and master many subjects, but we also realize that the opportunities are not limitless and children need to focus on what is important for them.  Parents who strongly value the arts would gravitate towards schools that emphasize the arts.  Parents who instead think that science and math are the most important subjects would be attracted to ESTEM schools.  Parents who feel that physical education is the most important skill for the 21st Century would send their children to ancient Sparta.

Schools would adapt to the choices of parents, offering more programs in areas of excessive demand and fewer programs in areas of lesser demand.  Although education experts might warn us that curricula driven by parental choices might be intolerant of diversity and devoid of rigor in areas such as reading, writing, math, and science, I have seen no evidence that such worries are valid.  The research on school choice and tolerance actually indicates that tolerance and a variety of other civic values tend to increase when parents are allowed to choose schools.  I’ve spoken to many low-income parents in focus groups and all of them want their children to develop mastery of core competencies.  Many, but not all of them, also express a desire for their child to learn a foreign language or develop as an artist or musician.

The point is that parents have views of what educational programs are best for their children that differ from each other and from many of the experts, but those views tend to gravitate towards rigor in traditional and important academic areas.  If we let parents choose schools, thereby pressuring schools to provide programs that are responsive to those preferences, we can’t be certain what we would get but it probably would be quite reasonable.  Our children wouldn’t get everything on the menu, but what they would receive from a 21st Century education driven by parental choice would likely be both nutritious and delicious.  Plus, nobody would have to explode.