Jay P. Greene's Blog

(Guest post by Greg Forster)

I have a post at OCPA on taking ignorance seriously:

Some states managed to have success in some cases—Massachusetts’ standards reforms and Florida’s mix of school choice, exit exams, and incentives to raise test scores across demographic groups are notable examples. But the overall story was failure. We just couldn’t take these good ideas to scale…

Why was school choice the only winner? Because it takes our ignorance seriously. It doesn’t try to generalize the content of education across millions of unique children.

Throwback to my review of the evidence on Pre-K included at no extra charge!

Education policy research is not really a scientific enterprise.  If it were, the field would be equally open to accepting research of equal rigor regardless of the findings.  That is simply not the case.  Research with preferred findings is more easily published in leading journals and embraced by scholars than research supporting less favored results.

There are countless examples of this, but here is one to illustrate the point…

The Journal of Policy Analysis and Management, a top journal in our field, has just published an analysis of vouchers in Indiana based on a matching research design.  Despite the fact that matching is normally intended to produce treatment and comparison groups that are nearly identical on observed characteristics, in this study the treatment group differed significantly from the control group in their pre-treatment measure of math performance.  Specifically, the treatment group has significantly higher scores on math tests.  And the one negative effect observed by the study was on math test scores, which was roughly comparable in magnitude to the amount by which the treatment group was higher on math scores pre-treatment.  So, basically the treatment group reverted to having about the same math scores as the control group once treatment began.  This negative effect, which was really the equalizing of the matched groups, was detected the first time students enrolled in a private school and did not grow in magnitude as students persisted in private school.  One might think that if private schools really harmed math scores, that harm might compound over time, but that did not occur.

These results certainly deserve publication and ought to inform the school choice policy debate despite the obvious limitations of the matching design that failed to make the groups comparable on the one outcome measure for which a negative outcome was observed.  While worthy of publication and discussion, it is questionable whether this article deserves publication in one of the field’s top journals and even more doubtful that it should be given as much credence as some folks in the field seem willing to give it.

Corey DeAngelis and Pat Wolf have a similar school choice study based on a matching research design with similar imperfections.  It examines whether students enrolled in the Milwaukee voucher program were more likely to be accused or convicted of a crime in later years than comparable students who had attended Milwaukee’s public schools.  Students in the treatment group were matched to public school students on a number of observable characteristics, including the neighborhood in which they lived.  Despite that matching effort,  the treatment and control groups were significantly different, with the treatment group having higher reading scores and more likely to be female.  Unlike the JPAM study, neither of these variables were the same as the outcome for which they observed effects.  Controlling for observable student and parental characteristics, students who had enrolled in Milwaukee’s voucher program were significantly less likely to be accused of a crime in later years.

The defects of Corey and Pat’s study are similar to those of the JPAM study.  It also uses a matching research design, and as I have said many times before, I don’t think we should have much confidence in matching designs to produce causal inferences.  And like the other study, Corey and Pat’s matching fails to produce treatment and control groups that are similar on all observed characteristics.  But unlike the other study, Corey and Pat’s research is not being published in JPAM.  In fact, JPAM desk rejected Corey and Pat’s study, deeming it unworthy even of being sent out for review.  A number of other journals did the same and they are now struggling to get it published in any journal.  I’m convinced that if only they had found that vouchers increased criminal behavior, their piece would already be in print in a respected journal.  But because they found a positive result for vouchers, the bar is higher and editors and reviewers can rightly note the defects in the study to justify rejection.

All research has limitations that might be invoked to support rejection or overlooked to support publication.  The double-standard used when judging voucher studies with favorable or unfavorable findings is a function of political bias and is an indication that our field is much less scientific than we would like to imagine.

It’s a shame that education policy researchers are largely uninterested in this problem of political bias.  Despite considerable energy devoted to promoting many dimensions of diversity within our field, there is virtually no effort to promote ideological diversity.  My department has a few researchers who would describe themselves as conservatives (while we also have had two faculty members who describe themselves as socialists), but I suspect most departments don’t have any self-described conservatives while others have no more than one or two.

It is interesting to note that despite having a department with six endowed chair holders, half of whom have Harvard doctorates, and all of whom have impressive research records, none of us have ever been asked to serve on the editorial boards of any journals (excluding the Journal of School Choice that my colleague, Bob Maranto, edits).  We’ve tried to play a part in governing our profession, but because we are branded (sometimes incorrectly) as conservatives we have been shunned.  The composition of editorial boards shapes who reviews submissions, which shapes what is published in those journals, which shapes what people in the field imagine the research consensus to be on various issues.

There are consequences to this political bias in our field.  First, the scientific quality of research is harmed by an increasing groupthink that fails to critically examine the key assumptions, methods, and implications of much of the work being produced.  Second, research in the field has diminished credibility and policy influence because others increasingly look at the field as more ideological and less scientific.  Some of the leading people in our field regularly take to Twitter to deride policymakers and the public for failing to heed what they believe research has to say. But why should policymakers obey “science” when it is being produced by an increasingly insular group of researchers who may confuse their political agenda for science? Third, frustrated conservatives are likely to give up trying to be accepted by the dominant professional associations and journals and instead build their own parallel institutions.  The Bar Association drove out conservatives who built the Federalist Society, which now seems to be thriving more than the “mainstream” organization at exercising policy influence.

I don’t expect this piece to alter this state of affairs.  Leading scholars in our field seem quite adept at defending their prior convictions, sometimes in remarkably unscholarly ways on social media, rather than critically examining their own beliefs and behaviors.  As far as I’m concerned they can rail away, but they will be left with the kind of nasty, unscientific, and irrelevant field they seem determined to build.

The Arnold Foundation’s Straight Talk On Evidence web site provides a very useful summary of a recently published large RCT on a state-funded pre-K program in Tennessee.  Consistent with a previous, nationally representative RCT of Head Start, this study found that students given access to government-funded pre-school by lottery initially score higher than those who lose the lottery on standardized test scores but then fare worse later.

In the TN study, treatment students score higher at the end of pre-K.  But, as the Arnold summary puts it:

At the end of third grade, the study found statistically-significant adverse effects on student math and science achievement. In math, the VPK group scored 0.12 standard deviations lower than the control group, which equates to roughly 13 percent less growth in math achievement than would be expected in the third grade year.[ii] In science, the VPK group scored 0.09 standard deviations lower than the control group, which equates to roughly 23 percent less growth in science achievement than would be expected in the third grade year.[iii]

In an effort to explain the negative longer-term result, the authors suggest that special education may be to blame.  Students admitted to the government-funded pre-K program were more likely to be labeled as needing special education services and that designation may have lowered academic expectations.  But this explanation is inconsistent with Hanushek, Kain, and Rivkin’s finding that special education tends to improve test score results.  Straight Talk at least considers the possibility that children being with family or in a non-government-funded pre-school may just be academically superior.

The hard reality is that the process of human development is complex and highly varied, so we just don’t know the optimal arrangements for all children.  Andy Smarick has an excellent piece along these lines in the Weekly Standard, suggesting that education policy experts suffer from a Hayekian information problem.  And this was also the subtext of my post last week on how parents are smarter than Technocrats.  Even when Technocrats are armed with the best science, they generally do not have enough information to centrally plan the lives of others.  This doesn’t mean that we never regulate anything.  It just means that if we do regulate we should do so with great caution and large dollops of humility because the experts are typically missing a lot of important information that the individuals they are regulating are more likely to posses.

But caution and humility are no fun, so the Arnold Foundation’s Straight Talk chooses instead to double-down on Technocracy by suggesting that the disappointing results of pre-school as shown in RCTs of both Head Start and the TN program be remedied by identifying which subset of pre-schools seem to be more effective and regulating programs toward imitating those schools:

The above findings and observations, we believe, underscore the need to reform programs such as VPK and Head Start by incorporating (i) rigorous evaluations aimed at identifying the subset of local approaches that are effective, and (ii) once such approaches are identified, strong incentives or requirements for other local program sites to adopt and faithfully implement them on a larger scale.

Keep in mind that the TN program already has regulations in place meant to ensure quality, including requiring at least 5.5 hours of instructional time per day, a cap of 20 students per classroom, a licensed teacher in each classroom, and the requirement that schools choose among a state approved set of curricula.  Also keep in mind that short-term test scores, which are the most common tool by which regulators monitor quality, showed positive results.

If these regulatory practices are insufficient to avoid harming students over the medium term, why would Straight Talk believe that doubling down on the Technocratic approach would make things better?  It would be nice if they at least considered the possibility that we are suffering from a Hayekian information problem and may be unable to devise optimal arrangements for education.

The technocratic brand of ed reform that is currently dominant is based on the premise that policy elites, guided by science, need to ensure school quality.  Parents should have choices, but they should only choose among quality options.  Mostly using test scores, technocrats believe they can identify quality schools and quality-promoting educational practices, which should over-ride parental preferences about which schools and practices offer a quality education.

A new study by Diether W. Beuermann and C. Kirabo Jackson suggests that parents may be better at detecting which schools promote long-term positive outcomes for their children than technocrats guided by short-term test scores.  They examine the school system in the Barbados in which parents seek admission for their children into schools they prefer, but those schools use test-score cut-offs to determine which students gain admissions.  The cut-offs create a discontinuity that allows for a rigorous causal identification of whether students who barely gain admission to a desired school have different outcomes than those with barely lower lower test scores who are denied admission.

They find that test score gains are no greater for students who were admitted to the schools their parents preferred than those not admitted.  For boys there are some signs that the effect on test score gains may actually be negative.  But when they look at longer-term outcomes, including educational attainment, employment, and earnings, they find significant benefits for students who were admitted to the schools the parents preferred.  These positive effects were driven mostly by gains for girls.  When they explore mechanisms for why these gains occurred, they find a significant reduction in teen motherhood for girls admitted to preferred schools, which contributed to their educational attainment and later employment and earnings.  They also found that both boys and girls experienced significant long-term health benefits as measured by a healthy BMI, regular exercise, and dental check-ups if they gained admission to the schools their parents preferred.  The researchers conclude: “This suggests that preferred schools may promote productive habits and attitudes that are not measured by test scores but contribute to overall well-being. This may represent a significant, previously undocumented, return to school quality.”

So, parents, on average, could detect important aspects of school quality that technocrats guided by test scores would get wrong.  Technocrats would conclude that the schools that parents prefer do nothing to improve student outcomes because test scores don’t rise or even go down when students get into the school their parents want.  But parents are smarter than the technocrats.  They prefer schools that improve long-term outcomes for their children.  Specifically parents seem to be able to choose schools that are more effective in developing the “character” of their children, making the students less likely to get pregnant as teens and more likely to be engaged in positive health behaviors later.  For boys this may not make a big difference in the labor market (although it does not harm those outcomes), but for girls these health improvements seem to drive higher educational attainment, employment, and earnings.

This study is consistent with a long line of research that finds a disconnect between short-term test score outcomes and long-term life outcomes, as described in a recent meta-analysis by my colleagues, Mike McShane, Pat Wolf, and Collin Hitt.  It’s amazing to me how champions of the technocratic approach continue to have faith that they have access to scientific tools to identify school quality that less well-informed parents lack despite the growing body of scientific evidence that demonstrates the very real defects of the technocratic approach.  Despite their daily hymns of praise to science, the technocrats don’t seem very scientific at all.

(Guest post by Greg Forster)

In my latest for OCPA, I look at how Oklahoma’s religious left is baptizing the blob, including support for the teacher strike.

It’s not my position that religious leaders should have nothing to say about education policy:

It’s true that America’s great experiment in religious freedom implies our public policy can be based on shared moral commitments even if we disagree about the ultimate cosmic basis of those commitments. But as George Washington rightly pointed out in his farewell address, we can’t talk only about the morals of public policy and ignore the religious foundations of the morality upon which we draw. For if the foundations are neglected, the building collapses.

But if religious leaders are going to speak about education policy, they should make a serious theological argument and not just parrot the political talking points of secular special-interest groups. Otherwise they end up captive to political manipulators. This is exactly what happened to the religious right:

As a matter of fact, I’ve spent almost 10 years speaking out against the ideological captivity of the religious right. I appreciate that the fight for the sanctity of human life and other issues has accomplished some good. But the larger effect of the religious right movement was to push churches to become voter registration offices of the Republican Party. As it became clear what was going on, this did incalculable damage to the religious credibility of the churches involved. We are still living in the disastrous aftermath, as huge portions of our culture have disconnected themselves from faith entirely.

So I’m only playing fair when I say that I see the same dangerous sellout in the efforts of Oklahoma’s religious left to baptize the blob. The pronouncements of Oklahoma’s religious left on education don’t bring any theological light to the public policy questions. They’re not saying anything the secular left isn’t saying. They’re just pasting Bible verses on self-interested interest group politics. Organizing events and statements to support a secular special interest’s demand for money, parroting its secular talking points, doesn’t become a spiritual discipline because you do it with a clerical collar on—quite the reverse.

There are, in fact, serious theological arguments to be made on education policy. I’ve participated in some of them, including my response to theological arguments from the religious left as well as theological arguments from the religious right. So I welcome – though I often disagree on the merits – real theological arguments from the religious left and right. What’s alarming is when religious leaders make themselves tools of secular selfishness in the name of, yet to the detriment of, better schools for kids.

(Guest post by Greg Forster)

This post contains mega major spoilers. I mean it. You have been warned.

Okay, let’s get the tiresome part out of the way first. Pass the Popcorn hereby presents:

EVERYTHING WRONG WITH INCREDIBLES II

• Ridiculously patronizing fan service: A visit to Edna, with the conveyor belt and everything, is shoehorned in; Frozone gets yelled at by Honey
• Act I totally undermines the psychological climax of the first movie: They put on their masks to fight the Underminer – they are now supers again, and proud of it! But then we needed to make another movie, so suddenly they’re not.
• The visual look of this movie is just different enough from the original that you constantly notice it – especially Dash, did no one go back and review what he looked like in the first movie? – but not so different that your brain accepts it as different. It’s the animation version of the uncanny valley.
• The villain’s back story is insultingly contrived: It would have been sufficient if she’d just hated supers for the reasons given in the big Screenslaver speech, which would have made her a really interesting political/ideological villain with a megalomaniacal vision of reshaping the world by force; but no, it all had to go back to a ham-handed story about a personal trauma, because we’re all babies now.
• Mind control is always bad: It negates the only part of any story that’s really interesting: the characters’ choices and struggles.

What bugs me is not that these are huge problems, because they aren’t, but that they would have been so easy to fix if the studio had respected our intelligence just a wee bit more. I think these issues collectively made the difference between I2 being a really good movie (which it is) and a really great movie, on par with the original (which it isn’t, alas).

Whew! Now let’s get to the fun part.

I2 continues the Pixar/New Disney tradition of goring our cultural sacred cows, but doing it in just the right way so people will take it. In this case, as in some others (see: Frozen) it’s done by giving us a real encounter with the reason people believe in those sacred cows – the other side gets a full airing of its case before the movie pulls the rug out.

The Incredibles franchise, here as in the first movie, takes on two big cultural dimensions at once. Which is what would have made I2 a really great movie if they hadn’t fumbled too much of the small stuff, since taking on just one is tough enough.

The first dimension is the male ego. In the first movie this was simply (simply!) the conflict between Mr. Incredible’s longing for the glory days and the prosaic task of being a father. He must learn that parenting is heroic. “You are my greatest adventure, and I almost missed it!”

That theme is echoed in I2 by having Mr. Incredible become a full-time dad to three unruly kids with superpowers, the comedic value of which is expertly milked. And it leads, halfway or so through the movie, to the same conclusion – parents are the real heroes.

But now there is a new twist. The male ego comes up against the female ego.

Mr. Incredible is not just invested in his own professional success – his glory days. Now he’s also deeply threatened by the possibility that his wife might outshine him.

All civilizations always needed to mortify the male ego sufficiently to make men into good fathers. But in the new social order that is emerging after feminism, the male ego must be further mortified to make room for female competition in the workplace – without the compensatory satisfaction of the paternal role being valued in the same way the maternal role is valued. Whether that is sustainable is an open question, but if it is, it will only be so if men learn heroic virtue similar to the heroic female virtue Tocqueville praised as the foundation of the American regime. Tocqueville said (in substance) that the American experiment in constitutional democracy avoids degeneration into atomistic indiviudalism only because its women had not demanded the equal rights to which they were clearly entitled under the governing principles of the regime; if they ever did demand those rights, he warned, the regime could not deny them, but the result would be the collapse of the traditions by which the family rather than the atomized indiviudal is the basic unit of society. That, in its way, is one of the lessons of I2 – only men of heroic moral virtue can sustain the new social order feminism has catalyzed.

You see what I mean about goring sacred cows? But we’re not done yet.

The female ego comes in for a subtle but no less sharp skewering in I2. From the moment they meet, Evelyn begins stroking Helen’s female ego – her sense of resentment and exclusion in a man’s world – in order to get under her defenses and take advantage of her. And Helen falls for it hook, line and sinker. The two of them spend half the movie just stroking each other’s female egos, right up to the point where Evelyn plunges in the knife. Like Iago worming his way into Othello’s trust by flattering his male ego and then twisting that very ego to his own purposes, Evelyn has used Helen’s feminist pride to destroy her.

The other big social topic is of course the role of superheroes.

In the first movie, Syndrome spoke for the envy and resentment of all those who hate heroes – and they have been a prominent and influential voice throughout the modern period of history. So pervasive is this resentment that Helen herself parrots it without really thinking – “Everyone is special, Dash.” The modern period could almost be defined as the period during which it became plausible to say that it is evil to admire heroes.

I2 tries to pick that thread back up. In the Screenslaver speech, the critique of superheroes is even broadened into a critique of the Big Media and Internet culture for which superhero franchises are a sort of proxy. “You don’t talk, you watch talk shows; you don’t play games, you watch game shows.”

The makers of I2 have seen that under the surface of the standard-issue snobbery about mass media we constantly hear is an aristocratic (or worse) contempt for democracy and egalitarianism. Evelyn’s attack on bourgeois society strikes the note of every totalitarian ideology: You’re all SHALLOW and WEAK and LAZY. That’s why mass media is nothing but a tool of social control – you’re all so easily controlled because you’re pathetic and worthless. WAKE UP, SHEEPLE!

This is all botched, however, because it does not remain at the center of the villain’s character and motivation. The contrast with Syndrome is instructive. Syndrome has a personal trauma to provide resentment and a motive for megalomania, yes. But Syndrome’s revenge consists of remaking the world according to a new political vision, a vision whose principles he understands and articulates as a coherent ideology. This is precisely what makes him interesting. “When everyone is super, no one will be!” If Evelyn’s big “monologuing” scene with Helen in the frozen chamber had been a further elaboration of that political vision and not the recitation of a mind-numbingly boring series of convolutions designed to give her a stronger personal trauma, she could have ranked with the best of the Bond villains.

But in spite of this misfire, I2 does remember to pull the rug out from under Evelyn’s ideology. And the character who does it is Violet.

Let’s face it, there’s a great deal to be said for the critique of bourgeois society as shallow and morally undeveloped. But the bourgeois society is a little like Winston Churchill’s democracy – it’s the worst kind of society, except for all the others that have been tried from time to time.

The self-appointed superiors who look down upon the shallow complacency of bourgeois society never actually rise to moral heights. Time after time, in arrogance and contempt, they sink to moral depths.

Who is it that learns real humility? The supers, who – precisely because they have so much power – must learn to use it rightly or face catastrophe.

Violet spends the whole movie wanting to be a superhero. Even when she says she doesn’t, she only says it because she really does – and the sacrifices are so painful. And then, in the climactic moment, she chooses to stay out of the fray in order to protect her baby brother rather than seek glory and adventure.

And her father says, “that’s my girl.”