Jay P. Greene's Blog

My students, Charlie Belin and Brian Kisida, have a new article in the journal, Educational Policy, that examines the relationship between state science standards and science achievement according to NAEP.  As an indicator of the quality of state science standards they use Fordham’s ranking of those standards.  They find no relationship between Fordham’s ranking of standards and achievement.

Possible explanations for this result include:

1) Fordham is lousy at judging the quality of standards.

2) The quality of standards doesn’t matter.

I’m inclined toward the latter explanation, but either way, would it seem like a good idea to blow hundreds of millions, engage in endless and destructive in-fighting, and consume nearly all of the energy of the reform movement on something that makes virtually no difference?

I know, I know… the standards crowd readily admits that standards, by themselves, are not the issue.  It’s the way we link standards to teacher training, professional development, and assessments with consequences for teachers and students that really matters.  OK, so standards only matter if we also achieve a level of benevolent, top-down control over key aspects of the education system that has never been accomplished before.

Where have I heard this kind of argument before?  Oh yeah! That’s what the crazy guy in Harvard Square was yelling about when he said that the past failures of communism didn’t matter because it would finally work if we just did it correctly and completely.  And how much coercion and forced conformity would be required in the futile effort to achieve this level of top-down control?

Of course, it is also possible that Charlie and Brian’s analysis failed to capture the true causal relationship between standards and achievement given that is is only an observational study.  But if that is the case, the burden would still be on the advocates for national standards to demonstrate the causal connection between the reform they advocate and improved outcomes.  We shouldn’t remake all of American education on a hunch and a rationalization borrowed from the failure of communism.

The Wall Street Journal has an article that you might mistake for something in The Onion.  The federal government has a program that provides surplus military equipment to state and local governments at no cost other than the expense of shipping.  A number of public schools have taken advantage of this program to acquire military gear.  As the WSJ reported:

Some school districts, including the Los Angeles Unified School District, stocked up on grenade launchers, M-16 rifles and even a multi-ton armored vehicle…

And Matt will be pleased to know that:

In south Texas, near the Mexican border, the sprawling Edinburg Consolidated Independent School District has 34,700 students and operates its own SWAT team, thanks in part to military gear it was given in recent years.

This is a toxic combination of 1) school districts lining up for anything the feds are handing out, 2) the excessive militarization of local police (and apparently school security) forces, and 3) schools focusing on incredibly rare events, like school shootings, as opposed to incredibly common ones, like incarcerating millions of children in schools that fail to serve their needs.

But don’t worry, one school district official explained that “These officers are trained in tactics. Some are former military.”  And a Defense Department spokesperson assured that “each state is visited biannually for a program compliance review to further look at records, property and usage.”  Well… if these are trained school employees who are inspected by the federal government biannually, I’m sure it will be fine to have SWAT teams with grenade launchers in our schools.

My only question is when will public schools be able to get surplus nukes.  I mean, how else will they maintain Mutually Assured Destruction to deter the growing threat of private school choice?

Brian Kisida, Dan Bowen, and I have a new article in the journal, Sociology of Education, about how field trips to art museums help develop cultural consumers — people who want to visit cultural institutions in the future.  This piece is a more focused and technical follow-up on the summary of our art museum field trip study last year in Education Next.  Earlier we also published a more technical piece in Educational Researcher focused on the critical thinking results from our experiment.  There are more technical pieces on particular aspects of the study in the works.

Also keep you eyes out for a random-assignment study on what students learn from field trips to see live theater performances of Hamlet and A Christmas Carol.  It should be published by Education Next some time in October.

And if you are hiring, keep your eyes out for Dan Bowen and Brian Kisida, who are both now applying for academic jobs.  They each have at least 8 peer-reviewed articles.  Dan also has two grants that he earned while working as a post-doc at Rice University.  Another one of our graduate students, Anna Egalite, is currently in a post-doc at Harvard and is also on the academic job market for next year.  All thee would be great hires for any university smart enough to snap them up.

My colleague, Lisa Margulis, has a great Ted-Ed video summarizing her new book, On Repeat: How Music Plays the Mind.  Lisa, Brian Kisida, and I recently published an article together in the Psychology of Music on how student experience of a musical performance is altered by receiving information about the music.

Let’s see if this information alters your experience of music.  The question in Lisa’s book (and video) is why we like so much repetition in music.  Lisa provides the answers:

And in case you need an example, here are the Vulgar Boatmen playing Drive Somewhere, which I think captures repetitive pop perfection:

I came across a fantastic website called Postmodern Jukebox featuring wonderfully creative interpretations of pop songs, TV theme songs. and video game music.  Here’s their jazz interpretation of Meghan Trainor’s All About that Bass:

And here is Girls Just Want to Have Fun as a waltz:

And here is Blurred Lines as a bluegrass barn dance:

And here is Wake Me Up… mariachi style

And here is Livin’ on a Prayer as a jazz standard:

For those of you who are TV and videogame junkies, here is the Pokemon theme song:

And here is a Nintendo medley:

Enjoy!

Brookings has another excellent and useful study out this week.  This one examines how much superintendents, on average, contribute to student learning.  The authors, Matt Chingos, Russ Whitehurst, and Katharine Lindqiust, analyze student level data in Florida and North Carolina between 2001 and 2010 to see how much variance in achievement can be explained by changes in superintendents.  The answer is not very much — only .3%.  Other aspects of the school system, including the student, teacher, school, and district matter much more in explaining the variance in student achievement.

The authors are careful to explain that their research does not suggest that there are no dud or superstar superintendents.  It’s just that, on average, superintendents don’t make much of a difference.  They liken this to the effect of money managers who on average add no value, although it is possible that some of them are great and some awful.  Of course, much or all of that difference between great and awful could be random chance.  So when you pick a superintendent (or a money manager) you should rationally expect that they don’t make much of a difference.  It’s a shame that they still cost so much.

This report helps illustrate how Brookings is really the model of what think tanks should be.  It is solid empirical work on a policy relevant question that is written in a way that is accessible to policymakers and other non-experts.  Other think tanks would do well to consider how they could emulate Brookings rather than produce more agenda-driven hatchet  job research.  And more foundations should think about how they could fund this type of quality, policy-relevant work and stop paying for talking points masquerading as research.

Some people may be wondering why a researcher like me, who has always been interested in school choice, would develop an interest in studying how the arts and humanities affect students.  What do art museums or Shakespeare have to do with school choice?

It’s true that I spent much of the last few years working on a large-scale experiment in which we assigned by lottery nearly 11,000 students to tour an art museum or serve in the control group to identify what students learn from field trips to art museums.  And that work has been published in Education Next, the New York Times, Educational Researcher, Sociology of Education, and in another piece currently under review.  It’s also true that I have been conducting experiments on how the performing arts affect students, including this piece on a musical performance, and a forthcoming piece in Education Next about an experiment in which students were assigned by lottery to see Hamlet and A Christmas Carol or to serve in the control to identify what students learn from seeing live theater.

Given how much I have worked on school choice and care about that issue, why have I been spending the bulk of my time over the last few years studying how students are affected by the arts and humanities?  The simple answer is that the arts and humanities are how people try to understand the human condition and to pursue the good life given that condition.  But because our understanding is necessarily limited, we will differ on what constitutes the human condition and living the good life.  This is why we have liberty — to give people the freedom over their conscience, religion, tastes, etc.. so they can pursue the good as they see best.  Freedom over education is just another aspect of liberty that allows people to prepare their children for the pursuit of the good as they think best.

The arts and humanities, however, are suffering because of two views of education that are not only antithetical to the liberal arts but also to educational choice.  The first views education primarily as preparation for one’s future life as a worker.  This utilitarian view of education has little use for the arts and humanities because they simply distract the education system from vocational training.  People whose mantras are “21st Century Skills” and even “College and Career Ready” may not readily admit it, but deep down they are vocational education people.  In their view there is a path to being educated (how else can they report if one is on track or not?) and that path ends in students becoming workers.  This view might tolerate choice, but only choice among providers who are moving students along the same pre-employment path.  You might even call this approach tight-loose.

The other view superficially embraces the arts and humanities but has little appreciation for educational freedom and therefore fails to really understand the arts and humanities.  These are the folks who think they know what should constitute a liberal education and are perfectly happy to impose it on everyone else.  In their view choice is either irrelevant or a hindrance, since people would too often choose to stray from their correct vision of a liberal education.  Let’s just get everyone to teach what I know is best and in the way I think it should be done.  But these liberal arts authoritarians clearly lack an understanding of a central feature of the human condition — that we are all imperfect.  They don’t know the correct content of a liberal education.  And any system they build to impose and maintain the correct vision will inevitably be hijacked for other purposes.  Truth and goodness are best protected by liberty.  If they are not aware of their own corruptability and the corruptability of any system they would build, then they obviously have learned little from art and the humanities.

So, I am interested in the arts and humanities because I am interested in education including some understanding of the human condition.  But I am also interested in choice because that’s how I believe the humanities are most likely to be pursued and effectively promoted.  The real argument for choice is to be found in the arts and the humanities, so that is why I have been devoting the last few years to this line of research.

Marty West and colleagues have an incredibly important study described in Education Next this week.  It’s based on a piece published earlier this year in Psychological Science, a leading psychology journal, but the Ed Next version is probably easier for ed reform folks to access and grasp.

At its heart, the study applies well-established concepts from cognitive psychology to the field of education policy, with potentially unsettling results.  Intelligence, or cognitive ability, can be divided into two types: crystallized knowledge and fluid cognitive skills.  Crystallized knowledge is all of the stuff you know — facts, math formulae, vocabulary, etc… Fluid cognitive skills are the ability to think quickly, keep things in memory, and solve new problems.  The two are closely connected, but there are important distinctions between the two types.

West and his colleagues collected data from more than 1,300 8th graders in Boston, including some of the city’s famously high-performing charter schools to see how these schools affected both types of cognitive ability.  The bottom line is that schools believed to be high-performing are dramatically improving students’ crystallized knowledge, as measured by standardized tests, but have basically no effect on fluid cognitive skills.  That is, Boston’s successful charter schools appear to be able to get students to know more stuff but do not improve their ability to think quickly, keep things in memory, or solve new problems.

Perhaps we should be happy with the test score gains and untroubled by the lack of improvement in fluid cognitive skills.  Chetty et al suggest that test score gains are predictive of later success in life, so who cares about those other skills?  Maybe.  But maybe the students in Chetty experienced improvements in both crystallized knowledge and fluid skills, but he only has measures of the former.  It could still be the case that both types were essential for success.

There are worrisome signs that graduates from schools like KIPP are struggling in college despite impressive test score improvement in K-12.  Perhaps the mis-match between improved crystallized knowledge and stagnant fluid skills cannot produce sustained success.  Perhaps these products of successful ed reform know more of the high school curriculum but are unable to do things, like think quickly and solve new problems, that are important for later life accomplishment.  E.D Hirsch and his followers have been convinced that gains in crystallized knowledge would translate into improved fluid skills, but that appears not to be the case — at least not in these model charter schools in Boston.  

If fluid skills really matter, ed reform is in a serious pickle.  First, we almost exclusively measure crystallized knowledge with our reliance on standardized tests.  If anything, we appear to be increasingly emphasizing (and measuring) crystallized knowledge to the exclusion of fluid skills.  So even when we manage to produce test score gains, we are more likely to neglect fluid skills.  

Second, no one really knows how to improve fluid skills in a school setting.  There are some laboratory experiments that have successfully altered fluid skills, but those effects are often fleeting and have never been replicated in a school environment.  It’s taken us decades to devise some effective strategies for improving test scores.  It may take us decades more to devise strategies for schools to affect fluid skills, even if we start caring about and measuring those outcomes.

Educational success probably requires addressing student needs and abilities on multiple dimensions.  Large, technocratic systems built around standardized test results have a hard time focusing on more than the one dimension of test scores.  The research by West, et al suggests that not all dimensions of academic progress necessarily move in sync.  The unattended dimension of fluid skills may spoil the progress on the attended dimension of crystallized knowledge by undermining later-life success.

(Guest post by James Shuls)

It wasn’t long ago that Jay and Marcus Winters asked the question, “How much are public school teachers paid?” Rather than compare the annual salary of teachers and workers in other professions, Jay and Marcus compared salary based on how many hours and weeks the workers actually put in on the job. Not surprisingly, public school teachers fared well when their relatively short work year was factored into the equation.

Of course, Jay and Marcus’ analysis was roundly criticized. You simply cannot claim that teachers are decently paid. The audacity!

Now it seems, an unlikely ally has taken up the Jay and Marcus mantle on teacher pay – Karen Lewis, the president of the Chicago Teachers Union.

Recently, it was announced that Lewis is considering a run for Mayor of Chicago. As with any political race, this led to a closer examination of Lewis’ finances. The Chicago Sun-Times reports Lewis makes more than $200,000 in combined compensation from the Chicago Teachers Union and the Illinois Federation of Teachers, where she serves as executive vice president. Here’s the good part:

When she first ran for CTU president four years ago, Lewis promised not to make more than the highest-paid teacher.

“How can you criticize [the CPS CEO] for making $230,000 a year during these hard times if you’re making so much more than your members?” she told the Chicago Reader then.

Chicago Public Schools’ payroll records show no teacher makes as much as Lewis’ $136,890 CTU base salary.

In an interview Tuesday, Lewis said she didn’t break her promise not to make more as union president than Chicago’s highest-paid teacher makes, saying her CTU salary is for working the full year, rather than a 39-week school year. (emphasis mine)

What Lewis is saying, is that teachers in Chicago are making the equivalent of $136,890 or more. They just work fewer weeks. Now, where have I heard that before?

It’s almost as if Karen Lewis is saying…

James V. Shuls is an assistant professor of educational leadership and policy studies at the University of Missouri – St. Louis and a Distinguished Fellow of Education Policy at the Show-Me Institute. Follow on Twitter @Shulsie

(Guest Post by Collin Hitt)

KIPP schools demand a certain kind of student – a student who is willing to put in long hours and put up with very strict rules. KIPP has been shown to substantially increase student test scores. But critics argue that the culture at KIPP has major effects on recruitment and retainment. KIPP schools attract better students and are more likely to weed out low performing students, the argument goes. If this is true, KIPP students who persist in school are more likely to have a high-achieving peer group – and the effects of simply being in a peer group are really what explain any positive effects at school. A new study from Mathematica destroys this critique.

At its core, the critique of KIPP is a restatement of larger questions facing the charter school sector. Do charter schools cream the best students from nearby schools? And, compared to surrounding schools, are the lowest performing students at charter schools most likely to leave? Two rigorous studies reviewed here at JPGB answer an unequivocal “no.” But KIPP is a crucial case. The average charter school might take all the students it can find and do anything to keep those students. But surely if anybody engineers the makeup of their student body, it would be a school like KIPP, right?

So, does KIPP cream the best students (or at least better-than-average students) from nearby schools? The following chart shows, clearly not.

Entering KIPP students perform the same or worse than students in surrounding schools. But does KIPP then take exceptional efforts to push low performers back into surrounding schools. Again, clearly not.

Students transfer out of KIPP schools at the same rate as surrounding schools. And the students who transfer perform the same on standardized tests. So the only manner that KIPP may in some way create a measurably different peer group is through the quality of students in later grades who replace the KIPP students who transfer out. In this respect, the students who later transfer into KIPP are higher performers on average than students who transfer into district schools, according to the Mathematica study. But this, of all the ways to create a higher-performing peer group, is the least likely to have any meaningful impact on the performance of students who enter KIPP early on. The high performing peer group wouldn’t even be formed until students’ time at KIPP was almost over.

With their typical class, the Mathematica authors give their critics a charitable hearing, in fact constructing the strongest possible case for the peer-effect hypothesis. So, do peer effects explain KIPP’s impact on test scores? From the Mathematica study itself:

“One way to estimate the possible size of peer effects at KIPP is to combine our findings with other research on how peers’ prior scores affect student achievement. Unfortunately, published estimates of the effect of peer ability on student achievement range widely, from close to zero to nearly half a standard deviation impact for each standard deviation of difference in peer achievement. Even if the largest estimates of peer effects are correct, however, the improvement in peers’ prior test scores would appear to benefit KIPP students’ achievement only by about 0.07 to 0.09 standard deviations after four years at KIPP. KIPP’s cumulative impacts in middle school are three times that size, so even the largest estimates of the size of peer effects suggest that they are unlikely to explain more than one-third of the cumulative KIPP impact.

“Moreover, the best available evidence shows that KIPP produces large impacts on students in their first year at a KIPP school—before late-entering students could possibly have any effect. Consequently, the true peer effect resulting from late entrants is likely to be substantially below the back-of-the-envelope estimate of 0.07 to 0.09 standard deviations.”

The peer-group critique of KIPP essentially says this: anybody could get KIPP’s results if they had KIPP’s students. This simply isn’t true. KIPP is getting better results because of the work being done by teachers and staff. Rather than wonder, if only other schools could have students like KIPP’s, perhaps we should wonder why other schools don’t have adults like KIPP’s. (And, for that matter, why don’t other think tanks have scholars like Mathematica’s?)