Rigorously Studying Cultural Education

December 6, 2012

In my last post I mentioned a large-scale random assignment study of the effects of school tours of an art museum that I am conducting with my colleagues, Brian Kisida and and Dan Bowen.  Some people have asked for more information about that project.  So, here is a brief summary of what we are doing in that study as well as some related projects examining cultural education.

The random assignment study of field trips was made possible by the fact that the Crystal Bridges Museum of American Art opened in Northwest Arkansas, an area that had never before had a major art museum.  Because there was intense interest from schools in the area in having school tours there were many more applicants for field trips than the museum could accommodate right away.  We worked with the museum to randomly assign tours to applicants.

Specifically, the museum received more than 300 applications for tours during the first semester.  We organized those applicants into matched pairs, which were often adjacent grades in the same school or the same grade in different schools with similar demographic characteristics.  We then randomly assigned one school in each matched pair to be the treatment group and one to be the control group.  We randomly ordered the matched pairs and the museum scheduled the first 55 treatment groups for school tours last spring.  The 55 matched control groups were guaranteed a tour during the next semester for participating in the study.

We then administered surveys to the randomly assigned treatment and control group students and teachers a few weeks after the treatment group visited the museum.  Those surveys were designed to measure five types of outcomes: 1) whether the school tour helped create cultural consumers (students who want to return to museums and engage in other cultural activities), 2) whether the school tour helped create cultural producers (students who want to make art), 3) whether the school tour increased student knowledge about art and history, 4) whether the school tour improved student critical thinking about works of art, and 5) whether the school tour altered student values, like empathy and tolerance.

We have already collected results from almost 6,000 K-12 students and teachers from 80 different schools during last spring’s research.  This fall we are adding another 4,000 students and teachers to the study from another 60 or so schools.  When it is all done and analyzed it will probably be the biggest, most comprehensive, and highly rigorous examination of the effects of school tours of an art museum.

As part of the study we are also asking students in grades 3-12 to write short essays in response to paintings that they have probably never seen before to assess how they critically analyze a new work of art after they’ve had a school tour of an art museum.  Last semester we coded almost 4,000 essays in response to Bo Bartlett’s painting, The Box, which was pictured in my previous post.  This semester we wanted to try something a little more abstract, so we we will be coding another 2,500 or so essays in response to Marsden Hartley’s painting, Eight Bells Folly, which is pictured above.  Dan Bowen has taken the lead in the coding and analysis of these essays and will soon be on the job market in case anyone is looking for a great and innovative researcher to hire.

There are obvious limitations to our study.  We can only measure short term effects since the control group receives the treatment the following semester.  And we can only measure a limited set of outcomes from an art experience.  But we will know a whole lot more and with higher confidence than we do now.

We are also conducting two studies with the Walton Arts Center, which is a performing arts theater in Fayetteville, Arkansas.  In one study we are are working with our colleague in the music department, Lisa Margulis, to learn about the effects of information in program notes on students’ experiences during school field trips to see performances.  We are randomly assigning students to receive program notes with information about the show they are seeing or “placebo” program notes  that do not tell them about the show they are seeing.  The question is whether information alters the experience.

And in the other study with the Walton Arts Center we are surveying more than 2,000 7th grade students in area schools to link the past performances they have seen on school field trips to their current behaviors as cultural consumers and producers as well as some empathy and tolerance outcomes.  We are also going to use attendance zone boundaries as an exogenous source of variation to make stronger causal claims about how past school field trips may have contributed to current behaviors and attitudes.

We are also in talks with various folks about additional studies, all of which will use random assignment or similarly rigorous methods.  This line of work is particularly exciting because there is a limited amount of rigorous research out there on how school cultural activities affect students.

(link edited)


Education Isn’t Entirely About Economic Utility

December 4, 2012
 2002-the-box-sm.jpg

As some of you may know, I’ve been working on a large-scale random assignment experiment of the effects of school tours of an art museum on students and their learning.  We spend hundreds of millions of dollars each year on school field trips and billions more on art museums, but we have relatively little rigorous evidence on how field trips and art museums affect students.  Soon we are going to have a lot more information.

Since the world of art and museum education is new to me, I’ve been trying to learn about how people in that field think about what they are trying to accomplish and what kind of evidence they present to justify the resources required.  Some  people try to justify the place of art in education by claiming that art positively affects achievement in math and reading — subjects whose importance is a matter of broad consensus.  Unfortunately, the evidence linking art education to improved math and reading achievement is generally weak and unpersuasive.

Why do people bother trying to justify art in terms of math and reading achievement?  Math educators don’t try to frame their accomplishments in terms of reading or vice versa.  Why do people in art try to frame the benefits of their field in terms of other subjects?

The problem is that a good number of  policymakers, pundits, and others who control the education system seem to think that the almost-exclusive purpose of education is to impart economically useful skills.  Math and reading seem to these folks to be directly connected to economic utility, while art seems at best a frill.  If resources are tight or students are struggling, they are inclined to cut the arts and focus more on math and reading because those subjects are really useful while art is not.

This economic utility view of education is mistaken in almost every way.  Most of what students learn in math and reading also has no economic utility.  Relatively few students will ever use algebra, let alone calculus, in their jobs.  Even fewer students will use literature or poetry in the workplace.  When will students “use” history?  We don’t teach those subjects because they provide work-related skills.  We teach algebra, calculus, literature, poetry, and history for the same reasons we should be teaching art — they help us understand ourselves, our cultural heritage, and the world we live in.  We teach them because they are beautiful and important in and of themselves.  We teach them because civilized people should know them.

Most parents understand that education is not entirely about imparting economically useful skills.  Yes, they want their children to get good jobs but they also want to have their children develop good characters, appreciate the good life, and generally be civilized human beings.  Of course, different parents may want a different mix of economic and cultural education for their children and school choice would allow them to find the schools that offered the mix that suited their needs and tastes.

But policymakers, pundits, and others suffering from PLDD who control our increasingly centralized education system focus almost exclusively on economic utility as the criteria for making education policy decisions.  Math and reading test scores are the only clubs they have to beat their opponents in establishing their preferred policies.  And economic payouts are the only objective measures they can use to justify expenditures.  Parents don’t think about education this way, but they have less and less say over what happens in the rearing of their children to become what they hope will be civilized human beings.

Some policymakers, pundits, and other PLDD sufferers have noticed that not everything taught in math and reading is economically useful and want to fix that.  You have folks like Tony Wagner and the 21st Century Skills movement suggesting that we cut algebra because students won’t “need” it.  Instead, students would be better off learning communication skills, like how to prepare an awesome Power Point (TM).  And you have Common Core cutting literature in English in favor of “informational texts.”

Of course, the logical culmination of the idea of school as job-skills provider is that we would do away with school altogether and just have apprenticeships.  I see nothing wrong with apprenticeship but it is not what I or most parents view as an “education.”

People in the art world can justify what they do by arguing for art in its own right.  They can rigorously measure art outcomes, as we are in our random-assignment field trip study.  In fact, as part of our study we had 4,000 students write short essays in response to Bo Bartlett’s painting, The Box (pictured above).  It may be harder to code and analyze essays about paintings than to run another value-added regression on the math and reading scores that the centralized authorities have collected for us, but that doesn’t mean it’s any less important.  The purpose of education isn’t only what the centralized authorities decide it is and bother to measure.


Hoosier Fiasco a Wakeup Call to Both Sides of Common Core Debate

November 9, 2012

(Guest Post by Matthew Ladner)

Let me start by noting that what I write here, as always, is my own personal view. It does not reflect the views of my employer or any other group with whom I collaborate. It is my hope, for reasons I will explain below, to serve as an equal opportunity offender. Three days later I can speak only daggers to both sides of our currently idiotic Common Core debate.

A few days before the election some polling data was released from Indiana showing that Superintendent Tony Bennett had a problem with-of all people-conservative Republicans. It has quickly passed into the Conventional Wisdom that Tony’s support for Common Core cost him re-election. This result is an insult to a dumpster fire for both sides of the Common Core debate.

Let’s get two things clear from the outset: no one has yet to convince me that Common Core is a good idea and Common Core opponents have revealed themselves to be unsophisticated ya-hoos as easily led by weak arguments as any Ravitch-zombie. Whether Indiana adopts or chooses not to adopt Common Core is ultimately of trivial to modest importance in driving academic outcomes in Indiana. Neither side of the argument in Indiana seemed to appreciate this stunningly obvious fact.

Supporters claim that CC is a little better than Indiana’s existing standards, opponents a little worse. This is all subjective and thus there is no truth to discern here.  Should Mississippi adopt Common Core-yes states where the stock picking chicken can pass the test have nothing to lose. Should Massachusetts? Certainly not-a state with the highest NAEP scores on all four main tests has much to lose. The correct response to “should Indiana adopt Common Core?” is “why should I care?”

Common Core in Indiana thus was not a hill worth dying on to defend, nor anything worth putting a teacher union puppet in charge of your Department of Education to prevent. If you think otherwise you have earned a spot carved in stone on my “Drooling Idiots” tablet that I keep out in the rock garden.

My ESP detects objections from Common Core opponents reading this now. What about the Obama administration interfering in state/local control of schools? States adopted CC voluntarily and can leave voluntarily. Yes Duncan put points into Race to the Top for CC adoption but note that participation in that was purely optional and RTTT it is now long gone. Virginia also got a waiver from NCLB despite not adopting CC, busting another cherished myth. There was some chatter about conditioning Title I on CC adoption, but that was all it was thus far-chatter.  Everyone should be on guard against this, but let us be rid of all illusions in noting that the reality of the situation “federal takeover” remains such an exaggeration that it constitutes a tin-foil hat argument.

Think the federal government violated a law from the 1970s to bankroll Common Core? Maybe they did-how would I know? Either put up by going to court to prove it or shut up because you don’t really believe it.

Mark however that the fact that Indiana’s adoption of Common Core is relatively unimportant cuts both ways. An Indiana school board official said something to the effect that rather than picking his battles, Tony never saw a mosh-pit that didn’t make him want to jump in and start breaking noses of punks who deserve it. True enough- one of the many qualities that I love about Tony. Tony believed in Common Core and he fought for Common Core. Tony however gave a great deal more to the Common Core effort than it gave back.

The pitiful weakness of the Common Core nexus in making a coherent and visible case for Common Core against unsophisticated attacks like “federal takeover” and “Obamacore” means that Common Core does not deserve champions like Tony Bennett. This effort needs to be more convincing that “ummmm……….high standards are good or something” and needs to move beyond the Beltway blogo-echo chamber into the public quickly. If Common Core supporters have a persuasive case to make, now would be a great time to start making it.

The reason is simple: the reactionaries now have a play book to peel off uninformed conservative voters and add them to their coalition. This lesson seems unlikely to be lost on teacher unions or upon either political party in states with elected Superintendents.  It remains to be seen whether some enterprising group of reactionaries will successfully scale this model up to a Governor’s race, but I can’t see any reason for them not to give it a try.

In short, the combined ineptitude of the Common Core effort and the mouth-breathing stupidity of Common Core opponents stands as a risk to the broader education reform agenda. Love Common Core or hate it, let’s be perfectly clear that Tony Bennett was up to far more things, and far more important things in order to equip Indiana children with the academic skills they need. This farce has ended in tragedy with an entirely avoidable setback.

A plague on both houses! I hope both sides will accept my invitation to pull their heads out of their asses. This is very serious business we are engaging in here and we do not have the luxury of this kind of pointless stupidity.

P.S. Just in case no one else was going to say in public what many are saying in private, I hope that Governor Daniels enjoys those faculty teas discussing the finer points of Mechanical Engineering  because his decision to opt out of races is looking terribly misguided right about now. Tony deserved much better from all of us, but I am trying to imagine a better person than a popular and successful conservative Indiana Governor to talk sense to right-wing Hoosier yayhoos.

If any of you take offense at any of this, regardless of the tribe you hail from, feel free meet me by the bike racks in the comments section.  I will be happy to make further efforts to beat sense into you.


Why E.D. Hirsch Should re-examine his position on parental choice

September 26, 2012

(Guest Post by Matthew Ladner)

So a few years ago when Sol Stern decided to attack parental choice for reasons that are still largely only known to him, City Journal posted an online debate concerning Sol’s article, which included a full-throated endorsement of Sol’s position by E.D. Hirsch.

I had a hard time making much sense of the Hirsch critique. It seemed to read much more as an indictment of bad state standards than of the parental choice movement.  The parental choice movement’s original sin seemed to be in being a “structural reform” that ignored the vital importance of imposing Core Knowlege on everyone.

Or something to that effect, near as I could tell. I was and still am confused with exactly how this is supposed to happen, but I’m sure someone has a fail-safe plan this time.

My own contribution to the debate attempted to make the point that of course the political constraints facing parental choice programs keep them from being some sort of miracle-drug cure-all, but that was hardly a reason to oppose it. I haven’t seen any other miracle cures either. Moreover, there is no reason to imagine that the parental choice movement and the standards movement need to necessarily be at odds.

In any case, above is a picture of the district middle school in my neighborhood-Shea Middle School in the Paradise Valley School District. Shea is proudly announcing that Hirsch’s Core Knowledge Curriculum will begin in August 2013 in a 9000 point font banner you see above. At least one of the elementary schools that feed in to Shea Middle School has also  adopted Core Knowledge.

Shea’s adoption of Core Knowledge might have something to do with the fact that two of the highest performing charter schools in country opened campuses in the area this fall. Arizona homegrown outfits BASIS and Great Hearts both opened new schools within a few miles of Shea Middle School in the Fall of 2012.  Both BASIS and Great Hearts have an impressive record of academic achievement. Some of the Great Hearts schools have generated 1,000 student waiting lists, and both operators have attracted the interest of out-of-state philanthropists.*

Of course it could be the case that these new schools opening in the neighborhood had nothing to do with the decision to adopt Core Knowledge, or to hang a giant banner advertising the adoption for that matter. Other Paradise Valley schools have used the Core Knowledge curriculum for years. It is within the realm of the plausible that Shea Middle School would have been adopting Core Knowledge in 2013 whether facing competition from BASIS and Great Hearts or not. If I were to have the opportunity to ask PV officials about this, they might very well make such a claim with conviction.

And if I hadn’t seen an email from a Parent-Teacher group from one of the feeder elementary schools full of steely determination not to lose students to the new charter schools, I might have even believed them. The email expressed (rational) concern about losing students and listed a number of possible strategies including the adoption of IB, foreign language immersion and (yes) Core Knowledge as reform strategies….and now the banner.

Smoking gun? No. Enough to convince a reasonable person? Certainly.

Parental choice mechanisms have done a great deal to satisfy parental demand for Core Knowledge and CK type schools. If we had more of it, we would also have a higher use of CK and similar curriculum both in district and non-district schools. Hopefully it will prove useful for Shea Middle School. Alternatively, we could dream of a master plan that transforms millions of public school teachers into Allan Bloom in one great non-incremental stroke, but I think we all know how that story ends.

Oh well, back to the old super-genius drawing board…

Personally I am a fan of traditional curriculum and want it to be available to those who desire it. I’m also leery of imposing it on those who don’t. I view American schools as having serious curriculum problems, but plenty of other problems as well. Dirigisme got us into this mess, and some of us are naturally skeptical that a new and improved version is going to get us out of it all by itself.

* Disclosure: I serve on the board of a BASIS school (not the one discussed here) and two of my children very happily attend a Great Hearts Academy (but not the school alluded to here).

Edited for Typos


Rob Pondiscio on Writing

September 25, 2012

Rob Pondiscio has a great piece about how we teach writing in The Atlantic.  It’s worth reading the whole thing, but here are some tasty bites:

Every decent human impulse we have as teachers shouts in favor of not imposing rules and discipline on students, but liberating them to discover the power of their voice by sharing their stories. Of course children will be become better writers if they write personal narratives instead of book reports. Obviously children will be more engaged and motivated if they can write from the heart about what they know best, rather that trudge through turgid English essays and research papers.

Grammar? Mechanics? Correcting errors? Please. Great writing is discovery. It is the intoxicating power of words and our own stories, writing for an audience and making things happen in the world. We know this works. We all saw the movie Freedom Writers, didn’t we?

Like so many of our earnest and most deeply humane ideas about educating children in general, and poor, urban children in particular, this impulse toward authenticity is profoundly idealistic, seductive, and wrong. I should know. I used to damage children for a living with that idealism….

Every day, for two hours a day, I led my young students through Reader’s and Writer’s Workshop. I was trained not to address my kids as “students” or “class” but as “authors” and “readers.” We gathered “seed ideas” in our Writer’s Notebooks. We crafted “small moment” stories, personal narratives, and memoirs. We peer edited. We “shared out.” Gathered with them on the rug, I explained to my 10-year-olds that “good writers find ideas from things that happened in their lives.” That stories have “big ideas.” That good writers “add detail,” “stretch their words,” and “spell the best they can.”

Teach grammar, sentence structure, and mechanics? I barely even taught. I “modeled” the habits of good readers and “coached” my students. What I called “teaching,” my staff developer from Teacher’s College dismissed as merely “giving directions.” My job was to demonstrate what good readers and writers do and encourage my students to imitate and adopt those behaviors….

“When our students resist writing, it is usually because writing has been treated as little more than a place to expose all they do not know about spelling, penmanship and grammar,” observes Lucy Calkins, probably the workshop model’s premier guru. She is almost certainly correct.

This leaves exactly two options: The first is to de-emphasize spelling and grammar. The other is to teach spelling and grammar. But at too many schools, it’s more important for a child to unburden her 10-year-old soul writing personal essays about the day she went to the hospital, dropped an ice cream cone on a sidewalk, or shopped for new sneakers. It’s more important to write a “personal response” to literature than engage with the content. This is supposed to be “authentic” writing. There is nothing inherently inauthentic about research papers and English essays.

Earlier this year, David Coleman, the principal architect of the widely adopted Common Core Standards, infamously told a group of educators, “As you grow up in this world, you realize people really don’t give a shit about what you feel or what you think.” His bluntness made me wince, but his impulse is correct. We have overvalued personal expression. The unlived life is not worth examining. The pendulum has swung too far.

 


Odds and Ends

March 12, 2012

In case you missed the 60 Minutes segment on the Khan Academy.  You can watch the video and read the transcript here.

And the New York Times reports on a study conducted in New York City comparing student achievement at 10 schools using a Core Knowledge approach against the achievement at 10 schools using existing (mostly Balanced Literacy).  It find greater gains in the Core Knowledge schools in reading comprehension as well as content knowledge in social studies and science.


Gates, the Bizarro Foundation

January 31, 2012

Comic book geeks are familiar with Bizarro World, a place where everything is the opposite of what it is in the normal world.  In Bizarro World, people would abandon a policy strongly supported by rigorous evidence while embracing an alternative policy for which the evidence showed little promise.

I was thinking about Bizarro World and then it struck me — Perhaps the Gates Foundation has somehow fallen into the Bizarro World.  It’s just about the only thing that makes sense of their Bizarro choices with respect to education reform strategies.

The dominant education reform strategy of the Gates Foundation before 2006 was to break large high schools into smaller ones, often using school choice and charter schools.  As a Business Week profile put it:

The foundation embraced what many social scientists had concluded was the prime solution: Instead of losing kids in large schools like Manual, the new thinking was to divide them into smaller programs with 200 to 600 students each. Doing so, numerous studies showed, would help prevent even hard-to-reach students from falling through the cracks. The foundation didn’t set out to design schools or run them. Its goal was to back some creative experiments and replicate them nationally.

But the Gates Foundation wasn’t patient enough to let the experiments produce results.  Instead, they hired SRI and AIR to do a very weakly-designed non-experimental evaluation that produced disappointing results.  Gates had also commissioned a rigorous random-assignment evaluation by MDRC, but it would take a few more years to see if students graduated and went on to college at higher rates if they were assigned by lottery to a smaller school.

Gates couldn’t wait.  They were convinced that small schools were a flop, so they began to ditch the small school strategy and look for a new Big Idea.  Tom Vander Ark, the education chief who had championed small schools, was out the door and replaced with Vicki Phillips, a superintendent whose claim to fame, such as it was, came from serving as Portland’s superintendent where she consolidated schools (not breaking them into smaller ones) and centralized control over curriculum and instruction.  As one local observer put it:

In her time in the famously progressive, consensus-driven city, she closed six schools, merged nearly two dozen others through K-8 conversions, pushed to standardize the district’s curriculum, and championed new and controversial measures for testing the district’s 46,000 children-all mostly without stopping for long enough to adequately address the concerns her changes generated in the neighborhoods and schools where they played out.  During her three years in Portland, Phillips’ name became synonymous with top-down management, corporate-style reforms, and a my-way-or-the-highway attitude.

Under Phillips and deputy education director, Harvard professor Tom Kane, the Gates Foundation has pursued a very different strategy: attempting to identify the best standards, curriculum, and pedagogy and then imposing those best practices through a national system of standards and testing.

And here is where we see that Gates must be the Bizarro Foundation.  The previous strategy of backing small schools has now been vindicated by the rigorous random-assignment study Gates couldn’t wait for.  According to the New York Times:

The latest findings show that 67.9 percent of the students who entered small high schools in 2005 and 2006 graduated four years later, compared with 59.3 percent of the students who were not admitted and instead went to larger schools. The higher graduation rate at small schools held across the board for all students, regardless of race, family income or scores on the state’s eighth-grade math and reading tests, according to the data.

This increase was almost entirely accounted for by a rise in Regents diplomas, which are considered more rigorous than a local diploma; 41.5 percent of the students at small schools received one, compared with 34.9 percent of students at other schools. There was little difference between the two groups in the percentage of students who earned a local diploma or the still more rigorous Advanced Regents diploma.

Small-school students also showed more evidence of college readiness, with 37.3 percent of the students earning a score of 75 or higher on the English Regents, compared with 29.7 percent of students at other schools. There was no significant difference, however, in scores on the math Regents.

Meanwhile, as part of their newly embraced top-down strategy, the Gates effort to identify the secret formula for effective teaching has failed to bear fruit.  The Gates -operated Measuring Effective Teachers Project failed to identify any rubric of observing teachers or any components of those rubrics that were strongly predictive of gains in student learning.  And the Gates-backed “research” supporting the federally-orchestrated Common Core push for national standards and testing has been strikingly lacking in scientific rigor and candor.

In short, the Gates Foundation has ditched what rigorous evidence shows worked and is pushing a new strategy completely unsupported by rigorous evidence.  They must be in Bizarro World.  Somebody please get me some blue kryptonite.