Jay P. Greene's Blog

(Guest post by Patrick J. Wolf)

New York City public charters have been much in the news of late (see here & here) for hitting it out of Yankee Stadium on student achievement.  When Judge-ing (sorry, couldn’t resist) The Big Apple’s charter school sector, add this little fact to the case:  charters out-slug their peers at a lower cost.

That is the conclusion of my latest study of charter school funding inequity, co-authored with Larry D. Maloney.  It is fun to study New York City, in part because of great potential for wordplay but also because the place is so darn big that you can disaggregate results by borough and still have district v. charter comparisons informed by large samples.  So, “start spreading the news…”

There are over 1,000,000 public school children in The Big Apple.  Seven percent of them attended charter schools during Fiscal Year 2014, the focus of our study.  Cash revenue to charter schools averaged $15,983 per-pupil while payments to district-run schools averaged a much more generous $26,560 per-pupil.  “You just wait a New York Minute Mr. Henley,” you might caution, “The New York City Department of Education actually provides in-kind services to students in charter schools that represent a funding resource not accounted for in your cash calculations.”  You would be right.  After factoring in the cash value of such in-kind services, charter schools receive a mere $4,888 less in per-pupil funding than district schools (Figure 3).  New York City charters schools are outperforming the City’s district schools at about 81 cents on the dollar.

I’ll admit that my figure isn’t nearly as MoMA-worthy as Matt’s post-modernist depiction of the Arizona school districts that refuse to accept students through inter-district choice, but it makes a crucial point.  Even accounting for the value of everything contributed in support of charter schools in New York City, district schools still get more money per student.

Critics of our prior charter school funding studies (available here and here) have claimed that we are making Big-Apple-to-Big-Orange comparisons, since district schools provide more extensive educational services to students than charters.  Our accounting for in-kind district services to charters fully addresses that argument.  After factoring in the value of co-located facilities, transportation, meals, special education services, health services, textbooks, software, etc., all of which are provided to charters in New York City so that the scope of their services is equal to that of district schools, the charters still receive less funding.  We even examined school spending patterns, in addition to funding patterns, and the story is the same.

Surely the student populations in district schools are needier than those in charter schools, thereby justifying the funding gap, right?  Actually no.  The population of charter school students in New York City contains a higher percentage of free-and-reduced price lunch kids than the population of district school students (Figure 4).

The percentage of students with disabilities is only slightly higher in district schools versus charter schools, 18.2% compared to 15.9%.  That means that districts enroll 21,342 “extra” students with disabilities compared to charters.  For the special education enrollment gap favoring districts to explain the entire funding gap favoring districts, each “extra” student with a disability in the district sector would have to cost an additional $214,376 above the cost of educating a student in general education.  It is simply implausible that the slight gap in special education enrollments explains the substantial gap in funding between district and charter schools in New York City.

Like rookie sensation Aaron Judge, this report has lots of hits besides just the homeruns described above, so check it out.  In sum, New York City has made a major commitment to provide material support to students in its public charter schools.  Still, inexplicable funding inequities persist depending simply on whether a child is in a charter or a district school.  Larry and I think this case study provides yet another Reason to support weighted student funding with full portability (see what I did there?).  Switching to such a simple and equitable method for funding all public school students definitely would put us in a “New York State of Mind.”

CREDO has produced a slew of studies comparing test score outcomes for students in charter and traditional public schools.  Those studies have come to dominate public policy and foundation discussions about charter schools and are sometimes thought to be the highest quality studies on charter effects.  They are not.

We actually have more than a dozen random-assignment studies of charter school achievement effects.  For a summary of what those gold-standard studies find, see this systematic review by Cheng, Hitt, Kisida, and Mills (or if you have difficulty with the pay-wall you can find an earlier working paper here).

CREDO’s research design is not gold standard.  It’s not even silver.  Maybe it’s formica.  It would be understandable for you to be confused and think CREDO was gold standard given how much people in policy circles talk about that research as opposed to the set of gold-standard random-assignment experiments.  And you might be further confused by the language CREDO uses when they describe their research design as comparing “virtual twins.”

CREDO’s methodology does not compare twins, virtual or otherwise.  All they are doing is comparing students who are similar on a limited set of observable characteristics — race, age, gender, and prior achievement scores.  “Matching” students on those observable characteristics is just as prone to selection bias as any other observational study that controls statistically for a handful of observed characteristics when comparing students who choose to be in different school sectors.  That is, students who choose to attend charter schools are very likely to be different from those who choose to remain in traditional public schools in ways that are not captured by their race, age, gender, and prior test score.  In particular, their desire to switch to a different kind of school may well be associated with developments in their life that might affect the future trajectory of their test scores.  In short, school choice is prone to bias from selection in observational studies like CREDO.

CREDO overstates the strength of their methodology by referring to their approach as one that compares “virtual twins.”  They say: “a ‘virtual twin’ was constructed for each closure student by drawing on the available records of students with identical traits and identical or very similar baseline test scores.” ( p. 3)  It is probably unintentional, but this description gives the false impression that they are comparing “identical” students in different sectors.  In reality they are only comparing students who are similar on a handful of observed characteristics.  Ladner and I may both have beards, enjoy a malt beverage, and are interested in school choice but that does not make us “twins” nor would it be reasonable to describe us as having “identical traits.”

Unlike CREDO, gold-standard random assignment studies are not subject to selection bias because only chance distinguishes between whether students are in charter or traditional public schools.  On average, the students being compared in randomized control trials (RCTs) are truly identical on all observed and unobserved characteristics.  They really are virtual twins.

Backers of CREDO can point to the fact that the CREDO methodology has produced results that are similar to experimental studies in a few locations and claim that selection bias must therefore not be an important problem.  This is a faulty conclusion.  Finding that CREDO’s observational method and randomized control trials sometimes produce similar results only proves that selection did not bias the results in those cases.  In other cases charters may attract students who are very different in their future achievement trajectory and RCTs would produce results that are very different from an observational study.  Online charters are likely a clear example of where this selection bias would be severe.

The problem is compounded by the fact that policymakers and foundation officials are too eager to use CREDO results for a number of reasons that have nothing to do with the quality of the methodology.  Sometimes they want to use CREDO because it supports their preferred policy conclusions.  They also have a strong preference for studies that name the city or state they are considering.

It’s as if Jonas Salk proved that the polio vaccine works in an RCT, but policymakers and foundation officials want to know if it prevents polio in New Orleans or Detroit.  Rather than rely on a lower quality research design that mentions their town, policymakers and foundation officials should focus on the highest quality charter randomized control trials, of which we have more than a dozen.  If that evidence shows the polio vaccine to work, then they should assume it also works in their town.

In my last post I described how Atila Abdulkadiroglu, Parag Pathak, and Christopher Walters (APW) released an evaluation of the 1st year results from Louisiana’s voucher program through NBER dated December 2015 (although actually released in January 2016) that failed to cite or provide appropriate credit to earlier conference presentations by Jonathan Mills and Patrick Wolf and a dissertation by Mills.  In response APW issued a statement that raises many issues, but fails to address the heart of the matter. Before replying to some of those extraneous issues, let’s focus on the key questions:

1. Did Jon and Pat conduct analyses, write papers, and present findings to the academic community of the same program using the same basic methodology and data as APW prior to their December 2015 NBER paper?
2. Were APW aware of this prior work?
3. Did APW fail to cite and give appropriate credit to that prior work in their December 2015 NBER paper?

If you read their statement closely you will see that APW do not deny the existence of prior work, do not deny being aware of that work, and do not deny failing to cite it. Let’s take a look at what they did say.

The No Prior Working Papers Claim

APW do not deny that Jon and Pat had multiple conference papers, starting with one to the Association for Public Policy Analysis and Management on November 8, 2013 followed by 8 more during 2014 and 2015.  Instead they focus on whether there were prior working papers: “Mills and Wolf released a working paper in February 2016. This is the first working paper by the Arkansas team that we are aware of.” Christopher Walters reiterates this point in a comment on the blog: “We would have cited a public working paper had we known of one in December 2015.”

There are two problems with this no prior working paper argument.  First, a working paper is not the only format of prior academic work that requires citation.  There is no exemption to the scholarly obligation to credit prior work if that work is in the form of conference papers and presentations.  In fact, the American Economics Association helpfully provides guidance on the appropriate way to cite “Lectures and Papers Presented at Meetings.”

Second, there is a very good reason why Jon and Pat did not have a working paper for APW to cite and APW are almost certainly aware of that reason: The Louisiana Department of Education (LDE) asked Jon and Pat not to widely disseminate their findings until after the second year of results were complete. LDE had made the same request of APW, who similarly complied with that request before deciding that it would take them too long to finish the 2nd year analyses.  As APW describe it in their statement: “We postponed the release of our paper because the LDE promised us additional data in exchange for delaying public disclosure of our results. We released the paper when we judged that this data was not forthcoming.”

While Jon and Pat did not widely publicize the results of their 1st year evaluation by issuing a working paper online, they did not keep their findings a secret.  They presented their results at numerous conferences and in a dissertation, and they solicited feedback and suggestions from colleagues (including from Abdulkadiroglu).  And they took that time to resolve missing data issues, complete the 2nd year analyses, and release both 1st and 2nd year results in a working paper only a month after APW released their 1st year results.

By contrast, APW operated in a different manner.  As far as I can tell they did not present their work at any meeting of a professional association prior to December 2015.  They did not disclose to Pat and Jon that they were planning to conduct or were already conducting their own study, even when Pat and Jon discussed the research with Abdulkadiroglu.  They also did not take the time to work out data issues with LDE and wait for the 2nd year of results before releasing their NBER report. And the failure of that report to cite and credit the work produced by Jon and Pat cannot be excused because Jon and Pat didn’t put a working paper online.

The Data Sharing Agreement Claim

APW’s statement does not deny being aware of prior work by Jon and Pat.  In Walter’s comment on the blog, however, he does say “We were not aware of the Mills dissertation chapter …”  But Abdulkadiroglu, not Walters, was at the lunch with Jon and Pat in June 2015.  The statement they issued together does not deny that Abdulkadiroglu was told about Jon’s dissertation work at that lunch.  Instead it denies that Jon’s dissertation was the origin of APW’s work: “This conversation centered on the use of school assignment mechanisms for program evaluation. The conversation occurred more than two years after we signed our data agreement to evaluate the LSP using lotteries, so it was clearly not the inspiration for our work.”

Whatever inspired their work does not obviate their scholarly obligation to cite and give appropriate credit to academic work on the same program that had been produced prior to their December 2015 NBER report. The existence and date of APW’s data sharing agreement with LDE is irrelevant to whether they failed to cite and give appropriate credit to previously produced work.

But in a comment on the blog, Walters offers a novel interpretation of what “prior work” means: “Your team’s research is not ‘prior work.’ As shown by the date on the data agreement provided in our response, the two projects were in progress simultaneously.” By prior work I mean research findings that had been produced and shared with the academic community before December 2015.  The fact that both research teams had data sharing agreements does not erase the fact that Jon and Pat had produced, presented, and published (in a dissertation) results prior to the December 2015 NBER report.

In addition, it should be noted that Walters’ description that “the two projects were in progress simultaneously” is very different from APW’s characterization of Jon and Pat’s work as a “followup analysis” in the footnote they added to their NBER report after Jon and Pat released their February 2016 working paper with the 1st and 2nd year results. It should be further noted that even that amended paper does not provide a proper citation because Jon and Pat’s work is missing from the reference section.

Even though the existence and date of the data sharing agreements is irrelevant to the scholarly obligation to cite prior work, the suggestion that APW had an agreement that pre-dated Jon and Pat’s by two years is incorrect.  Jon and Pat started negotiating a data sharing agreement with LDE during the fall of 2012, around the same time as APW, and concluded that agreement on January 8, 2013, two months after APW.  Jon and Pat were completely unaware of the existence of APW’s data sharing agreement or that APW were pursuing a similar line of research with the same data.

The data sharing agreements are further irrelevant because they do not establish when the research teams actually started the research in earnest.  We know that Jon and Pat had started their research during 2013 because they presented preliminary findings at the APPAM conference in November 2013. The earliest we know APW had written-up results, according to reporting by The 74 Million, was in October 2015, when they presented them to LDE.  And when Jon and Pat discussed their research with Abdulkadiroglu in June 2015, it is unclear whether he failed to disclose that they were working on a similar study because APW had not actually started that work yet.

The You Didn’t Cite the Dissertation Either Claim

Finally, the APW statement seems to suggest that they are somehow absolved of the responsibility of citing Jonathan Mills’ dissertation because Jon and Pat also failed to cite that dissertation in some of their subsequent papers, including the February 2016 working paper with 1st and 2nd year results. First, it is an entirely different thing to deny oneself credit than to deny someone else credit.  Whether Jon and Pat made an error in failing to credit all of their own prior work does not excuse APW in making that error about someone else’s work.  This is especially the case because Jon and Pat’s failure to cite Jon’s dissertation does not mislead the reader about who had been the first to produce these results.  Second, Jon and Pat’s February 2016 working paper did cite a series of their own prior conference papers from 2014 and 2015, so they did clearly establish that they had been presenting results on the Louisiana program well before December 2015. APW might have noticed those references and acknowledged that record of prior work when they added the footnote in an update to their December 2015 NBER paper mentioning Jon and Pat’s February 2016 working paper.

Raising this issue is certainly not pleasant.  In fact, it’s down-right nerve-wracking and I can completely understand why Jon was reluctant to press this matter earlier.  But I think credit is being given to other researchers for being the first to produce an evaluation of achievement effects from the Louisiana voucher program that properly belongs to Jon and Pat.  I actually tried to resolve this amicably with Parag Pathak when he came to give a lecture at the University of Arkansas in 2016.  Parag had been invited prior to the December 2015 NBER report and his visit was uncomfortable.  It would have been more uncomfortable if either Jon or Pat were there, but Jon had already left for a post-doc at Tulane and Pat had another obligation.  Contrary to the APW statement, I did raise the bruised feelings over credit and data access with Parag and suggested that he might smooth things over with Jon and Pat and perhaps regain access to LDE data if he were to suggest collaboration on some future project with them.  I suggested at the very least he should call Pat and talk to him.  He never did.

Chutzpah is jokingly defined as murdering one’s parents and then complaining about being an orphan. Atila Abdulkadiroglu, Parag Pathak, and Christopher Walters ( hereafter APW or the MIT team) sure show some chutzpah when complaining about not having continued access to data regarding the Louisiana Scholarshp Program (LSP) in a recent article. While I don’t know for sure why they were denied continued access to data, I believe that it is related to their rush to release 1st year results from their evaluation. Why they were rushing is an incredibly depressing story about how status and power in our field contributes to academic abuse and dishonesty– a story the reporter who wrote the article entirely missed.

It is not widely known or acknowledged, but the original analysis of 1st year result from LSP was conducted by Jonathan Mills when he was a doctoral student along with his advisor, Patrick Wolf, at the University of Arkansas.  They presented those findings at academic conferences 8 times during 2014 and 2015 and they were contained in Jon’s dissertation published in July 2015. APW were at some of those conferences.  Atila actually had lunch at one conference with Jon and Pat during which they discussed that study in June of 2015.  Atila never indicated that he was conducting or planning to conduct a similar study.  He offered to help and they sent him some materials.  He never responded with help but he did move forward with his own study with the MIT team without informing Pat or Jon that they were doing so.

APW released their own study as an NBER report in December 2015.  Nowhere in that report did they acknowledge or cite Jon and Pat’s earlier work of which they were almost certainly aware, having discussed it with them. Nor did APW acknowledge that their study was essentially a replication of Jon and Pat’s earlier study. The research designs were nearly identical.  The data were almost the same.  The only difference was that Jon and Pat had a more complete data set and as a result reported more negative results.

That’s right.  Jon and Pat had more negative results.  They released those results along with the negative 2nd year results in February 2016.  So the fact that Jon and Pat continued getting access to LA data while APW did not does not appear to have anything to do with reporting negative results.  It seems to be related to the fact that APW were rushing to release results.  They didn’t take the time like Jon and Pat did to solve missing data issues.  Instead they were determined to move fast to get their results out first.

Why did  it matter that they be first?  By being first to release they could act like they had the original analysis rather than a replication.  Top Econ journals tend not to be as  interested in replications of a grad student’s dissertation.  And by being first to release and not citing Jon’s work they could act like theirs was the original analysis.

Failing to credit and cite earlier work is a form of academic fraud.  I have not come forward earlier with this story because Jon was entering the academic job market and did not want to get on the wrong side of high status and powerful people in the field.  Pat and I, as his advisers, deferred to his wishes and remained quiet.  Now that Jon has a secure job ( with us) and a news article wrongly implies that APW were denied access because (presumably unlike us) they wouldn’t withhold negative result, I felt compelled to tell this story.  It’s an ugly one.

UPDATE: Pat Wolf checked his records and found that he also had a discussion at a conference in April 2015 with Atila regarding the Louisiana evaluation that he and Jon were doing. The materials he sent, however, were following that conversation, not following the June conversation as Pat had earlier remembered, and those materials were not directly related to the study.  In any event, it is clear from multiple conversations and multiple conference presentations that APW were aware of the existence of prior research.

2nd Update:  APW have a statement here.  My response to it is here.

(Edited for typos and to add links)

Read the rest of this entry »

Education policy debates, like national political disputes, are increasingly frustrating because we are having difficulty agreeing on and sticking to a set of facts.  It’s particularly frustrating because people are mis-stating basic facts that quite often are not even necessary for continuing to hold their opinions.  I’m not sure what accounts for this sloppiness in public discourse, but I suspect that it is a general decay in professional norms and standards of behavior encouraged by the stupid brevity and speed of Twitter.

Let me describe a recent example of an obvious, factual error from a debate about charter schools and regulations.  To be clear — there’s nothing wrong with making mistakes.  So what is remarkable about this example is how the people making the demonstrable, factual error never say “Oops, I made a mistake.”  They could easily admit the mistake and still continue to hold their opinion and make their arguments.  The fact that they don’t is what is truly troubling.

A Recent Example — Greg Richmond, who is the head of the National Association of Charter School Authorizers (NACSA), wrote a piece for The 74 in which he claimed: “A recent report from Tulane University’s Education Research Alliance found that a school’s rating from the independent application review process was the only factor that predicted its success.”  But if you look at the report, specifically in Table 7, you will see that the NACSA rating of charter school applications is not a significant predictor of any student outcome.  Richmond’s claim is demonstrably false.  He says that a report found something that it simply does not find.  I had previously noted the falsehood of this claim, but this has not deterred the folks at NACSA from repeating it.

Rather than admit the error, NACSA tries to rescue the claim.  Importantly, they don’t rescue the claim by showing that it is true.  They can’t.  Instead, they shift the discussion to a different point.  NACSA’s Nelson Smith, echoed by their spokesperson Kristen Forbriger, quote a policy brief written by Doug Harris summarizing the findings of the report he co-authored and that they falsely invoke:

“None of the application measures predict the value-added performance of schools, though there are signs of a positive relationship between the NACSA ratings and value-added (emphasis added). It is not surprising that our statistical confidence is weak here because value-added measures are imprecise and the NACSA ratings did not vary much among approved applications.” In other words, it’s hard to detect correlation to specific outcomes when the approved applications all scored at high levels.

Note that this does NOT say that the report “found that a school’s rating from the independent application review process was the only factor that predicted its success.”  In fact, it clearly states that Richmond’s original claim is false: “none of the application measures predict the value-added performance of schools.”  And saying there are “signs of a positive relationship” is just spin, not the actual finding.

Rather than support the truth of the original claim, this quotation provides a rationalization for why the report did not unearth the desired finding — the sample size was too small and there was too little variation in the NACSA rating.  Whether the rationalization is persuasive or not, the fact remains that the report did not find what Richmond claims it found.  His assertion is demonstrably false.  But neither he nor his colleagues at NACSA will say so.  Instead, in Trump-like fashion, they continue to assert that they were right all along even as the evidence contradicts them.

Unfortunately, this recent example of a factual error that is never acknowledged or corrected is becoming part of a pattern.  If people in our field can make demonstrably false claims without having to acknowledge or correct them, it’s unclear to me how we are going to make progress in policy debates.  I am not arguing here what the correct policy should be.  I’m simply arguing that if people in these debates can make false claims with impunity, we have allowed the Trumpization of debates to creep into the area of education policy and will suffer similar highly negative consequences.

Despite living in Fayetteville, AR, which is the greatest center of Yiddishkeit in all of the Ozarks, I had to travel to Boston to see the wonderful new film, The Wedding Plan, in a theater.  You should make an effort to see it if you live in a city where it is now showing and, if not, you should plan to see it when it arrives eventually on Netflix.

The Wedding Plan is the second film from writer/director Rama Burshtein, following her debut movie that I also loved and previously reviewed on JPGB, Fill the Void. Like the previous film, The Wedding Plan is set in an Orthodox Jewish community in Israel and focuses on questions of love and marriage.  But unlike Fill the Void, The Wedding Plan is not primarily about love and marriage.  It is really about faith and whether it is reasonable to expect that good things will happen — perhaps even miracles.  In this sense it is more like another fantastic film set in an Israeli Orthodox community, Ushpizin.

The premise of The Wedding Plan is that Michal is finally set to be married when her her fiance gets cold feet and backs out a month before the nuptials.  She has the dress and the wedding hall is reserved, so she decides to go ahead with the plan.  All she needs is a groom.  I know this sounds like the sort of Rom-Com plot that might feature Julia Roberts hilariously racing from one bad date to another until she finds Mr. Right just in time (and the American trailer has that feeling), but this movie is about much more.

The Hebrew title of the film means Through the Wall, which highlights a scene in which Michal explains that her plan to get married by the end of the month is like a karate chop.  She says she has to believe 100% to break through the wall.  If it is 99% she’ll break her hand.

Her friends and family fear that she will break her hand regardless.  They bring a rabbi to advise her. “What you are doing is counting on miracles,” he tells her. “Who gave you the right?” She answers quoting the Talmud: “The world was created for me.”  Is this just hubristic entitlement or should each of us feel like there is a plan for oneself in the world that includes enjoying the good?

Given the excessively cynical — even nihilistic — tilt of our culture and politics, it is worth considering whether we might be better off just having confidence that good things will somehow work out.  Or does believing this just set us up for even greater disappointment?  As Michal says, sitting alone in her wedding dress, “The bubble is about to burst.”  Should we protect ourselves from disappointment by not expecting anything good to come from the world?

These are questions that a Julia Roberts movie would not ask, but The Wedding Plan does.

And here is the American trailer:

I’ve written several times before about the disconnect between changing test scores and changing life outcomes.  In general, even when we can rigorously identify changes in math and reading test results caused by schools or programs, we have weak and inconsistent evidence that this produces commensurate changes in later life outcomes.  Keeping in mind that test-based accountability mostly focuses on the level of test scores, not changes, and virtually never relies upon a rigorous identification of how test scores are caused by schools and programs, we have no way of knowing that that the kinds of schools, programs, and practices that we are pushing in education will actually help kids later in life.  We might actually be hurting them and are certainly foreclosing potential opportunities based on our false confidence that we (policymakers, researchers, and pundits) are better at judging quality than are families.

Rather than contemplate the implications of our ignorance, people in our field are inclined to press ahead.  Yes, our measures are imperfect, they’ll admit, but we have to do the best we can with what we have.  Or they will say we just have to try harder to develop better measures of knowledge acquisition or expand our measures to include non-cognitive skills that provide a more complete picture of the recipe for success.

But what if there is simply no recipe for success?  Or more precisely, what if the recipe for success is highly context-dependent so that distant policymakers, researchers, and pundits are unable to prescribe what programs and schools should be cooking?  Maybe only parents, communities, and local educators are well-enough positioned to make reasonable (if imperfect) judgments about what each child needs.

If you were holding out hope that the expansion of educational measures to include non-cognitive skills would give policymakers, researchers, and pundits a stronger ability to prescribe how and where students should be educated, I have some bad news for you.  The disconnect between educational measures and later life outcomes is at least as severe in non-cognitive measures as it is in test scores.  Let’s leave aside the fact that most non-cognitive measures are too easily gameable to be used for any accountability purposes.  Even if only for research purposes, there does not appear to be a straightforward and consistent connection between non-cognitive measures and later life outcomes.

A new study led by Nicholas W. Papageorge at Johns Hopkins University and IZA examines the connection in Great Britain between teacher reports about behavior when students are 11 and later life outcomes for those students.  Because non-cognitive measures are in their infancy, we aren’t entirely sure how to slice and dice the measures and do not have a clear system for labeling the related concepts we are measuring.  In this study, if we simply lumped all of the teacher reports of misbehavior together we would find that students who misbehave more tend to do worse later in life.

But if we split misbehavior into two categories — one that captures misbehavior directed toward others (externalizing) and another that captures whether students are misbehaving because they are withdrawn (internalizing) — the picture gets more complicated.  Students who score poorly on measures of internalizing misbehavior still seem to fare poorly later in life.  But for students who score lower on the externalizing misbehavior, how they fare later depends on their social class.  If students are from more advantaged backgrounds, externalizing is actually associated with higher earnings, while for more disadvantaged students externalizing seems to have no effect on earnings.

This null to positive effect on earnings for a certain type of misbehavior occurs despite that fact that externalizing is associated with lower levels of educational attainment.  That is, students who misbehave toward others don’t go as far in school, but they earn more in the workplace if they come from more advantaged backgrounds despite that negative effect on educational attainment.

Yet gain, we see that what we think is “good” performance on a a near-term educational measure is highly dependent on context and is not connected to later outcomes in a straightforward way.  Policymakers, researchers, and pundits are inclined to say that scoring higher on a measure of behavior is better, but that is not necessarily the case.  It depends on the type of behavior and who the student is.

In addition, things we do to increase educational attainment may come at the expense of later earnings.  Increasing compliance with school authorities may help students go further in school, but may stifle the initiative and ambition necessary to make larger contributions to the economy later.  Policymakers, researchers, and pundits cannot simply identify a set of educational measures (test scores, non-cog measures, or educational attainment) and judge from afar which programs and schools are going to help students succeed later in life.  You can’t just maximize these measures and expect uniformly good results.

None of this should be surprising to parents engaged in the complicated task of raising their children.  We often want our children to possess certain qualities — but not too much of those qualities.  We want our children to be obedient, but not too obedient.  We want them to be ambitious, but not too ambitious.  We want them to value abstract knowledge, but not too much abstract knowledge.  We want them to value being in school, but not stay in school forever.  And we know that the approaches we take to produce these balanced outcomes vary for each child, even within our own family let alone across all families.

As it turns out, educating children is simply an extension of raising children to be the kinds of adults we hope they can be, requiring all of the same nuance, balance, and judgment.  Solutions imposed by distant policymakers, researchers, and pundits are no more likely to be effective in educating our children than they would be in raising them.  Yes, parents, communities, and local educators will make mistakes in raising children as well as educating them.  But they are better positioned to understand the context and achieve the appropriate balance for each child than are distant policymakers and experts, even if they are well-intentioned and highly knowledgeable.

The German movie, Er ist wieder da (available on Netflix as Look Who’s Back), is no ordinary political comedy.  On one level it’s quite disturbing and not a comedy at all, while at the some time it is a brilliant and hilarious satire.  The premise is that Hitler is somehow not dead and finds himself in modern Berlin.  He’s taken in by a desperate free-lance film-maker who introduces him to a set of conniving TV executives seeking ratings with what they think is a comedy act.  But he’s no comedian.  He’s really Hitler, adapting to our times and re-building political support.

There are many successful movies featuring Hitler, including The Producers, The Great Dictator, and (one of my all time favorites) Inglourious Basterds, but they generally portray Hitler as a buffoon.  In this film Hitler is occasionally buffoonish, but he is also a keen observer of people and politics.  He immediately detects that Germany’s current nationalist party, the NDP, would be an inadequate vehicle for his return to power.   He even storms their headquarters and denounces them as a pack of losers, which they obviously are.  Instead, he sees potential for the rise of authoritarianism in the Green Party.  That’s both astute and hilarious.

The film also mixes scripted scenes with improvised ones in which Hitler encounters real people on the streets of Germany.  He’s shown (for the most part) being warmly received, with people taking selfies and laughing.  Perhaps they also think he is a comedian and are going along with the joke.  But others give him Nazi salutes and describe their complaints about immigrants.  He listens as a very sympathetic and effective politician.

At one point a “man on the street” share his vision of democracy with Hitler.  He says that we need a democracy that is willing to put its foot down more: “That’s how we do it! Point, finished! No discussion!”  Hitler replies, “You’re absolutely right, and that is exactly my kind of democracy!”  Again, both astute and hilarious.

There are many ways a movie like this could go wrong and at times it does go off the rails, but not very often.  The film could become a heavy-handed parable about today’s nationalist politicians.  It avoids that by emphasizing how today’s nationalists lack the skill and energy that Hitler possessed.  At the same time, the film does not attribute to Hitler magical powers to hypnotize us into backing authoritarianism.  It shows us as either wanting authoritarianism or being too easily distracted by frivolous things to bother to stop it.  Hitler is just capable of exploiting that opportunity.

The film’s mixture of real and fictional, comedy and serious commentary, are disorienting.  At one point a TV executive warns what she thinks is comedian Hitler not to do jokes about Jews because they aren’t funny.  Hitler agrees that there is nothing funny about Jews.  That is simultaneously serious commentary and a hilarious joke.

Look Who’s Back at times feels like Borat but it is more like the brilliant 1976 Oscar-winning movie, Network.  It’s a satire that is often more serious than funny and more disturbingly accurately than most dramas.  If you haven’t seen Network you really should.  It’s as if the script writer, Paddy Chayefsky, had a time machine and could see how TV news would turn into its current manifestations of 24 Hour News channels and Twitter.  Let’s hope Look Who’s Back is not similarly prescient.

Some people have been puzzled as to why I’ve been studying how cultural activities, like visiting an art museum or seeing live theater, affect students.  Why don’t I do what almost everyone else in our field does and just study how various interventions affect math and reading test scores?

Well, I’ve been making the argument for a while now that there is remarkably little evidence linking near-term changes in test scores to changes in later life outcomes for students, like graduating high school, enrolling in college, completing college, and earnings.  I have yet to see anyone bother to refute my observation of this weak and inconsistent connection between test score changes and life changes.  No matter, researchers, foundations, and policymakers continue to plod along as if changing test scores should be the focus of our efforts. Whether kids go to art museums or see live theater is at best an amusing sideline or at worst a harmful distraction from the primary goal of education, which they believe is boosting math and reading test scores.

But now we have a rigorously designed study out of Denmark that shows cultural activity among students is strongly (and likely causally) related to later academic success.  The study appears in Social Science Research, a Sociology journal that was co-founded by James Coleman.  It examines a large sample of monozygotic twins in Denmark to see if their cultural activity was related to their teacher-given GPA, exam-based GPA, and rate of completing secondary school.  To measure cultural activity they relied on a survey administered to the mothers of those twins that asked about what their children did when they were 12 years old.  It asked things like: “How often child went to any type of museum” and “How often child went to the theater or a musical performance.”

By comparing outcomes among identical twins, the researchers hope to control automatically for a large set of unobserved environmental and genetic factors.  We could reasonably believe that a large portion of the variation in cultural capital among twins was due to chance and not differences in their upbringing or ability.

The researchers found that the twin whose mothers reported having higher cultural capital at age 12 had significantly higher marks on their end of compulsory school exams at age 15/16.  They also found “an
increase in cultural capital of one standard deviation is estimated to increase the likelihood of completing upper secondary education by 12.5 percentage points.”

Cultural capital was not a significant predictor of the grade point average students received from their teacher when they were 15, which was contrary to the researchers’ expectations.  Earlier theory had suggested that cultural capital might improve academic performance by making students falsely appear more knowledgable, even if their command of the material were no greater.  As they put it: “Bourdieu argued that cultural capital, that is familiarity with the dominant cultural codes in a society, is a key determinant of educational success because it is misperceived by teachers as academic brilliance and rewarded as such.”

This study found that not to be the case.  Instead, their findings are more consistent with the arguments advanced by E.D. Hirsch and others that cultural capital gives students a stronger foundation of broad knowledge that then facilitates future knowledge acquisition.  And the significant increase in completing secondary school may be a function of that broader knowledge, as opposed to the narrow knowledge captured in math and reading standardized tests.  Cultural activity may also increase graduation rates by giving students more ways to be engaged with school on top of traditional academic coursework.

So the next time someone asks me why it matters whether students go to art museums or see live theater, I can tell them that there is at least as much rigorous evidence showing the long term benefits of cultural activity as there is for interventions designed to boost standardized test scores.

(Guest post by Patrick J. Wolf)

We parents all have heard the claim that something wasn’t fair.  “Suzie got a bigger piece of cake than I did!”  “Tommy got to go fishing while I had to clean the garage!”  “Malachi had a lot more money spent on his education because you sent him to a traditional public school and me to a public charter school!”  Well, maybe we haven’t actually heard that last one very often but it would be a more legitimate gripe than the other ones.

Students in public charter schools receive $5,721 or 29% less in average per-pupil revenue than students in traditional public schools (TPS) in 14 major metropolitan areas across the U. S in Fiscal Year 2014.  That is the main conclusion of a study that my research team released today.    Of the cities we examined, some have large and well-established charter sectors, like Houston, Los Angeles, New York, and Washington, while others have more emerging charter school sectors like Little Rock, San Antonio, and Tulsa.

Twelve of the 14 cities have a disturbing charter school funding gap of more than 10%, which earned them a C grade or lower.  Tulsa, Little Rock, Indianapolis, Washington, Los Angeles, Oakland, and Camden earned an F for funding equity since there is a funding gap of more than 30% between what charter schools received versus what TPS received per pupil. Camden had the largest per-pupil funding gap in our study, with charter schools students receiving 45%, or $14,771, less per pupil than TPS students.

Shelby County, TN, which includes the city of Memphis, is the only metropolitan area in the study that funded students in public charter schools at a higher level than TPS.  Shelby County charter students received $10,624 in per-pupil funding in FY 2014 compared to $9,720 per student in the county’s TPS.  Houston’s charter schools were funded just 2% below their TPS, and earned the only grade of A in the study, in part because they were able to raise almost $900 per student in nonpublic revenue.  Funding gaps of $1,500 per student or more for charters remained in 10 of the 14 cities even after excluding all special education expenditures from the comparison.

The main source of the funding gap is local revenues.  Traditional public schools received $7,000 more per pupil in local revenues, on average, than did public charter schools.  Charter schools are public schools, in local communities, that must enroll all students who want to attend (or hold a random admissions lottery).  The parents of charter school students pay local taxes just like the parents of TPS students.  The fact that eight of the 14 cities in our study provided essentially no local education revenue to their public charter schools is shameful.  That’s simply not fair.

Our previous study of charter school funding equity at the state level was criticized for not exempting expenditures on such items as transportation and central administration that are mandatory for TPS but discretionary for public charter schools.  In our view, that’s exactly the point.  Charter schools are permitted to be innovative as an alternative to the more rigidly controlled administrative and spending structure of TPS.  True, the revenue amounts received by charters and TPS are more even once you exclude all of the ways that public schools are forced to be inefficient.  Like I said.  That’s exactly the point.

What are the takeaways for education policy?  Our results support the recommendations of the Fordham Institute and others to fund students directly, using a weighted student funding formula, a.k.a. “backpack” funding.  Placing public charter schools on a par with TPS in receiving local educational funds, as Colorado plans to do, would bring over half the cities in our study to funding parity across the two public school sectors.

States like Massachusetts, Texas, and Denver have tried to compensate for local funding discrepancies in their charter sectors by providing higher state funding to charter students, but that move hasn’t closed the funding gap.  It merely got Houston close enough so that the extra-ordinary fundraising efforts of its charter schools were able to move charter students close to parity.  Such bricolage arrangements are simple guesswork and no substitute for a rational student-based funding policy that treats the same student similarly regardless of the local public school their parent chooses for them.  Ask your children.  Anything else is just not fair.