Cultural Activity Matters

May 20, 2017

Some people have been puzzled as to why I’ve been studying how cultural activities, like visiting an art museum or seeing live theater, affect students.  Why don’t I do what almost everyone else in our field does and just study how various interventions affect math and reading test scores?

Well, I’ve been making the argument for a while now that there is remarkably little evidence linking near-term changes in test scores to changes in later life outcomes for students, like graduating high school, enrolling in college, completing college, and earnings.  I have yet to see anyone bother to refute my observation of this weak and inconsistent connection between test score changes and life changes.  No matter, researchers, foundations, and policymakers continue to plod along as if changing test scores should be the focus of our efforts. Whether kids go to art museums or see live theater is at best an amusing sideline or at worst a harmful distraction from the primary goal of education, which they believe is boosting math and reading test scores.

But now we have a rigorously designed study out of Denmark that shows cultural activity among students is strongly (and likely causally) related to later academic success.  The study appears in Social Science Research, a Sociology journal that was co-founded by James Coleman.  It examines a large sample of monozygotic twins in Denmark to see if their cultural activity was related to their teacher-given GPA, exam-based GPA, and rate of completing secondary school.  To measure cultural activity they relied on a survey administered to the mothers of those twins that asked about what their children did when they were 12 years old.  It asked things like: “How often child went to any type of museum” and “How often child went to the theater or a musical performance.”

By comparing outcomes among identical twins, the researchers hope to control automatically for a large set of unobserved environmental and genetic factors.  We could reasonably believe that a large portion of the variation in cultural capital among twins was due to chance and not differences in their upbringing or ability.

The researchers found that the twin whose mothers reported having higher cultural capital at age 12 had significantly higher marks on their end of compulsory school exams at age 15/16.  They also found “an
increase in cultural capital of one standard deviation is estimated to increase the likelihood of completing upper secondary education by 12.5 percentage points.”

Cultural capital was not a significant predictor of the grade point average students received from their teacher when they were 15, which was contrary to the researchers’ expectations.  Earlier theory had suggested that cultural capital might improve academic performance by making students falsely appear more knowledgable, even if their command of the material were no greater.  As they put it: “Bourdieu argued that cultural capital, that is familiarity with the dominant cultural codes in a society, is a key determinant of educational success because it is misperceived by teachers as academic brilliance and rewarded as such.”

This study found that not to be the case.  Instead, their findings are more consistent with the arguments advanced by E.D. Hirsch and others that cultural capital gives students a stronger foundation of broad knowledge that then facilitates future knowledge acquisition.  And the significant increase in completing secondary school may be a function of that broader knowledge, as opposed to the narrow knowledge captured in math and reading standardized tests.  Cultural activity may also increase graduation rates by giving students more ways to be engaged with school on top of traditional academic coursework.

So the next time someone asks me why it matters whether students go to art museums or see live theater, I can tell them that there is at least as much rigorous evidence showing the long term benefits of cultural activity as there is for interventions designed to boost standardized test scores.


NSVF 2017

May 18, 2017

(Guest Post by Matthew Ladner)

I had the pleasure of participating in a debate/discussion on tensions in the education reform movement at the New Schools Venture Fund Summit yesterday in San Fransico. I attempted to make the following points:

  1. Big Tents are good and disagreements are okay, failure even is okay, but an unwillingness to learn from failure is a huge problem.
  2. If you hold focus groups on K-12 education you learn that the public hates current standardized testing practices and that the deeply misguided federal opt-out provision that passed the United States House of Representatives was no fluke.
  3. The failure of Question 2 in Massachusetts is screaming a warning into our deaf ear about the dangers of excluding non-urban communities from parental choice efforts. Everyone should go back and read the Rick Hess 2011 National Affairs piece “Our Achievement Gap Mania” in light of the Question 2 disaster. School choice needs to decide whether it wants to be Social Security or AFDC. For much of the history of the movement we chose an AFDC model where everyone pays in but only certain people benefit, and, well…http://www.youtube.com/watch?v=Ubw5N8iVDHI
  4. The federal government is deeply in a growing amount of debt and has $75 trillion in unfunded entitlement liabilities to contend with. The pendulum has already swung against a large federal role in K-12, and Uncle Sam’s solvency issues are likely to freeze it that way for a long while if one assumes a prioritization for programs like Social Security and Medicare. We are only five years out from half of the Baby Boomers reaching age 65.
  5. Folks should do their best to remain calm on private choice because it ultimately not a threat to public education and helps kids find a good fit school.

I have a minority viewpoint on K-12 reform, and I appreciate Stacy Childress including me in the discussion. We should have far more discussions like this, and less bomb throwing over social media.

 


Against Federal School Choice (Even Tax-Credit Scholarships)

May 16, 2017

img_1358

(Guest post by Greg Forster)

OCPA’s Perspective has posted the second of my two articles making the case against any federal school choice program that goes beyond D.C. schools – or other legitimately federal jurisdictions (other territories, military bases, etc.). This is only my own opinion; I recognize the reasons why others, including at EdChoice, are supportive of federal choice or are at least fed-curious. But I’m here to make the case in opposition.

Having already argued against federal vouchers, through Title I or by other means…

If we want to continue living in a democratic republic and not in a technocratic oligarchy, we should be fighting tooth and nail to resist the process of federal takeover, not strengthening it…[Moreover,] it would be the states, not the federal government, which would create systems for parents to access choice through Title I portability. And not just the states, but the education bureaucracies of the states. So the bureaucrats most directly threatened by school choice would be the ones designing the programs. In other words, these programs would be designed to fail.

…in my latest article I argue against federal tax-credit scholarships:

The idea behind federalism is that governance should be kept as close as possible to local communities. That is partly because big, distant legislatures and bureaucracies are not likely to serve people well if they’re not directly connected to them. And that’s still going to be a problem even if you do find a clever way to circumvent the Constitution’s legal barriers to national education policy…

I never thought I’d live to see freedom-loving activists demanding to have the future of school choice put into the hands of the IRS. I feel like Rip Van Winkle. What did I miss here?

Federal choice of any kind also involves a sacrifice of moral legitimacy, which is destructive for any policy and fatal to a reform movement:

Lately I’ve heard a lot of talk from my conservative friends about how wrong it is when distant, powerful elites who are culturally alienated from the population at large shove laws down our throats that we regard as unjust. The question is, do we dislike that because we would rather it was our distant, powerful elites imposing our preferred laws upon populations from whom we are culturally alienated, and who view those laws as unjust? Or because elites shoving things down people’s throats is inherently wrong, whoever does it?

I also canvas the danger we run of a high-profile, national political loss should the bill fail, and other fun topics.

The school choice movement has gained enormous ground by focusing on the states. Let’s stick with what works and not sell our birthright for a D.C. mess.


That’s Not Fair!

May 10, 2017

(Guest post by Patrick J. Wolf)

We parents all have heard the claim that something wasn’t fair.  “Suzie got a bigger piece of cake than I did!”  “Tommy got to go fishing while I had to clean the garage!”  “Malachi had a lot more money spent on his education because you sent him to a traditional public school and me to a public charter school!”  Well, maybe we haven’t actually heard that last one very often but it would be a more legitimate gripe than the other ones.

Students in public charter schools receive $5,721 or 29% less in average per-pupil revenue than students in traditional public schools (TPS) in 14 major metropolitan areas across the U. S in Fiscal Year 2014.  That is the main conclusion of a study that my research team released today.    Of the cities we examined, some have large and well-established charter sectors, like Houston, Los Angeles, New York, and Washington, while others have more emerging charter school sectors like Little Rock, San Antonio, and Tulsa.

Twelve of the 14 cities have a disturbing charter school funding gap of more than 10%, which earned them a C grade or lower.  Tulsa, Little Rock, Indianapolis, Washington, Los Angeles, Oakland, and Camden earned an F for funding equity since there is a funding gap of more than 30% between what charter schools received versus what TPS received per pupil. Camden had the largest per-pupil funding gap in our study, with charter schools students receiving 45%, or $14,771, less per pupil than TPS students.

Shelby County, TN, which includes the city of Memphis, is the only metropolitan area in the study that funded students in public charter schools at a higher level than TPS.  Shelby County charter students received $10,624 in per-pupil funding in FY 2014 compared to $9,720 per student in the county’s TPS.  Houston’s charter schools were funded just 2% below their TPS, and earned the only grade of A in the study, in part because they were able to raise almost $900 per student in nonpublic revenue.  Funding gaps of $1,500 per student or more for charters remained in 10 of the 14 cities even after excluding all special education expenditures from the comparison.

The main source of the funding gap is local revenues.  Traditional public schools received $7,000 more per pupil in local revenues, on average, than did public charter schools.  Charter schools are public schools, in local communities, that must enroll all students who want to attend (or hold a random admissions lottery).  The parents of charter school students pay local taxes just like the parents of TPS students.  The fact that eight of the 14 cities in our study provided essentially no local education revenue to their public charter schools is shameful.  That’s simply not fair.

Our previous study of charter school funding equity at the state level was criticized for not exempting expenditures on such items as transportation and central administration that are mandatory for TPS but discretionary for public charter schools.  In our view, that’s exactly the point.  Charter schools are permitted to be innovative as an alternative to the more rigidly controlled administrative and spending structure of TPS.  True, the revenue amounts received by charters and TPS are more even once you exclude all of the ways that public schools are forced to be inefficient.  Like I said.  That’s exactly the point.

What are the takeaways for education policy?  Our results support the recommendations of the Fordham Institute and others to fund students directly, using a weighted student funding formula, a.k.a. “backpack” funding.  Placing public charter schools on a par with TPS in receiving local educational funds, as Colorado plans to do, would bring over half the cities in our study to funding parity across the two public school sectors.

States like Massachusetts, Texas, and Denver have tried to compensate for local funding discrepancies in their charter sectors by providing higher state funding to charter students, but that move hasn’t closed the funding gap.  It merely got Houston close enough so that the extra-ordinary fundraising efforts of its charter schools were able to move charter students close to parity.  Such bricolage arrangements are simple guesswork and no substitute for a rational student-based funding policy that treats the same student similarly regardless of the local public school their parent chooses for them.  Ask your children.  Anything else is just not fair.


Do Parents Care About Test Scores?

May 8, 2017

yhvkb2a1iyrt6oh4hoys

(Guest Post by Jason Bedrick)

To test, or not to test? That is the question at the center of one of Matt Barnum’s latest and last articles at The 74 (he’s moving over to Chalkbeat). Taking opposite sides of the “testing and educational choice” question are Doug Harris and yours truly.

To test:

“If you’re not going to require anything, we’re not going to know anything about whether these programs worked or not,” said Doug Harris, a Tulane University professor who has done extensive research on school choice in New Orleans and has been a critic of DeVos’s approach to education reform. “When you put it that way, even the people who are somewhat supportive of the idea get a little squeamish.”

Not to (impose the state standardized) test:

“Since there is strong evidence that state testing mandates tend to have negative unintended consequences, such as narrowing the curriculum and distorting how schools teach kids, we’d rather that schools have the freedom to use the tests that are more closely aligned to their curriculum and give parents the freedom to choose the schools that work best for their kids,” said Jason Bedrick, of EdChoice, an Indianapolis-based group that backs voucher and tax credit programs. [links added]

… Bedrick, of EdChoice, says he’s fine with requiring schools to administer a national test, but not the state exam, even if it means that studying the program then becomes difficult.

“As researchers, we’d like the apples-to-apples comparisons that a single test provides,” he wrote in an email. “However, as policymakers, we also have to do what’s in the best interests of kids.”

“The main point of choice policies is to empower parents to choose schools based on the criteria that are important to them, not just to raise [specific] test scores,” Bedrick said, citing factors such as student-teacher ratio, curriculum, and college acceptances.

Later in the piece, Barnum cites me saying, “We don’t think it is necessary to impose standardized tests. If parents demand that information, schools will provide it.” Harris disagrees:

“I think that’s hooey … Schools themselves, even if they wanted to provide that kind of information, can’t provide that in a way that’s comparable across schools — you have to have a coordinating mechanism for information.”

If only the market had a coordinating mechanism for information that parents used! Oh right, not only do such mechanisms already exist, but they’re also growing in popularity, and they get far more web traffic than any state education agency website:

GreatSchools.org is not the only website that ranks pre-K-12 schools. The Internet search engines Yelp and Google offer school ratings, as do websites such as Schooldigger.com and Privateschoolreview.com. But with 40 million annual unique visitors, GreatSchools is the one most used, according to Alexa Internet, which tracks Web traffic.

The site’s founder and chief executive officer, Bill Jackson, says GreatSchools wants to be more than just a school ratings site: He sees it developing into an association that serves parents in the same way that the AARP serves retirees, or that AAA represents drivers.

Perfect information? Of course not. Lots of room for improvement? Absolutely. But they already do more than any state agency to empower parents with the info they want.

And “want” is the key word there. EdChoice’s More Than Scores survey of parents of tax-scholarship students in Georgia found that most parents do *not* place a very high value on test scores:

Student performance on standardized test scores is one of the least important pieces of information upon which parents base their decision regarding the private school to which they send their children. Only 10.2 percent of the parents who completed the survey listed higher standardized test scores as one of their top five reasons why they chose a particular private school for their child.

Just ask real parents, such as, well… Doug Harris:

Research, including Harris’s, suggests that parents do generally place significant value on a school’s test scores, but it may not be the biggest factor driving school preferences.

As a parent himself considering private schools, Harris says he’s never asked any of them for their test scores. [emphasis added]

“Most of these decisions, and this is true public, private, or otherwise, are based more on reputation, on what your neighbors say,” he said.

Tests can be informative for parents, but policymakers should dial back their obsession. After all, they measure only a thin slice of what we want schools to provide. Parents take a more holistic view of their child’s education, and the narrow focus on tested subject can have negative unintended consequences. For example, one researcher recently acknowledged that one reason there “hasn’t been as much actual innovation [in the charter school sector] as maybe the original charter folks hoped” is that test-based accountability is hampering innovation:

[W]hen you have intense test-based accountability it really restricts what you can do and to what degree you can innovate because you have to put so many of your resources towards the same end. There are only so many ways to make test scores go up. So, I think that really restricts what they can do.”

That researcher? Doug Harris.


D.C. OSP: Don’t Jump to Conclusions!

May 2, 2017

jump-to-conclusions-mat

(Guest Post by Jason Bedrick)

Much has been said about the recent IES study of D.C.’s Opportunity Scholarship Program (see the responses from Jay Greene and Neal McCluskey, as well as Matt Ladner’s response to a related “flawed and shallow” NYT op-ed), but that didn’t stop Marty Lueken and I from wading into the debate. Our main point: don’t jump to conclusions.

What was ignored by most of the media coverage of the report is that most of the “control group” of non-voucher students either attended a chosen charter school (42%) or a private school (10%), so using the IES results to bash school choice is simply irresponsible. Moreover, most coverage failed to report the significant differences in funding. As we note:

The average OSP voucher ($9,422 in FY 2015–16) is less than half the per-student amount received per pupil at D.C. charter schools (over $21,000) and less than one-third of the per-student amount received in the D.C. district schools (over $30,000).

Importantly, the expanded availability of school choice over the last decade has coincided (not coincidentally) with a substantial increase in overall parental satisfaction among non-voucher parents:

The first IES study of the OSP, published in 2007, found that 74 percent of scholarship parents gave their child’s school an “A” or “B” compared to only 55 percent of parents in the control group. A decade later, the control group’s satisfaction had risen to 72.4 percent, an increase of more than 17 percentage points. The greater availability of school choice means parents are more likely to find a school that meets their child’s needs.

Finally, we should reiterate that test scores are but one of many factors parents consider when deciding on a school. Indeed, the obsessive focus on boosting math and reading test scores may have unintended consequences:

[The] improved math and reading test scores among the charter and district schools may reflect a worrisome narrowing of the curriculum. The IES report noted that district and charter school principals reported more instruction time spent on math and reading than private school principals. It may be the case that there is more “teaching to the test” going on in these schools.

While tests can be informative, parents know their kids are more than scores.

Politicians should avoid using test scores as the sole basis for policy decisions, especially because, as Jay Greene has shown, there is an increasing disconnect between test scores and important measures of later-in-life outcomes. Moreover, evidence suggests that a more holistic education is better in the long run than a narrow focus on reading and math.

We have not yet discovered the educational panacea — and we likely never will. Until then, parents are in a better position than technocrats to figure out what’s best for their kids.


If You Mostly Care About Test Scores, Private School Choice Is Not For You

April 28, 2017

If you mostly care about test scores, private school choice is not for you.  Despite the vast majority of randomized control trials (RCTs) of private school choice showing significant, positive test score effects for at least some subgroups of students, some of those gains have been modest and other effects have been null for at least some subgroups.  And now we have two RCTs, in Louisiana and DC, showing  significant test score declines for at least some subgroups and in some subjects.  The Louisiana decline is large and across-the-board, but the significant, negative effect in the new DC study appears to be  “driven entirely by students in elementary grades not previously in a needs-improvement school.

People will quibble over why these new DC results showed at least a partial decline.  They will note that the prior RCT of DC vouchers showed significant test score gains after three years (although the p value rose to .06 in year four even as the positive estimate remained).  They will note that vouchers in DC are worth almost 1/3 as much as the per pupil funding received by DC’s traditional public schools and almost half as much as DC’s charter schools.  Imagine how they might do if they received comparable resources (and yes resources can matter if there are proper incentives to use resources productively).  They will note that almost half of the control group attended charter schools, so to a large degree this study is a comparison of how students do in vouchers relative to charters.

But these largely miss the point — the benefits of private school choice are clearly evident in long term outcomes, not near-term test scores.  In the same DC program that just produced disappointing test score effects, using a voucher raised high school graduation rates by 21 percentage points.  Similarly, private school choice programs in Milwaukee and New York City were less impressive in their test score effects than in later educational attainment, where private school students in both cities were significantly more likely to enroll in college.

But if what you really care about is raising test scores, you’d be pushing no-excuses charter schools.  Rigorous evaluations, like the one in Boston, show huge test score gains for students randomly assigned to no-excuses charter schools.  You don’t even have to have school choice to produce these gains.  The same team of researchers showed that schools converted into no-excuses charters as part of a turnaround effort produced similarly big gains for students who were already there and did nothing to choose it.  The lesson that a fair number of foundations and policymakers draw is that we don’t need this messy and controversial choice stuff.  They believe that they have discovered the correct school model — it’s a no excuses charter — and all we need to do is get as many disadvantaged kids into these kinds of schools as we can, with or without them choosing it.

Unfortunately, no excuses charters don’t seem to produce long-term benefits that are commensurate with their huge test score gains.  The Boston no excuses charter study, for example, shows no increase in high school graduation rates and no increase in post-secondary enrollment despite large increases in test scores.  It’s true that students from those schools who did enroll in post-secondary schooling were more likely to go to a 4 than 2 year college, but it is unclear if this is a desirable outcome given that it may be a mismatch for their needs and this more nuanced effect is not commensurate with the giant test score gains.

This same disconnect between test scores and later life outcomes exists in several rigorously conducted studies of charter schools, including those of  the Harlem Promise Academy, KIPP, High Tech High, SEED boarding charter schools, and no excuses charters in Texas.  While of course we would generally like to see both test score gains and improved later life outcomes, the thing we really care about is the later life outcomes.  And the near-term test scores appear not to be very good proxies for later life outcomes.

So, what should we think about these new test results from DC vouchers, showing some declines for students after one year in the program?  We already know from rigorous research that the program improves later life outcomes, so I don’t think we should be particularly troubled by these test results.  It may be that control group students are in schools that will fare as well or better on test score measures.  But we should remember that 42% of that control group are in the types of charter schools that other research has shown can produce giant test score gains without yielding much in later life outcomes.  And we know that treatment group students are in a program that has previously demonstrated large advantages in later life outcomes.

I understand that many reporters, foundations, and policymakers act like they mostly care about test scores and these new results from DC have them all aflutter.  But if people could only step back for a second and consider what we are really trying to accomplish in education, the evidence is clearly supportive of private school choice in DC and elsewhere.

(edited to correct error noted in comments)