John Katzman Has It Totally Right on Ed Reform

November 2, 2016

Every donor, every foundation or advocacy staffer, every academic, or anyone else who cares about having an intelligent strategy for improving education should watch this video.  It’s as if he’s been reading this blog and stealing our thoughts, but Katzman puts it all together in an incredibly compelling way.

The bottom line is improving schools is going to require more markets and choice and less testing and accountability.  And you don’t have to worry too much about testing and accountability because it is so politically unpopular that it will mostly destroy itself.

Win-Win Update at Perspective

November 1, 2016


(Guest post by Greg Forster)

OCPA’s Perspective carries my article on the research showing school choice improves academic outcomes at public schools – including an update since the publication of my latest Win-Win report, with a new study from Fordham finding Ohio’s voucher program improves reading and math scores in affected public schools:

This study improves on the two previous studies of the Ohio program (one of which I conducted), both of which also found it improved public schools. Figlio and Karbownik had access to individual student data, rather than having to use aggregate school-level scores, which is more accurate. They also use a “regression discontinuity” method, comparing schools that landed just barely above and just barely below the threshold for voucher eligibility. This is a better apples-to-apples comparison of schools.

But why be surprised? These days even NEPC and Christopher Lubienski have finally admited that school choice improves outcomes, although only participant outcomes. Lubienski mysteriously refused to look at my report’s review of effects on public schools, fiscal effects, segregation effects and civic effects. I can’t imagine why!

The win-win solution continues to rack up wins. Stay tuned.

Losing My Religion?

November 1, 2016

My former students, Dan Bowen and Albert Cheng, have a new study that was just published in the Journal of Catholic Education on how religious priming may affect student character or non-cognitive skills.  They find that priming students to think about religion increases students’ willingness to delay gratification as well as their political tolerance.  No effects are observed if students are instead primed to think of secular success.  This work suggests that there may be particular benefits from religiously-based education that are more difficult to produce in a secular context.  Abandoning private religious education for secular charter schools may come at a cost to these character skills.

These results come from an experiment we conducted at the Arkansas School for Mathematics, Sciences and the Arts (ASMSA), a public boarding school in Hot Springs.  In the experiment we randomly assigned 180 students to one of three conditions.  All students were asked to work on a sentence scramble exercise in which there are ten sets of five words.  Students were asked to look at each set of five words, drop one word, and then make a sentence out of the remaining four words.

In 5 of the sentences one word was altered, changing only five of the 50 words across the three conditions.  One group was primed to think about religion by having the words “worship, preacher, heaven, devotion, and commandments” included in the sentence scramble.  A second group was primed to think about secular equivalents: “honor, leader, success, commitment, and expectations.”  And a third group saw neutral words: “eat, path, man, cabbage, and numerous.”

Prior research had found that students primed in this way to think about religion demonstrated higher levels of self-regulation.  The idea of this experiment was to attempt to replicate those previous findings while exploring whether secular equivalents could produce similar effects.  Several observers have noted that KIPP and other high-achieving charter schools appear to simulate the religious rituals of Catholic schools but replace religious rhetoric with talk of secular success and achievement.  The question this study explores is whether talk of secular achievement appears to be as motivational for students as religious rhetoric.

Dan and Albert find that something is lost when we substitute secular aspirations for religious ones.  Students exposed to the religious priming expressed a stronger feeling of religiosity.  So, the priming worked in getting students to think about religion, even though changing only 5 words out of 50 is very subtle and the students were not consciously aware of the nature of the manipulation.

Students exposed to this religious priming experienced an increase in delayed gratification in that they were more willing to receive $6 the following week as compensation for participating in the study rather than $5 right then.  Students in the religious priming condition were also more likely to express political tolerance on the Sullivan scale, which measures people’s willingness to allow disliked groups to engage in political activities, like holding rallies, having books in the library, or running for office.  But students exposed to the secular success priming were no different from the neutral priming control group in that both were less likely to delay gratification or express political tolerance.

This was a small scale experiment and the effects were observed at p<.1, so there are limits to how confident we should be about the results.  But given that the results are consistent with prior research, we should have some concerns about dropping religious school choice in favor of secular charter schools.  This is especially so given that the No Excuses charter model that has become the darling of ed reformers often comes up short at improving later life outcomes, while private school choice programs seem to fare better at improving high school graduation, college enrollment, and even earnings.

And the Winner of the 2016 “Al” is… Master Sergeant Roddie Edmonds

October 31, 2016


This year’s set of Al Copeland Humanitarian Award nominees was particularly strong, making selection of a winner exceptionally difficult.  As Greg noted in a comment, “this is clearly a ‘political’ year for The Al, in the sense that we’re all nominating witnesses against injustice rather than the creative entrepreneurs who usually dominate.”

Well, almost all.  Matt, as is his habit, nominated the entrepreneurs, Tim and Karrie League, who developed the Alamo Draft House chain of movie theaters.  The Alamo Draft House is one of the greatest places on earth.  The theaters carefully select movies, audience activities, food, and drink to create a completely engaging and entertaining experience.  Some people give hundreds of millions of dollars to art museums that fail to package their offerings nearly as well as Tim and Karrie League do.  And the Alamo does it without any donations while making a profit.  Improving the human condition while also making profit is a quintessential characteristic of winners of The Al.  And I almost slected Tim and Karrie League for this honor.

But as Greg said this seems like a political year in which selecting a traditional entrepreneur-type as the winner just didn’t seem right.  All of the other nominees fell in the “witnesses against injustice” category and with so much injustice all around us, I felt like I should choose one of them.  I could have chosen Jason’s excellent nominee, Remy Munasifi, whose musical parodies expose and help rebut oppression, hypocrisy, and other types of foolishness.  I could also have chosen my own nominee, Yair Rosenberg, whose trolling of neo-Nazis and other anti-Semites on Twitter deprives these bullies of the sense of power that drives much of their behavior.

Instead, I have chosen Master Sergeant Roddie Edmonds over Remy Munasifi and Yair Rosenberg because Edmonds was more than a witness to injustice.  He actively took steps, at enormous danger to himself, to promote justice in the world.  By refusing to comply with Nazi orders to separate Jewish POWs and insisting that he and all of the soldiers under his command were Jewish, Edmonds risked being shot to defy the Nazi’s hateful and murderous plans against Jews.

I hesitated for a moment in selecting Edmonds only because The Al does not typically go to people who have been widely recognized elsewhere, like Steve Jobs or John Lasseter, and Edmonds was recently honored as one of the Righteous Among the Nations at Yad Vashem, Israel’s Holocaust museum.  Unfortunately, being honored by Yad Vashem does not constitute being widely recognized, so drawing more attention to Edmonds seems important and fitting.

As much as I love Remy Munasifi and Yair Rosenberg, mocking injustice on the internet just isn’t enough.  As Ken M, last year’s winner of The Al, taught us, social media is a pretty useless forum for trying to improve the world. So, that silly video you shared on Facebook or that sly remark you made on Twitter doesn’t really do much other than amuse you.

There’s nothing wrong with some amusement. After all, that is the Prime Directive of this blog — to amuse ourselves rather than to change the world.  And being amusing is a lot better than those insufferable political rants or self-righteous internet petitions, which are all talk and no action.

If you want to fight injustice you can’t really do much with a blog, Twitter, or Facebook.  You need to find real injustices, not trumped-up (pardon the pun) minor slights like:

Image result for I am a cat not a costume

And then you need to follow the example of Master Sergeant Roddie Edmonds and take action that might even put yourself at risk.  Evil will always remain in the world, but we will suffer less from it if we have more people like Master Sergeant Roddie Edmonds.

Don’t Know or Don’t Care?

October 28, 2016


When we examine the results of standardized test scores we typically think we are seeing evidence of what students know.  As it turns out, that is only partially right.  Test scores capture both what students know as well as their willingness to exert effort to show us what they know.

A new paper by my colleagues, Gema Zamarro, Collin Hitt, and Ildefonso Mendez uses multiple, novel techniques to demonstrate that between 32% and 38% of the variation in PISA test performance across countries can be explained by how much effort students are willing to exert rather than what they know.  The implications of this finding for ed reform are huge.  When we see low test score performance we are often misdiagnosing the problem as poor content instruction when it may in fact be insufficient development of student character skills.  If we focus all of our energy on the former without addressing the later, we’ll fail to make as much progress.

So, how do Gema, Collin and Ildefonso know that between 32% and 38% of variation in PISA test performance across countries is explained by effort?  They used three different methods to measure the influence of effort.  First, they took advantage of the fact that the order of questions without the PISA was randomly ordered.  They then compared how well students performed on the first set of items relative to the last.  Because the order of items was randomized the first and last questions were, on average, of equal difficulty.  The decline in getting items correct from the start to the end of the exam is therefore a function of the decline in effort students are willing to exert, not the difficulty of the items.

If you compare performance in the US and Greece (as can be seen in the figure above), students in the two countries do about as well at the beginning of the test.  That means that students in Greece and the US know about the same amount of stuff.  But students in Greece decline much more rapidly across the test, which means that those students are less willing to exert consistent effort.  When we compare PISA results from the US and Greece we wrongly conclude that content instruction in Greece must be much worse.  In reality, Greek students know as much as students in the US but simply exert a lot less effort.

A second way the paper measures effort is by examining responses students gave to a survey that was administered at the same time as the PISA.  Using novel techniques that have been validated in previous research, they measure the extent to which students skip answers (or say “don’t know”) as well as the extent to which students give careless answers as proxies for their effort.  Both skipped answers and careless answers yield very similar results to what they find from the decline across the test.

Some people have expressed skepticism about the focus on “non-cog” or character in education research because they believe that these capture personality traits that are largely inherited and immutable.  This research contradicts that claim.  Unless we think there are big and important genetic differences across countries, the variation in effort across countries has to be explained by factors that are social constructed and, at least in theory, could be changed.  In addition, great work by Gema and Albert Cheng has found that student effort can actually be changed when students are randomly assigned to different teachers who themselves possess different character skills.

The evidence is becoming clear that character matters and is subject to influence by the education system.

Where Do You Consistently Find the Highest NAEP Scores? Where Everybody Knows Your Name

October 28, 2016

(Guest Post by Matthew Ladner)

So hidden deep in the NAEP data explorer is a variable for school enrollment.  Yesterday we saw how Arizona charter schools crushed the ball on the 2015 NAEP science exams, but I was curious- would there be evidence suggesting that small schools of choice perform especially well? NAEP provides such a number in a crosstab for Arizona charter/district by school enrollment. Small district schools in Arizona performance is nothing to write home about, and are probably mostly rural. Arizona’s small charter schools-schools of choice-however, well, that is a different story. These are the 8th grade science NAEP scores for Arizona charter schools with 399 or fewer students compared to statewide averages for all students:


I thought that was interesting, so I checked to see how this would look in the 2015 NAEP Reading exam for 8th graders. Well-


Well but the whole thing would fall apart in the math test. Except, it didn’t:


Obviously this evidence is only suggestive, but do keep in mind that we have a large number of formal studies finding positive outcomes associated with attendance at small high schools. So perhaps high quality education involves authentic community with a shared vision of what constitutes high quality learning, and this process is facilitated by the ability of a child and parent to choose. It certainly appears to be the case out here in the Cactus Patch. Let’s call it the “Cheers theory of learning” in that you want to go where everybody knows your name. If that is you want to learn to read, figure some math, and understand science. If you prefer to fade into the background and then drop out of school- we’ve got plenty of Big Box schools to choose from as well.

So you see dare Normy....

So you see dare Normy dare used to be this big Foundation that had a great idea but then…

Arizona Charters Blow the Doors Off 2015 NAEP Science Gains

October 27, 2016

(Guest Post by Matthew Ladner)

Ok sports fans, I know you are all chomping at the bit, wanting to know “but Ladner, how do Arizona charter school science gains compare to statewide averages?!?” Oh I am glad you asked, here it is for 4th grade:


Hmmm…almost twice the gain as top ranking Arizona as a whole. Would it be running up the score to note that Arizona would not have done nearly as well without charters? I’ll just skip that part for now. Here are the NAEP 8th grade science gains:


Well, would you look at that-twice as large as the largest state gain.  I’m crunching these out on a Prescott Library computer after taking a mountain bike ride on a “day off” but feel free to run the numbers for yourself here. I’ll breakdown subgroups later when I have more than 16 minutes left on my public computer use.

In the meantime I’m just going to go out on a limb here and say that there just might be something to this whole parental choice thing. Just maybe.