These go to Eleven: New Research on KIPP

February 27, 2013

(Guest Post by Collin Hitt)

Mathematica released a major study of KIPP charter schools today. KIPP is primarily a middle school network, with schools across the country. The Mathematica study uses a random-assignment research design, making it at least the eleventh such study we now have on charter schools. It found significant gains in math, positive but insignificant gains in reading. So, again, every random assignment study yet conducted on urban charter schools finds positive effects.

The random assignment study was limited to 13 KIPP charter schools in six states. KIPP’s network is much larger than that. So the authors employed a matching technique in order to evaluate the impact of a larger number of KIPP schools: they compared KIPP students to other kids who on paper were nearly identical. Matching techniques are far less rigorous than lottery-based estimates. But, since Mathematica had lottery-based estimates against which they could compare their matching technique estimates, they were able to validate their matched sample of students as a credible comparison group. They found that their random assignment estimates closely tracked their matching estimates, at relevant schools.

So they employed their matching techniques at a larger sample of 41 schools.  Mathematica then concluded that after just three years in KIPP, students made gains in math, reading, science and social studies that ranged from 8 to 14 months of additional learning. In the parlance of Matt Ladner, “boom.”

5 Comments | charter schools | Permalink
Posted by Jay P. Greene

The Other (More Important) Value-Added Measure

February 26, 2013

(Guest Post by Collin Hitt)

Some teachers are better than others when it comes to raising test scores, which in turn can raise students’ earnings in adulthood. But test scores aren’t everything. A new study looks at whether individual teachers can have similar impacts on suspension rates, school attendance, GPA and even graduation rates. It finds that they can, and do.

To put the non-test score estimates into perspective, having an Algebra or English teacher 20 at the 85th percentile of GPA quality versus one at the 15th percentile would be associated with 0.09 and 0.054 higher GPA, respectively. For both subjects, a teacher at the 85 percentile of ontime grade progression quality versus one at the 15th percentile would be associated with being 5 percentage points (0.14σ) more likely to enroll in 10th grade on time. Given that not enrolling in 10th grade is a strong predictor of dropout, this suggests significant teacher effects on dropout…

That’s from Northwestern’s Kirabo Jackson, His working paper uses state-of-the-art value-added methods to identify North Carolina high school teachers who have significant impacts on test scores. He then uses the same methods to see which teachers have an impact on “non-cognitive” behaviors. One would expect – at least, I expected – that the teachers who raise test scores also raise non-cognitive outcomes. Not so.

For all outcomes, Algebra teachers with higher test score value-added are associated with better non-test score outcomes, but the relationships are weak…This indicates that while teachers who raise test score may also be associated with better non-test-score outcomes, most of effects on non-test score outcomes are unrelated to effects on test scores. The results for cognitive ability are consistent with this…

Results for English teachers follow a similar pattern. English teacher effects on English test scores explain little of the estimated effects on non-test score outcomes…

Because variability in outcomes associated with individual teachers that is unexplained by test scores is not just noise, but is systematically associated with their ability to improve typically unmeasured non-cognitive skills, classifying teachers based on their test score value added will likely lead to large shares of excellent teachers being deemed poor and vice versa.

So teachers can have large effects on matters that are supposedly out of their hands. Suspensions, absence rates and GPA are functions of a student’s conscientiousness – or more generally, of his character. This study delivers another blow to the cop out lobby.

But it also presents a huge challenge to the proponents of test-based teacher policies. Read this line again: “classifying teachers based on their test score value added will likely lead to large shares of excellent teachers being deemed poor and vice versa.”  This is not a trivial matter. Jackson shows that non-cognitive outcomes are more strongly correlated with life outcomes than are test scores – especially for students with limited cognitive ability. This paper cannot be ignored.

Hat Tip: Joanne Jacobs

[Edited to correct formatting error and typos]

4 Comments | civic values, teacher quality | Permalink
Posted by Jay P. Greene

Chingos Strikes Again

February 22, 2013

Yesterday, I blogged about a new study by Matt Chingos and Marty West about pension reform in Florida.  Now I see that Matt has struck again with a great study about on-line learning in the current issue of Education Next.  Matt, along with co-authors William Bowen, Kelly Lack and Thomas Nygren, conducted a random assignment evaluation of an online statistics course that was offered at six universities.

Students were assigned by lottery either to a traditional course or a course where the bulk of the instruction was provided by inter-active software supplemented by weekly discussion sections.  The bottom line is that students did no better or worse in measured learning outcomes regardless of whether they received the course in the traditional way or via the internet.  The authors suggest that these results should temper wild claims about improved learning from online instruction as well as wild accusations that online fails to deliver.  They seem to be equally effective.  But the authors add that online delivery has significant potential to reduce the cost of delivering education and may have significant benefits for retention of students.

Here’s their conclusion in their own words:

In the case of online learning, where millions of dollars are being invested by a wide variety of entities, we should perhaps expect that there will be inflated claims of spectacular successes. The findings in this study warn against too much hype. To the best of our knowledge, there is no compelling evidence that online learning systems available today—not even highly interactive systems, which are very few in number—can in fact deliver improved educational outcomes across the board, at scale, on campuses other than the one where the system was born, and on a sustainable basis….

We do not mean to suggest that ILO systems are a panacea for this country’s deep-seated education problems. Many claims about “online learning” (especially about simpler variants in their present state of development) are likely to be exaggerated. But it is important not to go to the other extreme and accept equally unfounded assertions that adoption of online systems invariably leads to inferior learning outcomes and puts students at risk. We are persuaded that well-designed interactive systems in higher education have the potential to achieve at least equivalent educational outcomes while opening up the possibility of freeing up significant resources that could be redeployed more productively.

They also consider the implication of this higher education study for online instruction in K-12:

Extrapolating the results of our study to K–12 education is hardly straightforward. College students are expected to have a degree of self-motivation and self-discipline that younger students may not yet have achieved. But the variation among students within any given age cohort is probably much greater than the differences from one age group to the next. At the very least, one could expect that online learning for students planning to enter the higher-education system would be an appropriate experience, especially if colleges and universities continue to expand their online offerings. It is not too soon to seek ways to test experimentally the potential of online learning in secondary schools as well.

You can read the full article here.

[Edited to correct omitted co-author and for clarity]

3 Comments | Digital Learning | Permalink
Posted by Jay P. Greene

Chingos and West on Florida’s Pension Reforms

February 21, 2013

Matt Chingos and Marty West have a new paper published by Fordham examining pension reforms in Florida.  Specifically, Florida offered its new teachers the option of choosing between a defined benefit and a defined contribution retirement plan.  The defined benefit plan is the type most commonly found for teachers and defined contribution is more commonly found for private sector workers.

Defined benefit plans have some unusual characteristics that may push some teachers out of the workforce before they really should leave and may keep others as teachers longer than they should.  These defined benefit plans also reward long-serving and immobile teachers at the expense of shorter-serving and more mobile teachers, like those most commonly found in charter schools.  And defined benefit plans shift all of the risk for achieving sufficient investment returns to the government, which given recently weak investment returns, government under-funding of plans, and overly generous promised benefits is putting many states in serious financial trouble.

So states like Florida are considering shifting more teachers to defined contribution plans, which are more like 401k plans where the employer and employee each contribute money to an investment account and then the employee bears the risk of investment returns.

Matt and Marty addressed four questions in their study: 1) What portion of new teachers have chosen the defined contribution (DC) option? 2) What kinds of new teachers were more likely to make that selection? 3) Did the teachers who chose DC more likely to be effective teachers? and 4) Is there a difference in attrition between new teachers who choose DC or defined benefits (DB)?

The quick answers are 1) Between a quarter and a third of new teachers chose DC.  This is a surprisingly large share choosing DC, especially given that DB was the default option.  2) Teachers with more advanced degrees and degrees in math and science (presumably those with the most attractive options outside of teaching) were more likely to choose DC.  3) There was relatively little relationship between whether a teacher chose DC and their later effectiveness as measured by value-added scores. 4) New teachers who chose DC were more likely to leave their teaching positions.

Check out the full report to see all the details.

4 Comments | politics, teacher union | Permalink
Posted by Jay P. Greene

Aerial Inquiry in Texas

February 20, 2013

(Guest Post by Matthew Ladner)

A spirited debate over testing and accountability has been going on in Texas. The Texas Tribune ran the above photograph by Marjorie Kamys Cotera yesterday of the Texas Association of Business posing a rather important question via crop duster.

For those with eyesight as poor as mine, the message asks “Is 37 percent correct on Algebra too hard?” I believe it is, errr, a reference to cut scores…

5 Comments | teacher quality | Permalink
Posted by matthewladner

Interlude: Iggy and Kate Duet

February 16, 2013

(Guest Post by Matthew Ladner)

Random 1990 nostalgia flashback.

6 Comments | random pop culture | Permalink
Posted by matthewladner

Washington Post on Charter Schools in DC

February 15, 2013

(Guest Post by Matthew Ladner)

Check out this fascinating article in the WaPo regarding the ever growing market share of charter schools in the District of Columbia. Blended learning schools will debut soon, DCPS continues to shrink, making some tetchy. Money quote:

Rocketship’s charter application — which is the largest ever to come before District officials, and which might win approval this month — arrives on the heels of Chancellor Kaya Henderson’s decision to close 15 half-empty city schools, highlighting an intense debate about the future of public education in the nation’s capital.

A growing number of activists have raised concerns that the traditional school system, facing stiffer-than-ever competition from charters, is in danger of being relegated to a permanently shrunken role. And they worry that Washington has yet to confront what that could mean for taxpayers, families and neighborhoods.

“Maybe we need an entire school system full of charters,” said Virginia Spatz, who co-hosts a community-radio talk show on D.C. education. “But we need to have that after public conversation, not by accident.”

With due respect to Ms. Spatz, there doesn’t seem to be anything accidental about this to me- DC parents will ultimately decide how many charter and district schools they want by voting with the feet of their children.

8 Comments | charter schools | Tagged: , , | Permalink
Posted by matthewladner

SEED Charter School: The Charter Research Keeps Piling Up

February 12, 2013

(Guest Post by Collin Hitt)

Earlier this winter on JPGB I summarized the nine studies of charter schools (that I knew of) that use a rigorous random-assignment research design. I missed one. Harvard’s Roland Fryer and Stanford’s Vilsa Curto have released a study of the SEED charter school in inner-city Washington D.C.

SEED enrolls most poor, minority students. But it isn’t your typical charter school. While open to anyone who wins a seat through its enrollment lottery, SEED is a boarding school that’s quite expensive. It’s a ‘No Excuses’ school with strict discipline and high expectations. With control over students’ schooling, diets and leisure time, the school’s founders promised big results. From Fryer and Curto’s November 2012 report:

Our lottery estimates reveal that SEED is effective at increasing achievement among poor minority students. Students who enroll in SEED increase their achievement by 0.211 (standard deviations) in reading and 0.229 (standard deviations) in math, per year. Thus, SEED schools have the power to eliminate the racial achievement gap in four years.

Those are remarkable gains. But SEED comes at a steep cost.

At the SEED School in Washington, D.C., about $39,275 is spent per pupil per year, compared to $20,523 per student in District of Columbia Public Schools (DCPS).

Leave aside for a moment that DC Public Schools spend $20,523 and get abysmal results. Sending a kid to SEED nearly doubles that cost. That marginal increase is somewhat understandable, since the school is housing and feeding adolescent kids. Nevertheless, those costs don’t typically fall squarely on the DC school system. So is it worth it?

Our lottery estimates suggest that attending the SEED school for one year is associated with a 3.8 percent increase in earnings (Chetty et al., 2012), a 1.0 to 1.3 percent decrease in the probability of committing a property or violent crime (Levitt and Lochner, 2001), and a 4.4 to 6.6 percent decrease in the probability of having a health disability (Auld and Sidhu, 2005; Elias, 2005; Kaestner, 2009). If SEED affects educational attainment as dramatically as achievement, the implied returns are dramatic (e.g. Card, 1999; Philip Oreopoulos, 2007). The public benefits alone from converting a high school dropout to graduate are more than $250,000.

For obvious reasons, SEED’s results are not indicative of all charter schools. It’s a unique type of charter school. But SEED’s results, while incredibly strong, shouldn’t have caught anyone by complete surprise. Fryer and Curto’s study is the tenth random assignment study of charter schools. All of them find gains for urban, low-income students.

(For the pointer, I thank Jon Mills)

3 Comments | charter schools | Permalink
Posted by Jay P. Greene

Pass the Popcorn: Justice League for Dummies

February 10, 2013

20130210-150410.jpg

(Guest post by Greg Forster)

Hollywood is so dumb, they’re actually having trouble making a Justice League movie. Over on Hang Together, I am not impressed:

Look, you have here a team consisting of:

1) A virtuous hero raised by decent ordinary folk on a farm in Midwest corn country;

2) A self-made billionaire genius whose parents were slaughtered in front of him in a big east coast city;

3) A beautiful, fascinating noblewoman from an advanced but bizarre civilization who doesn’t believe in our ways but is stuck here and is trying her best to make our home hers; and

4) A couple other less important characters (choose any two from dozens of DC universe possibilities).

In other words, you have:

1) The moral backbone of America;

2) The cosmopolitan entrepreneurial genius of America;

3) The exotic immigrant from aristocratic Europe; and

4) Comic relief.

If you can’t make that movie, get out of the storytelling business.

Read the rest here, including my suggested opening scenes for the film.

Leave a Comment » | Pass the Popcorn | Tagged: , | Permalink
Posted by Greg Forster

Hogs Triumphant!

February 6, 2013

In a remarkably exciting upset, The Arkansas Razorbacks crushed the #2 Florida Gators last night in basketball at the Bud Walton Arena (“The Basketball Palace of Mid-America,” as the announcer likes to say).  My experience last night re-affirmed my confidence in the theory that school sporting events increase social capital, as I argued in yesterday’s post.  When the Spirit Squad forms their pyramid with the backdrop of a giant Arkansas flag, as pictured above, I have to admit that I get a little misty-eyed.  Sports make me feel more connected to Arkansas and my university, as I’m sure they do for others.

We were even graced with an appearance last night by Bubba Hog, whose dance enhances social capital with a good belly laugh in addition to the flag- pyramid’s tear to the eye.

Lastly, everyone should keep their eye on Michael Qualls, a freshman who jumped so high last night that I believe his head bumped the hanging scoreboard.  Here’s a highlight reel for Qualls from earlier games:

2 Comments | college sports | Permalink
Posted by Jay P. Greene

« Previous Entries
Next Entries »