Hitt, McShane and Wolf Meta-Analysis leads to a call for humility

March 19, 2018

(Guest Post by Matthew Ladner)

My favorite quote from Hitt, McShane and Wolf’s new study:

Even with these caveats in mind, the policy implications from this analysis are clear. The most obvious implication is that policymakers need to be much more humble in what they believe that test scores tell them about the performance of schools of choice. Test scores
are not giving us the whole picture. Insofar as test scores are used to make determinations in “portfolio” governance structures or are used to close (or expand) schools, policymakers might be making errors. This is not to say that test scores should be wholly discarded.
Rather, test scores should be put in context and should not automatically occupy a privileged place over parental demand and satisfaction as short-term measures of school choice success or failure.

P.S. Letting parents take the lead on which schools expand and/or close can work out fine on the types of tests schools have almost no ability/incentive to game:

The implications of this meta-analysis of the research literature could stretch far beyond the choice sector in time. If test scores continue to show a weak and inconsistent relationship with long-term outcomes, broad rethinking will be required. Let’s see what happens next.


Good reads

February 3, 2017

(Guest Post by Matthew Ladner)

The WSJ detects the Stockholm Syndrome of the six-inch Dark Lord of Nightmares MA Charter School Association and Eli Broad.

Derrell Bradford earns a BOOOOOOM! by explaining to the charter school bear that the hunter has plenty of bullets left for them.

So does Max Eden by exposing the NYT phony “analysis” of Detroit charter data.

Mike McShane joins the fun by explaining how our notions of accountability need an update.

Rick Hess cautions choiceniks to be careful what we wish for from the feds.


Going bold in Missouri with Education Savings Accounts

July 19, 2016

(Guest Post by Martin F. Lueken)

Last year, Missouri was one of 18 states that introduced legislation to create an education savings account (ESA) program for families. While it didn’t ultimately become law, it’s stoked the conversation about educational choice in the state and how we can empower families to find schooling options that work for their kids.

Under an ESA program, state officials deposit money into an account for education expenses for children who sign up for the plan. Parents can spend the money on a host of education expenses ranging from books to special needs services, online education, tutoring, SAT and ACT preparation or private school tuition. Parents can also roll over unused funds and use them in the future to pay for college tuition.

Currently, there are five K-12 ESA programs operating in five states – Arizona, Florida, Mississippi, Nevada, and Tennessee.

ESAs are a new and promising innovation with lots of potential because they move beyond just giving parents a say in what school their children attend. ESAs empower parents to tailor an educational experience that they want for their own children.

In essence, it expands on what Nobel Laureate and economist Milton Friedman’s vision of providing parents with freedom to choose the school that best suits their children’s needs. Going a step further, ESAs allow parents to unbundle educational goods and services and choose the ones that best meet their needs. School choice is getting an upgrade.

Critics of ESAs and other school choice efforts like to allege that the programs will “siphon” resources from public schools or harm students in some way. Fortunately, school choice has been around long enough to have produced a large body of research to learn from.

Researcher Greg Forster, for instance, systematically reviewed 100 empirical studies. His findings: school choice affects all of these areas mentioned above in a positive way. Students who choose score higher in reading and math, are more likely to graduate and are more likely to succeed in college. They also are more likely to learn civic values. Moreover, increased competition from school choice makes students remaining in public schools better off. When students choose, schools also tend to become more integrated. And not a single study found that school choice cost taxpayers any money.

Although greater educational freedom for Missouri families would be reason enough for many to adopt a program, some, including taxpayers and legislators, want to know how an ESA program would affect the state’s bottom line – a legitimate concern. A paper I recently co-authored with Mike McShane, Director of Education Policy at the Show-Me Institute, estimated the fiscal impact of a broad-eligibility ESA program on Missouri taxpayers and public school districts. This program would be funded by tax credits for private donations, in which nearly all Missouri K-12 age children (88 percent) would be eligible. We considered a program that is capped at $50 million in its first year, which is a drop in Missouri’s $5.7 billion K-12 education budget’s bucket.

Using a variety of circumstances to make our estimates, we found that state government and local school districts combined would save between $8 million and $58 million per year under an ESA program. The school districts alone would save $21 million to $40 million per year. The state – which is footing the bill by issuing tax credits – could save up to $18 million annually.

What does this mean? For starters, public school districts would have more resources for each student who remains in public school (as well as other tangential benefits such as smaller class sizes and better matches between Missouri students and schools).

Overall, however, Missourians and their children would have little to worry about and a whole lot to gain. The Show-Me State has tried many things to improve their schools, especially in the areas that struggle the most, with little success. It’s time to go bold, and try something that’s already a demonstrated success. It’s time for Missouri to create an education system fit for the future.

Update: rephrased for clarity

Martin F. Lueken is the Director of Fiscal Policy and Analysis at the Friedman Foundation for Educational Choice.


Tell me, friend, when did a bipartisan super-majority abandon reason for madness?

July 7, 2015

(Guest Post by Matthew Ladner)

So welcome back from your Independence Day holiday break, glad you are still ready to get your wonk on, so let’s talk a little academic transparency. The forces of light and darkness are on the march in this Middle Earth.

A few months ago I had the chance to meet with Bill Jackson from Greatschools here in Phoenix. At that meeting Bill told me something I’ve been trying to wrap my head around ever since- that more than half of parents visited Greatschools last year.

My instant reaction was to pick my jaw off the table and then mumble incoherently about probably far less than that percentage of parents had visited the Arizona Department of Education website ever, and of those that did only some unknown fraction did so to learn information about particular schools.

So then I decided to compare the Great Schools rating system with that of the state of Arizona on schools in my neighborhood. I started with Arcadia High. Arcadia High is a large high school south of Camelback Mountain. You may have heard of a prominent former student by the name of Spielberg- local legend has it that the architecture of the school inspired the space ship design in Close Encounters. The NCES put their FRL % at about half the statewide average at 25% back in 2010.

Great Schools gives Arcadia High a 7 out of 10, while easy grading Arizona gives it an “A” grade. So Greatschools essentially gave the school a C-minus, while the state of Arizona gave it an A. Which one is more accurate? Well one can never be certain (the same school can work very differently for different kids) but the Arizona Board of Regents tracked the academic progress of the entire public school Class of 2006 and found that 39% of the Arcadia High cohort finished a four-year degree within a six-year window.

Hmm, if 39% = “A” then what does a “B” look like? Well Shadow Mountain High School, near where I used to live a few years ago, got a “B” from the state, but saw 29.7% of their Class of 2006 get a BA in six years. The more discerning Greatschools ranking system gave the school a 6 out of 10 rating, which one can reasonably translate to “D-minus” and which seems to fit much more comfortably with the school’s higher education outcomes.

I could go on beating this horse into horseburger, but it misses the point. The (main) point is not that Arizona’s grading system needs a lot of work- although it clearly does- but rather that Greatschools provides an invaluable public service by providing a more rigorous ranking system. In addition, Greatschools also provides a source for student, parent and teacher reviews of schools. Mike McShane cited research showing that parents rate this reviews highly in their decision-making (see page three). This makes it all the better that Greatschools gets more traffic than the state website- it has more of the information they desire. It’s worth noting that the state of Arizona recently took a couple of year time out on school grades as they transition to a new state test. There is precedent for this, as the Arizona Department of Education just stopped publishing school report cards for several years within the last decade. Fortunately Bill Jackson and company have no need for timeouts, and parents are frequenting his website anyway.

Greatschools is performing a very valuable service in a fashion clearly superior to that provided by my state government-may God and His Angels bless Bill Jackson and all of his descendants. Greatschools gives highly valuable information to parents for free in helping them make difficult education decisions here in Arizona, and elsewhere. Greatschools however must make use of public data, and therein lies a growing threat to this invaluable parental choice resource.

It is with great sadness that news has reached my ears that an Arizona Congressman plans to offer an amendment to the ESEA reauthorization bill that would fundamentally undermine Greatschool’s ability to provide comparable data between schools for consideration by parents. The amendment would create a parental opt-out of testing. This would create what seems like a fatal blow to data comparability between schools, and a rather powerful perverse incentive for schools to nudge their weaker students to “opt-out.” Parents and researchers would have little to no way of knowing the rates of strategic “opting out” and thus compatibility between schools would be utterly compromised.

In the midst of anti-testing hysteria being fanned by the unions (the NEA for instance has model opt-out legislation) and others I fear this amendment will in fact pass. I had a front row seat to watch Texas dispatch a 30+ year bipartisan consensus in favor of testing and transparency in a distressingly casual fashion. If it can happen in Austin, it can happen anywhere.

Yes you can argue the federal government should have no role at all in financing schools and/or bossing them around. So why should you care? Notice however that the option of getting the feds out of the schooling business will not be seriously considered anywhere in the process. The federal government will still be sending money, will still be bossing schools around, but just would not get data reliable enough for Greatschools to even pull an Arizona-style rescue manoeuvre, or to consider student learning gains in making more rational retention and tenure decisions. In other words, federal policy will have been entirely captured by adult interests- funding will continue to flow, but transparency will suffer a train wreck.

In fact, if Congress were to pass a law with a parental opt-out, it doesn’t take a very active imagination to foresee a veto by President Obama. If you lead with your chin you should expect your opponent to break your jaw. A veto would allow Secretary Duncan to complete the last two years of rule by lawless administrative fiat in technocratic peace. Some of the latest waivers, by the way, have already stretched into the term of the next administration. They’ve already done waivers for states and districts- why not have the Department of Education hire some seasonal workers and offer waivers to individual schools if they are willing to pant, beg and roll over as commanded? Congress may lack the seriousness to actually put a stop to it.

Jay will argue that reformers brought this down on themselves with naive overreach. If the House votes to kill academic transparency it should indeed be a cause for education reformers to stare long and hard in the mirror. They won’t however be alone. The broad bipartisan consensus that created some level of academic transparency in return for federal funding and bossiness, warts and all, had a better idea than our contemporaries who would provide federal funding and bossiness in return for approximately nothing.

Who can stand against the union of the two towers, when far-left meets far-right at far-gone? Any chance Bill Jackson can take a long hike and throw the One Ring into the lava of Mount Doom?

 

 

 

 

 


On the Beautiful Messiness of Freedom

July 3, 2015

(Guest Post by Jason Bedrick)

As Independence Day approaches, Americans are preparing to celebrate our freedom. This year, I will be celebrating the many significant advances in educational freedom.

Freedom is messy. An education system in which providers are free to experiment means a system that is open to disruption. But what exactly does that mean? Here are just a few examples of how we might rethink key components of our education system—including student advancement, student assessment, and learning environments—from our friends at the Clayton Christensen Institute:

Rethinking Student Advancement

Since the late 19th century, most schoolchildren have advanced according to credit hours (or “Carnegie units”). All students of the same age were expected to advance at the same pace across all subjects. Unfortunately, that meant some students would struggle just to keep up while more advanced students were bored. A better way, argues Michael Horn, would have students advance according to skill, not age:

Reengineering our education system in this way would also allow us to shift from focusing on how many years of schooling a student has—faulty measures that focus on time but not learning—to measures that allow us to see what students have mastered in terms of their knowledge, skills, and dispositions.

“Do. Or do not. There are no credit hours.”

Providing every child with an individualized lesson plan sounds like a logistical nightmare, but Horn believes the blended-learning model could make it feasible:

Blended learning can support competency-based learning by providing the tools to bring it to scale. As online learning improves, schools will be able to rely on it to deliver consistently high-quality learning adapted to each student. That will free schools to focus on fulfilling other functions critical to students’ life success. Digital innovations are rapidly reshaping the world around us, but we can harness those same innovations to prepare our nation’s students to meet the opportunities and challenges ahead.

Indeed, as Horn notes, New Hampshire has already begun moving toward competency-based education.

Rethinking Student Assessment

In the traditional classroom, teachers are tasked with both instruction and assessment. However, as Thomas Arnett explains, these two roles create tensions that can make the teacher’s job very difficult:

By way of comparison, the arrangement of having teachers act as both instructors and assessors is akin to asking the judges in a gymnastics competition to also coach the gymnasts leading up to the event, or asking a restaurant manager to conduct official food handling inspections at his restaurant. In many domains of life, we recognize that we create problematic conflicts of interest if we ask people who produce or perform to also provide the ratings of their outputs for external audiences. Yet, in education it is the norm to ask teachers to both coach their students and rate their students’ academic achievements.

One facet of this problem stems from the fact that traditional grades are highly subjective. Teachers’ assessments and grading systems are not developed using rigorous psychometric techniques to ensure validity and reliability. Instead, we ask teachers to develop their own grading systems based on their professional judgment and interpretation of learning standards and school policy. Translating standards into a rigorous and fair grading system is sticky business.

Arnett also highlights the conflict of interest teachers have because they “need to develop good relationships with their students’ and their students’ families”:

If their grading system is “easy,” few students and parents are likely to complain. On the other hand, if their grading system is “hard,” they can end up with a flood of phone calls and meeting requests from upset parents. This puts pressure on teachers to go easy on their students in order to keep everyone happy. But teachers who hand out “easy As” are not doing an adequate job preparing their students for future education and for life beyond school.

Overcoming these obstacles is not impossible—Arnett readily concedes that we “all know that the best teachers are highly skilled at navigating these dilemmas”—but he argues that it’s best not to put teachers in such dilemmas in the first place. One way to separate instruction and assessment, he proposes, is through online assessments. Compared to having teachers develop and grade their own tests, online assessments—done right—would be less expensive, provide crucial feedback to teachers and students more quickly, and would have greater validity and reliability. Moreover, they would free teachers to focus on instruction and shift the teacher from the role of referee to the role of coach.

Rethinking the Classroom

If schools are going to operate in new ways, it follows that we may have to rethink the physical space in which learning takes place. The traditional classroom was designed for the Industrial Era and has served us relatively well since then, but it assumes a certain relationship between teachers and students—as well as school operations generally—that may not apply in some modern learning environments.

In a fascinating interview with Michael Horn, architect Larry Kearns explains how he designed a building to meet the needs of a blended-learning school in Chicago:

In a traditional school, since learning is monopolized by large-group direct instruction, all you need are cellular classrooms, with rows of desks focused on a single instructor. Since this “habitat” for learning is so culturally ingrained, it often goes unquestioned. …

Since blended-learning schools leverage multiple modes of learning, their spatial needs differ. At a minimum, they need spaces designed for different types of personalized learning, which can occur individually through digital media or in small interactive groups. The small-group learning can be peer-to-peer or teacher-led. Ideally, spaces for all of these modes of learning can be located in the same physical space, interlocked to minimize disturbances between them.

This combination of learning spaces is inherently decentralized since it focuses on the students. The teacher’s desk, if there is one, is pushed to the margins. Consequently, blended-learning “habitats” look nothing like their predecessors. …

Scheduling activities in a blended-learning school is more challenging than a traditional school. You cannot rely on a traditional bell schedule dictating movement between classrooms that focus on a single discipline. In a blended-learning environment, students are continually rotating between activities in a single space as they engage with a myriad of topics. It is much less passive than a traditional classroom. Consequently, programming a blended-learning school is a four-dimensional exercise where time and space must be tightly integrated.

Image of blended-learning classroom from edSurge.

The Road Ahead

The ideas presented above are just some examples of how our education system might change in the coming years, particularly with the advent of ESAs. None of the above proposals are necessarily the best way of doing things. Some might prove ineffective. It’s likely that some reforms will work well in some circumstances but not in others—or for some students, but not others. The best way to sort the wheat from the chaff and to match individual students to what works best for them is through a decentralized process of experimentation, evaluation, and evolution. Policymakers must resist the urge to impose regulations that would undermine this process. As AEI’s Michael McShane wrote recently:

School choice programs simply establish the conditions for a market to emerge. That market relies on both demand- and supply-side responses. If parents cannot access the schools that they want their children to attend, they won’t support the program. If new schools or new providers don’t enter the market, either due to regulation or because the funding mechanism doesn’t offer them what they want, there won’t be enough seats. The better advocates understand this complex dance of supply and demand, the better they can design programs that will meet the desperate needs of so many American families. [Emphasis added.]

Friedrich Hayek wrote in The Fatal Conceit that the “curious task of economics is to demonstrate to men how little they really know about what they imagine they can design.” Indeed, the most well-intentioned of policies can produce results that are the opposite of what was intended. We need look no further than Wisconsin and Louisiana to see how policies intended to protect the poor from the uncertainties of the market ended up depriving the poor of its benefits.

If Hayek isn’t enough, perhaps the Book of Proverbs will do: “When pride comes, disgrace follows; but with humility comes wisdom.”


On the move with our man McShane

September 25, 2014

(Guest Post by Matthew Ladner)

See Mike go…see Mike dodge infographics!

 


Hess and McShane: Oppose CC if You Want but Please Grow Up

April 14, 2014

(Guest Post by Matthew Ladner)

Heh, what they said.  Money quote:

Common Core critics must keep in mind that policy debates are won by proposing better solutions. The Core standards were adopted with a big federal boost and little public debate, but adopted they were. Teachers and school leaders have been implementing the standards since 2010, and opponents can’t wish this away any more than Obamacare critics can wish away the new landscape produced by the Affordable Care Act.

 


The Disaggregation Era of K-12

February 10, 2014

Pay attention 007, and do try to keep up!

(Guest Post by Matthew Ladner)

Mike McShane hosted an event last week at the American Enterprise Institute, and I had the opportunity to serve on a panel with Mike, Andy Smarick and Kara Kerwin.  During the discussion, Andy confessed that what he found the “disaggregation” of K-12 unsettling. This came up in the context of a discussion of Arizona’s ESA program and students like Jordan Visser:

“How do you assign a teacher of record?” I recall Andy asking.  For Jordan, such a question is already antiquated. Should his tutor be classified as the teacher of record? Or the physical therapists? Mr. or Mrs. Visser?  What if Jordan is taking a MOOC from Stanford is a few years? Should the state of Arizona attempt to hold Stanford “accountable” for what Jordan learns?

Personally I choose “none of the above.”

The trend towards disaggregation in K-12 predates Arizona’s still tiny ESA program.  The ESA program can in fact simply be viewed as the best vehicle for managing a customization trend as a quasi-market mechanism that gets us as close as possible to realizing the benefits of markets while preserving the public funding of K-12. The disaggregation trend however has been moving out into the bloodstream for decades. Consider the following program data from Florida:

Florida disagregation

This is a snapshot of traditional “school choice as you knew it at the end of the 20th Century.”  Most but not all of these choices are mutually exclusive such that they are something any one student does to the exclusion of others. You don’t expect to find many students for instance enrolled in a private school full-time and doing full-time virtual instruction, for instance. Most of these options are either/or propositions you are either sitting in this type of seat, or that type of seat. Major avenues of part-time education, such as dual college enrollment and virtual education, are not included, so we are just getting warmed up.

Let’s take virtual education on next:

FLVS Credits

The Florida Virtual School is not the only supplier of accredited virtual courses in Florida, so the 148,000 or so courses they provided in 2011-12 underestimates the strength of the trend. Nevertheless FLVS long ago begged the question: if a child takes an online Mandarin course from an approved online provider, just what, if anything, does this have to do with the results on the host schools’ accountability scores?

“I’ll take ‘Absolutely Nothing at All’ for a Thousand, Trebek!

Needless to say, FLVS found it necessary to develop alternative methods for measuring student achievement related directly to course content.  High-school students have been taking classes at community colleges for decades with what appears to be an entirely understandable disinterest in sorting through just how much responsibility, if any, the Community College holds for what happens on the high-school students minimal skills accountability exam.

So what happens when we mix dual enrollment with virtual education?

MOOC 1

Since we live in an age of wonders, we have over a thousand Massive Open Online Courses provided by some of the finest universities in the world available for free. Oh and the number of courses keeps growing. Did I mention that it has already been worked out for MOOC students to take third-party proctored final exams and receive college credit for them? Yes, right, that too.  Has anyone thought through the fact that the $89 cost for a third-party end of course exam may prove incredibly attractive for both families but also to schools who don’t enjoy having a portion of their budget sent off to an online provider?

Let’s not get ahead of ourselves Trebek! I’ll take ‘Months that begin with Oct’ for five hundred…

So, let us imagine a 15-year-old taking a Calculus class from, say, the Massachusetts Institute of Technology. He or she successfully completes a third-party end of course exam, he or she either is or in the near future will be eligible for college credit from a large number of universities around the world. Obviously provision for this student to receive high-school calculus credit will need to be made as well if we are to maintain any semblance of sanity.  Should authorities in Arizona disallow this because MIT’s Calculus course doesn’t precisely fit the state of Arizona’s state standards?

I’ll take “Seriously, you have got to be kidding me!” for a thousand Alex.

In short, the disaggregation genie is out of the bottle, and the trend looks set to accelerate in the coming years. As our system of education evolves it will be necessary to update our thinking regarding transparency and accountability: they are already out of date and will be increasingly so moving forward.  It would be absurd to require Jordan Visser to take the AIMS test. The AIMS has nearly played itself out for the 19th Century factory model school system in Jordan’s home state and has nothing to do with Jordan.  Regarding the ESA program, the public’s interest in transparency would be better served by collecting national norm reference exam data and having them analyzed by a qualified academic researcher.  Regarding the broader education system, Texas has already moved to replace minimal skills tests with subject specific end of course exams at the high school level. If a student takes a Physics class, shouldn’t we be curious as to whether or not they learned any “Physics”?

Creative destruction usually kills outdated ideas before outdated organizations. Our notions about how to provide transparency in a changing K-12 world have been running behind schedule.


Wolf and McShane in NRO

February 1, 2013

(Guest Post by Matthew Ladner)

A few years ago, a rookie quarterback named Michael Bishop was brought into a game to perform a last second desperation bomb before the end of the half. It was his first pass as an NFL player, and against the odds it resulted in a long touchdown. Commenting on the pass for ESPN, Chris Berman said something to the effect of “Completion rate-100%. Pass to touchdown ration also 100%. QB Rating = INFINITY!!!!!”

This came to mind when reading this great piece by Wolf and McShane in that had Congress redirected money from the bloated and ineffectual DCPS for the Opportunity Scholarship Program, then  the cost of the program would have been nothing and the benefits substantial, meaning ROI = INFINITY!!!”

!!!BOOOOOOOOOOOOOOOOOOOOOOOOOM!!

[Note: This is based on their peer reviewed article that is in the current issue of Education Finance and Policy.]


Double Standards on Special Ed Placements v. Vouchers

July 25, 2012

(Guest post by Greg Forster)

In today’s Examiner, AEI’s Michael McShane (an official JPGB super best friend) wants to know why none of the people fighting to kill the DC voucher program seem to have any objections to DC’s high rate of outplacement for special education students. Could it be because there are a lot more rich white special ed parents? McShane is here to chew gum and kick the cans of edu-hypocrites, and he’s all out of gum.

McShane doesn’t make the mistakes others have made in characterizing DC’s high rate of outplacement. Still, the stats are eye-popping, and will no doubt have many readers asking questions. McShane really doesn’t have the opportunity in a short piece like this to provide the necessary background. Thankfully, Jay wrote this a while back to bring people up to speed.


%d bloggers like this: