Jay P. Greene's Blog

Editor’s note: This post is the fifth in an ongoing discussion between Fordham’s Michael Petrilli and the University of Arkansas’s Jay Greene that seeks to answer this question: Are math and reading test results strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded—even if parental demand is inconsistent with test results? Prior entries can be found here, here, here, and here.

Mike, you say that we agree on the limitations of using test results for judging school quality, but I’m not sure how true that is. In order not to get too bogged down in the details of that question, I’ll try to keep this reply as brief as possible.

First, the evidence you’re citing actually supports the opposite of what you are arguing. You mention the Project Star study showing that test scores in kindergarten correlated with later life outcomes as proof that test scores are reliable indicators of school or program quality. But you don’t emphasize an important point: Whatever benefits students experienced in kindergarten that resulted in higher test scores, they did not cause higher test scores in later grades—even though they produced better later-life outcomes. As they put it, “The effects of class quality fade out on test scores in later grades, but gains in non-cognitive measures persist.” This is an example of the disconnect between test scores and life outcomes, which is exactly what I’ve been arguing. If we used test scores as a proxy for school or program quality, we would wrongly conclude that this program did not help, since the test score gains faded even though the benefits endured.

You also draw the wrong conclusion from the Deming, et al. article. The authors did find that test score gains for lower-scoring students in lower-performing schools resulted in higher earnings for those students. But lower-scoring students in higher-performing schools experienced an even larger decline in later-life earnings. These results highlight two things. First, narrowly focusing on raising test scores helps some low-scoring students. But it harms other low-scoring students such that:

This negative impact on earnings is larger, in absolute terms, than the positive earnings impact in schools at risk of being rated Low-Performing. However, there are fewer low-scoring students in high-scoring schools, so the overall effects on low-scoring students roughly cancel one another other out. Again, we find no impact of accountability pressure on higher-achieving students.

Having no net effect on low-scoring students, as well as having no effect of any kind on higher-scoring students, does not sound like a ringing endorsement of using accountability pressure to focus narrowly on test scores.

Second, the pattern of results in that paper supports my argument about the disconnect between test score gain and changes in later-life outcomes. Low-scoring students in higher-performing schools only experienced a decline of 0.4 percent in the probability of passing the tenth-grade math exam, but they exhibited a decline in annual earnings of $748 at age twenty-five. The low-scoring students in low-performing schools experienced a much larger 4.7 percent increase in the probability of passing the tenth-grade math exam, but they only exhibited an increase of $298 in earnings at age twenty-five. A negligible drop in test scores was associated with a large decline in earnings, while a large increase in test performance resulted in a more modest gain in earnings. See the disconnect?

You’re also mistaken in your belief that the evidence of this disconnect is confined to high schools. There is a fairly large literature on early education that shows little or no enduring test score gains from preschool but large benefits later in life. Again, gains in test scores do not appear to capture very well the quality of schools or programs. In addition, a series of studies by David Grissmer and colleagues found that early math and reading achievement tests are not even very good predictors of later test results relative to other types of skills and more general knowledge. They conclude: “Paradoxically, higher long-term achievement in math and reading may require reduced direct emphasis on math and reading and more time and stronger curricula outside math and reading.”

I could go on, but I promised to be brief. The overall point is that if tests were reliable indicators of school and program quality, they should consistently be predictive of later-life outcomes. As this brief review of research demonstrates, it is quite common for test score results not to be predictive of later-life outcomes. If even rigorous research fails to show a consistent relationship between test scores and later success, why would we think that regulators and policy makers with less rigorous approaches to test scores could use them to reliably identify school and program quality? Rather than relying on test results anyway and making potentially disastrous decisions to close schools or shutter programs on bad information, we should recognize that local actors—including parents—are in a better position to judge school quality. Their preferences deserve strong deference from more distant authorities.

Editor’s note: This post is the third in an ongoing discussion between Fordham’s Michael Petrilli and the University of Arkansas’s Jay Greene that seeks to answer this question: Are math and reading test results strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded—even if parental demand is inconsistent with test results? Prior entries can be found here [by Jay] and here [by Mike].

Jay P. Greene:

It’s always nice to find areas of agreement, but I want to be sure that we really do agree as much as you suggest, Mike. I emphasized that it should take “a lot more than ‘bad’ test scores” to justify overriding parental preferences. You say that you agree. But at the end, you add that we may have no choice but to rely primarily on test scores to close schools and shutter programs—or else “succumb to ‘analysis paralysis’ and do nothing.”

This is a false dichotomy. If all we have are unreliable test scores, we don’t have to make decisions based on them or “do nothing.” Instead, we could rely on local actors who have more contextual knowledge about school or program quality. So if the charter board, local authorizer, and parents think a school is doing a good job even if test scores look “bad,” we should defer to them. That isn’t doing nothing; it’s relying on those who know more than can be gleaned from test scores. And quite often, those more knowledgeable local actors will be parents, which is why I think we should show strong deference to parental preferences. We don’t have to substitute uninformed decisions by distant regulators for those of more knowledgeable parents.

The danger with your argument—that we may have no choice but to rely on test scores—is that it rationalizes ignorant actions by policy makers whose knowledge of school or program quality consists almost entirely of test score results. Even worse, they almost always rely on levels of test results rather than gains. It’s important to emphasize how crude and inaccurate decisions based on test scores typically are, rather than to imagine them to be as sophisticated as analyses found in leading journals (which are still quite imperfect). Using only levels of test scores, regulators and policy makers are quite content to label schools serving highly disadvantaged populations as “bad.” The perverse result is that those schools trying to serve needy populations, or those that do not focus narrowly on math and reading test scores, are likely to be punished or closed.

I’m glad we agree that “it should take a lot more than ‘bad’ test scores” to “close a school or shutter a program in the face of parental demand.” And I concur that we may never be able to develop other reliable indicators of school quality to be used by distant regulators or policy makers, including measures of character skills like grit and conscientiousness. But if we’re unable to develop strong measures of school quality that can be used remotely, the logical conclusion to be drawn is not that we ought to rely on them anyway. Instead, we should rely on the judgments of those closer to the situation, including parents, who have better information about school quality.

I accept that this will sometimes mean closing schools or programs that some parents nevertheless want. But I believe that few schools with long waiting lists will also be poorly graded by local actors using their broad contextual knowledge. Of all charter schools closed by local authorizers or their own boards, the vast majority had financial problems—meaning that they generally suffered from a lack of parental demand. It will be a rarity for parental assessments of quality to be at odds with those of local authorizers making decisions based on a lot more than test scores.

So I hope that we really agree that math and reading test results are not strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded. This means that we should instead accept the judgments of those with much more information about school quality, and it will be extremely rare that these more informed assessments of quality will be at odds with parental preferences.

(Also posted at Flypaper)

(Guest Post by James Shuls)

Some things just go together–cookies and milk, surf and turf, the Three Stooges (Moe, Larry, and Curly, of course)–you get the picture.  At the same time, there are things that simply do not mix.  When it comes to public education, two fundamental ideals fit this bill–local control and equity.  Try as we might, we simply haven’t figured out, in our current public education system, how to deliver on both of these principles.

Recently, NPR launched an initiative focused on the latter, the “School Money” project.   Reporters for the news outlet asked the question, why do some schools have so much, while others have so little?  They summed it all up very succinctly:

“Two words: property tax”

The NPR reporters go on to say:

“The problem with a school-funding system that relies so heavily on local property taxes is straightforward: Property values vary a lot from neighborhood to neighborhood, district to district. And with them, tax revenues.”

There is just one problem with this answer–it’s wrong.

Imagine if we replaced the local property tax with a different local tax; it could be a local sales tax, a local income tax, whatever. No matter what we tax, as long as we collect it locally, independent school districts will generate different amounts of money. It is not the property tax that causes inequities. It is our very system of public education itself; it is the local school district that causes inequities.

More specifically, it is the combination of local school districts and local support for public schools which causes differences in school spending.  Local school districts use the power of taxation to build new schools, to increase teacher pay, and to provide services for students. Interestingly, when given the opportunity, many local school districts tax themselves above and beyond any amount required by the state.

Here is the real kicker – the rich tend to tax themselves more.  In my home state of Missouri, for example, the 50 highest spending districts have a tax rate ceiling for operating funds of $4.582 per $100 of assessed valuation. The 50 lowest spending districts tax themselves at just $3.029 per $100 of assessed valuation. These districts not only have lower property values, on average, they also choose to tax themselves less.

In other words, when we allow property rich school districts to tax themselves more and property poor school districts to tax themselves less, we allow taxpayers to willfully contribute to the inequities we see between districts.

Many people, including the authors of the NPR reports, point to the gaps in spending and take a reductionist approach to this complex system. It’s the property tax! As much as I enjoy discussions about school finance, the issues here are much more fundamental —some people willingly choose to invest more in their children’s education than others.  These people appear to sort into communities with like-minded people.

It is easy to ask the question, “Why do some schools spend more than others?” It is much harder to answer the question, “Should we allow local taxpayers to have some say in how much they will support their local schools?” The former can be answered objectively, the latter cannot. If you say, “No,” you are saying you are uncomfortable with local control. If you answer, “Yes,” then you support some level of inequity.  In the current system, we can’t have both local control and equity.

If at the end of NPR’s “School Money” project we have only answered the easy questions, we will be no better off.  We will simply continue to wrestle with the same issues that we have grappled with for decades.  We will continually struggle to reconcile two incompatible ideals – local control and equity.

If, however, we start to think about how the fundamental organization of our school system—a patchwork of 14,000 school districts with geographic monopolies over the residents who live within them—contributes both to spending and educational inequities and think about how we can reform that, we might be able to move the discussion forward.

———————————————————

James V. Shuls, Ph.D., is an assistant professor of educational leadership and policy studies at the University of Missouri – St. Louis and a distinguished fellow of education policy at the Show-Me Institute.

(Guest Post by Michael J. Petrilli)

Editor’s note: This post is the second in an ongoing discussion between Fordham’s Michael Petrilli and the University of Arkansas’s Jay Greene that seeks to answer this question: Are math and reading test results strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded—even if parental demand is inconsistent with test results? The first entry [by Jay] can be found here.

The prompt for this forum promised that we would explore “areas of agreement and disagreement.” I’m pleased, Jay (and not altogether surprised), to see that we share a lot of common ground. Let me start with that, then save what I see as our major dispute (what we can learn from reading and math scores) for another post.

I’m thrilled that you dismissed the extreme position of some libertarians, who argue that society should never override the choices of parents. You write:

I…do not mean to suggest that policy makers should never close a school or shutter a program in the face of parental demand. I’m just arguing that it should take a lot more than “bad” test scores to do that.

I agree entirely, and on both counts. First let me explain why “we” should, on rare occasions, close a school that some parents prefer. And second, let me discuss what else beyond “bad” test scores we might consider when doing so.

You and others have heard me argue ad nauseam that because education is both a public and a private good, it’s only fair that both parties to the deal have a say in whether that good is, well, good enough. We both abhor a system whereby a district monopoly assigns children to schools and parents must accept whatever is handed to them. But the flip side is that we should also reject chronically low-performing schools—those that don’t prepare their young charges for success academically or otherwise—and deem them undeserving of taxpayer support. They aren’t fulfilling their public duty to help create a well-educated and self-sufficient citizenry, which is what taxpayers are giving them money to do.

Furthermore, there are real financial and political costs to letting bad schools—including schools of choice—fester. We see this in many cities of the industrial Midwest—Detroit, Cleveland, and Dayton come to mind—where too many schools are chasing too few students. Perhaps the marketplace forces of “creative destruction” will eventually take hold and the weakest schools will disappear, allowing the remaining ones sufficient enrollment to ensure their financial sustainability and a higher level of program quality. But that process is taking an awfully long time, particularly when we’re talking about disadvantaged children who have no time to waste. The charter sectors in these cities would be stronger—academically, financially, and politically—if authorizers stepped in to close the worst schools. But some libertarians see that as paternalistic government intrusion. I think they are misguided; I hope that you agree.

Now to your second point, that “it should take a lot more than ‘bad’ test scores” to “close a school or shutter a program in the face of parental demand.” Hear, hear! This is the genius of effective charter school authorizers that look at a school’s big picture as well as its scores. Fordham’s Dayton office strives hard (and with fair success) to be that kind of authorizer. We certainly look at test scores—especially individual student progress over time, a.k.a. “value added.” But we also examine lots of other indicators of school quality, operational efficiency, and financial sustainability. (See our current accountability framework in the appendix here.) And most importantly, we know our schools intimately. We attend their board meetings, conduct site visits frequently, and get up close and personal.

So when we consider the painful step of closing a school (which we’ve had to do a handful of times), we’re hardly just sitting in our offices “looking at spreadsheets of test scores.” The same goes for other leading authorizers nationwide.

Not that it’s easy to identify measures beyond reading and math scores that are valid and reliable indicators of school success. I share your enthusiasm for character education, non-cognitive skills, high school graduation rates, and long-term outcomes such as college completion and labor market earnings. And I’d love to see states maintain regular testing in history, geography, science, and more. Whenever we can use those scores, we absolutely do. But as the early debate around the Every Student Succeeds Act illustrates, measures of character and non-cognitive skills don’t appear ready for prime time, and they may never be appropriate for high-stakes decisions. High school graduation rates, meanwhile, are perhaps the phoniest numbers in education. Long-term outcomes are just that—long-term, making them both difficult to tie to a school (especially and elementary or middle school) and not very helpful for making decisions in the here and now. And there’s no political appetite for more testing; if anything, everyone wants less. (Let me know if you disagree with my analysis.)

So where does that leave us? As far as I can tell, facing a trade-off, which is the normal state of affairs in public policy. We can either use reading and math gains as imperfect indicators of effectiveness while working to build better measures—buttressed by school visits and the like—or we can succumb to “analysis paralysis” and do nothing.

I know which one I prefer. What about you?

(Also posted at Flypaper)

Here’s my first round in the debate with Mike Petrilli over whether test scores are reliable indicators of quality that can be used by regulators and policymakers to identify schools to be closed or expanded…

——————————————————-

The school choice tent is much bigger than it used to be. Politicians and policy wonks across the ideological spectrum have embraced the principle that parents should get to choose their children’s schools and local districts should not have a monopoly on school supply.

But within this big tent there are big arguments about the best way to promote school quality. Some want all schools to take the same tough tests and all low-performing schools (those that fail to show individual student growth over time) to be shut down (or, in a voucher system, to be kicked out of the program). Others want to let the market work to promote quality and resist policies that amount to second-guessing parents.

In the following debate, Jay Greene of the University of Arkansas’s Department of Education Reform and Mike Petrilli of the Thomas B. Fordham Institute explore areas of agreement and disagreement around this issue of school choice and school quality. In particular, they address the question: Are math and reading test results strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded—even if parental demand is inconsistent with test results?

To a very large degree, education reform initiatives hinge on the belief that short term changes in reading and math achievement test results are strong predictors of long term success for students. We use reading and math test scores to judge the quality of teachers, schools, and the full array of pedagogical, curricular, and policy interventions. Math and reading test scores are the yardstick by which education reform is measured. But how good of a yardstick is it?

Despite the centrality of test scores, there is surprisingly little rigorous research linking them to the long-term outcomes we actually care about. The study by researchers from Harvard and Columbia (Chetty, et al.) showing that teachers who increase test scores improve the later-life earnings of their students is a notable exception to the dearth of evidence on this key assumption of most reform initiatives. But that is one study, it has received some methodological criticism (although I think that has been addressed to most people’s satisfaction), and its results from low-stakes testing may not apply to the high-stakes purposes for which we would now like to use them. This seems like a very thin reed on which to rest the entire education reform movement.

In addition, we have a growing body of rigorous research showing a disconnect between improving test scores and improving later-life outcomes. I’ve written about this at greater length elsewhere (see here and here), but we have eight rigorous studies of school choice programs in which the long-term outcomes of those policies do not align with their short-term achievement test results. In four studies, charter school programs that produce impressive test score gains appear to yield no or little improvement in educational attainment. In three studies of private school choice and one charter school choice program, we observe large benefits in educational attainment and even earnings but little or no gains in short-term test score measures.

If policy analysts and the portfolio managers, regulators, and other policy makers they advise were to rely primarily on test scores when deciding which programs or schools to shutter and which to expand, they would make some horrible mistakes. Even if we ignore the fact that most portfolio managers, regulators, and other policy makers rely on the level of test scores (rather than gains) to gauge quality, math and reading achievement results are not particularly reliable indicators of whether teachers, schools, and programs are improving later-life outcomes for students.

What explains this disconnect between math and reading test score gains and later-life outcomes? First, achievement tests are only designed to capture a portion of what our education system hopes to accomplish. In particular, they are not designed to measure character or non-cognitive skills. A growing body of research is demonstrating that character skills like conscientiousness, perseverance, and grit are important predictors of later-life success (see this, for example). And more recent research by Matt Kraft, Kirabo Jackson, and Albert Cheng and Gema Zamarro (among others) shows that teachers, schools, and programs that increase character skills are not necessarily the same as those that increase achievement test results. There are important dimensions of teacher, school, and program quality that are not captured by achievement test results. Second, math and reading achievement tests are not designed to capture what we expect students to learn in other subjects, such as science, history, and art. Prioritizing math and reading at the expense of other subjects that may be important for students’ later-life success would undermine the predictive power of those math and reading results. Third, many schools are developing strategies for goosing math and reading test scores in ways that may not contribute to (and may even undermine) later-life success. The fact that math and reading achievement results are overly narrow and easily distorted makes them particularly poor indicators of quality and weak predictors of later-life outcomes.

I do not mean to suggest that math and reading test results provide us with no information or that we should do away with them. I’m simply arguing that these tests are much less reliable indicators of quality than most policy analysts, regulators, and policy makers imagine. We should be considerably more humble about claiming to know which teachers, schools, and programs are good or bad based on an examination of their test scores. If parents think that certain teachers, schools, and programs are good because there is a waiting list demanding them, we should be very cautious about declaring that they are mistaken based on an examination of test scores. Even poorly educated parents may have much more information about quality than analysts and regulators sitting in their offices looking at spreadsheets of test scores.

I also do not mean to suggest that policy makers should never close a school or shutter a program in the face of parental demand. I’m just arguing that it should take a lot more than “bad” test scores to do that. Yes, parents can and will make mistakes. But analysts, authorizers, regulators, and other policy makers also make mistakes, especially if they rely predominantly on test results that are, at best, weak predictors of later-life success. The bar should be high before we are convinced that the parents are mistaken rather than the regulators poorly guided by test scores. Besides, we should prefer letting parents make mistakes for their own children over distant bureaucrats making mistakes for hundreds or thousands of children while claiming to protect them.

(Also posted at Flypaper )

My former students and sometimes co-authors, Dan Bowen and Collin Hitt, have a superb piece in Phi Delta Kappan Journal on the history and evidence on sports in schools.  They demonstrate convincingly that sports improve academic outcomes and play an important and positive role in K-12 schools.

They also successfully rebut claims by Amanda Ripley and others that sports should be taken out of schools and instead be provided by club teams, as is common in Europe.  As it turns out, the rigorous research contradicts the casual observations based on journalistic tourism.  In particular, Dan and Collin debunk three commonly made claims about sports in schools:

#1. Sports participation has no role in academic development; in fact, sports might undermine academics.

#2. Adopting European-style sports club programs would enable adolescents to participate in sports while eliminating any negative influences that school-sponsored athletics have on academics.

#3. Eliminating school-sponsored sports will increase student participation in other extracurricular activities.

Be sure to check out their piece in the Kappan.