The Year That Was

December 15, 2008

It’s getting to be that time when people make lists of good and bad things that happened during the preceding year.  Here’s mine from an interview with Michael F. Shaughnessy of EducationNews.org:

What were the 5 most important developments during 2008 that contributed to reform of K-12 education?

 1)     Barack Obama strongly endorsed the idea that expanding choice and competition is an important part of improving public schools.  He limited his support to expanding choice and competition through the introduction of more charter schools, but the theory is not fundamentally different than doing the same with vouchers.  Whether Obama follows through on this campaign position or not, it is now clear that it is considered politically desirable among both Democrats and Republicans to support choice and competition.  Holdouts from this view, including the teacher unions on the left and curriculum-focused reformers on the right, are being increasingly marginalized.

2)     Sarah Palin, in her only major policy speech, pushed the idea of special education vouchers.  Like Obama embracing choice and competition, Palin embracing special ed vouchers is a symbol of the political attractiveness of the policy.  Special ed voucher programs already exist in Florida, Georgia, Ohio, Utah, and Arizona (pending the resolution of a court case).  I’d expect the idea to spread to several more states in the next four years regardless of Sarah Palin’s political prospects.

3)     Reform ideas, including choice, merit pay, curbing teacher tenure, and promoting alternative certification, are gaining mainstream acceptance in the Democratic Party largely thanks to the efforts of Democrats for Education Reform.  An important indication of this political shift was an event at the Democratic National Convention organized by the Democrats for Education Reform at which an audience of about 500 cheered speakers denouncing the teacher unions and embracing reform ideas. The Democratic supporters of reform largely (but not exclusively) consist of urban minority leaders, including Michelle Rhee, Joel Klein, Adrian Fenty, Cory Booker, Kevin Chavous, Al Sharpton, and Marion Barry.  Go ahead and make all the Sharpton and Barry jokes you like, but this (mostly) minority defection of urban Democrats from union orthodoxy is like a political earthquake that will have important implications for future reform politics.  And it’s true that some conservatives have begun backtracking on reform ideas, including Sol Stern, Diane Ravitch, and depending on the day of the week, Checker Finn and Mike Petrilli.  But if the reform movement has traded some conservatives for the new generation of minority Democratic leadership, I think we’ve come out ahead.

4)     We saw a string of new or expanded school choice programs in 2008.  Georgia adopted a universal tax-credit supported voucher program.  Louisiana adopted a voucher program for New Orleans as well as a personal tax deduction for private school tuition.  Florida expanded and decreased burdensome regulation on its tax-credit supported voucher program.  And Utah increased and secured a source of funding for its special ed voucher program.  For a movement declared dead more times than Generalissimo Francisco Franco, school choice continues to grow.

5)     I’ll take the privilege of the final development to brag about the launch of the new doctoral program in education policy in the Department of Education Reform at the University of Arkansas.  It may not have been among the 5 most important developments in the whole country, but it was a big development in my little world.  With the first cohort of students starting in the Fall of 2009 (supported by a pool of generous fellowships) and a collection of outstanding faculty, we have the potential to significantly increase the number of reform-oriented researchers in academia, think-tanks, and foundations.

What were the 5 most important developments during 2008 that hindered reform of K-12 education?

1)     The reform movement lost two great champions this year with the passing of John Brandl and J. Patrick Rooney.  Brandl, who had been the Democratic leader of Minnesota’s state senate and Dean of the University of Minnesota’s Hubert Humphrey School of Public Policy, contributed significantly to the argument that choice was not only efficient, but also enhanced opportunities for the disadvantaged.  He helped create the state’s pioneering charter school law and other choice programs.  Brandl also served as mentor to many of today’s leading choice researchers.  Rooney, who had always been active in the civil rights movement, personally sponsored scholarships for disadvantaged students to attend private schools.  His privately financed program became a model for publicly funded voucher and tax-credit supported scholarship programs.

2)     In 2008 we saw a number of “implementation” problems undermine otherwise promising reform initiatives.  For example, Georgia adoption a social promotion policy that required students to pass a test or follow a formal exemption policy to be promoted in certain grades.  My research with Marcus Winters on a similar policy in Florida suggested that it would improve student achievement.   But in several districts around Georgia more than 90% of students were promoted without passing the test and without following the formal exemption procedure.  They simply disregarded the law on a large scale with no consequences for any district or school employee.  Another promising idea undermined by implementation was Reading First. There is a lot of rigorous science to support a phonics-based reading approach, but getting public schools to do it well is a completely different matter.  Implementation also appears to have done-in a promising teacher mentoring program.  I could go on, but the point is that there is no shortage of clever reform practices out there.  The problem is that without addressing the lack of proper incentives in the public education system to improve, we regularly see these clever practices fall flat. We need incentive-based reforms along with reform of educational practices.

3)     Earlier this year an Arizona court struck down voucher programs for students with disabilities and students in foster care on the grounds that the state constitution forbids aid to private schools.  This month defenders of the program argued on appeal to the state Supreme Court that the program aids students, not schools.  And the state already sends disabled students to private schools when it is determined that the public schools are unable to provide adequate services.  That practice may also be in jeopardy, even though it is actually required by federal law (IDEA).  Who knows how this will all be resolved, since courts can adopt any interpretation they like, reasonable or unreasonable.  But court action has prevented these beneficial programs from operating and threatens to kill them.

4)     A Florida court struck down the ability of a state commission to approve charter schools.  If upheld by the (notorious) Florida Supreme Court, only school districts could approve charters and existing charters approved by the state commission may have to be closed.  Giving districts the exclusive power to grant charters essentially allows the districts to decide with whom they will have to compete.  It’s like giving McDonalds the exclusive power to approve the opening of all new restaurants.  The state Supreme Court used the same narrow interpretation of clauses in the state constitution to strike down the Opportunity Scholarship voucher program, so the prospects for a vibrant and competitive charter sector in Florida are not good.

5)     And finally the most disappointing development of 2008 is that we spent another half trillion dollars on public education without significantly altering the dysfunctional system that fails to teach a quarter of 8th grade students to read at a basic level or get them to graduate from high school.  Results for minority students are significantly worse.  The economic bailout may be a $700 billion enterprise, but the public school system spends almost that much each and every year.  Every year that we spend that money without fundamentally altering how we operate public education is another fortune wasted and another year lost for millions of students.

(Note: corrected spelling of Marion Barry’s name)

3 Comments | politics, social promotion, vouchers | Tagged: , , | Permalink
Posted by Jay P. Greene

Why JPGB Beats Edwize

December 11, 2008

 

  Edwize is a blog by Leo Casey that is sponsored by the United Federation of Teachers (UFT), the New York affiliate of the American Federation of Teachers.  The UFT has tens of millions of dollars at its disposal and thousands upon thousands of members.  Jay P. Greene’s Blog (JPGB) by contrast has a $25 registration fee for the domain name and a couple of laptops. 

Despite this huge disparity in resources, JPGB has a significantly larger audience than does Edwize.  According to Technorati JPGB has an authority rating of 95 while Edwize has a rating of 74.  An authority rating measures how many other blogs link to a given blog during the last 180 days, which is meant to capture how much influence a blog has in the blogoshpere.  In addition, each post on JPGB generates about 4 or 5 comments, on average, while posts on Edwize generate about 1 or 2 comments, on average.  Fewer comments suggest fewer readers and/or material on which people do not care to comment. 

None of these measures is perfect, but it is clear that JPGB beats Edwize.  Why?

The primary challenge for Edwize is that it has to tout teacher union views on education issues.  And those views are mostly junky.  So, Edwize suffers because it takes significantly more resources to interest people in crappy ideas than in sensible ones. 

In case you doubt that the unions have to push junky ideas, ask yourself whether it is sensible to have a system of education in which students are mostly assigned to schools based on where they live; where teachers are almost never fired, no matter how incompetent they are; where teachers are paid almost entirely based on how many years they’ve been around rather than on how well they do their job; where teachers are required to be certified even though there is little to no evidence that certification is associated with quality; and where all teachers are paid the same regardless of subject, even though we know that the skills required for expertise on certain subjects have much greater value in the market than other subjects.

The mental gymnastics required to sustain the union world view has a much greater “degree of difficulty” than the views that are regularly expressed on JPGB.  And the resources required to generate support for these union views are enormous.  You need millions of people financially benefiting from these policies to volunteer as campaign workers.  You need millions of dollars in union dues for campaign contributions.  You need a large team of paid staff in every state and in Washington, DC.  It takes an army and a fortune for the unions to hold their ground.

This not only helps explain why JPGB beats Edwize, but also why reformers are able to beat the unions in the policy arena.  It’s true that the unions win most of the time.  But given their enormous advantage in resources, it is amazing that the unions ever lose.  The reason that the unions lose as often as they do is that their policy positions are much more difficult to defend intellectually.

So, we should feel sorry for Leo Casey and his union comrades.  They may have a lot more money and a lot more people, but they constantly have to defend obviously dumb ideas.

(edited for clarity and to add photo)

3 Comments | blogging, politics, teacher pay | Tagged: , , , , | Permalink
Posted by Jay P. Greene

How Much Do We Know About K-12 Education?

December 10, 2008

Some folks in Utah showed how little we know about K-12 education.  And the errors are not random.  People believe we spend less than we really do, that teachers are paid a lot less than they really are, and that our students fare better against students in other countries than they really do.

3 Comments | money myth | Tagged: , | Permalink
Posted by Jay P. Greene

Is Chuck E. Cheese a Bad Influence?

December 9, 2008

The WSJ has an article today about the surprising number of incidents of disorderly conduct and fights at Chuck E. Cheese children restaurants.  The piece reports:

“Chuck E. Cheese’s bills itself as a place “where a kid can be a kid.” But to law-enforcement officials across the country, it has a more particular distinction: the scene of a surprising amount of disorderly conduct and battery among grown-ups.

‘The biggest problem is you have a bunch of adults acting like juveniles,’ says Town of  Brookfield Police Capt. Timothy Imler. ‘There’s a biker bar down the street, and we rarely get calls there.'”

Hey, Ryan!  We know you have special familiarity with the big Cheese.  Any theories to explain this?

2 Comments | random pop culture | Tagged: , | Permalink
Posted by Jay P. Greene

The TIMSS Rorschach Test

December 9, 2008

The Rorschach inkblot test is a psychology test that was used to assess personality and emotions.  The way in which people saw ambiguous images, like the one above, was supposed to say something about who they really were.

The same is true for the interpretations being applied to the results of the 2007 TIMSS (Trends in International Mathematics and Science Study) released today.

Over at Flypaper, Mike Petrilli interprets the gains the US has made in math but not science as suggesting that accountability testing is shifting resources toward math and away from science: “The lesson is that what gets tested gets taught. Under the No Child Left Behind act, and state accountability systems before that, elementary schools have been held accountable for boosting performance in math and reading. There is evidence that American elementary schools are spending less time teaching science, and this is showing up in the international testing data.”

And Mike interprets the relatively good results that Minnesota had (yes, MN took the test as if it were a country) as supporting rigorous standards: “There’s also good news out of Minnesota today, which has made dramatic gains since adopting new, more rigorous math standards.”

But also at Flypaper, Diane Ravtich offers different interpretations.  She sees the gains even in math results as “actually small, only four points.”  She also declines to credit NCLB for any of those gains, even as a perverse result of resource shifting away from science.  She notes that gains were at least as large in the US during the period prior to implementation of NCLB.  And on the topic of Minnesota she takes issue with Mikes explanation for success: “Minnesota showed dramatic gains on TIMSS not because of ‘new, more rigorous standards,’ but because of that state’s decision to implement a coherent grade-by-grade curriculum in mathematics.”  Umm, I would explain the difference but I got so bored trying to distinguish standards from curriculum that I dozed off for a bit.

Rather than focusing on the gains (or lack of gains) made by the US relative to itself in the past, Mark Schneider at Education Week focuses on the comparison between the US and other countries.  He notes that while the US looks relatively strong on the TIMSS, that is distorted by the large number of  “low-performing countries in the calculation of the international average [including Jordan, Romania, Morocco, and South Africa that] drives down that average, improving the relative performance of our students.”

He further notes that we fare worse on the PISA, which reports results from the 30 OECD countries who are our major trading partners and economic competitors: “We do better in TIMSS than we do on PISA, but this is a function of the countries that participate in each, and we should not let the relatively good TIMSS results lull us into a false sense of complacency. Even in the relatively easier playing field of TIMSS, we are lagging far too many countries in overall math performance and in the performance of our best students.”

And at Huffington Post Gerald Bracey was able to offer his reaction to the results last week, before they were released.  He wrote: “It might be good to keep a few things in mind when considering the data:

1. The Institute for Management Development rates the U. S. #1 in global competitiveness.

2. The World Economic Forum ranks the U. S. #1 in global competitiveness.

3. The U. S. has the most productive workforce in the world.

4. “The fact is that test-score comparisons tell us little about the quality of education in any country.” (Iris Rotberg, Education Week June 11, 2008).

5. ‘That the U. S., the world’s top economic performing country, was found to have schooling attainments that are only middling casts fundamental doubts on the value, and approach, of these surveys…'”

Bracey also said that our students could beat up the students in other countries with higher TIMSS scores.  (Actually, I made that last bit up.)

To summarize, Mike Petrilli sees evidence supporting his past concerns about the narrowing of the curriculum and the need for rigorous standards.  Diane Ravitch sees no evidence to alter her negative view of NCLB.  Mark Schneider, the former head of the National Center for Education Statistics, sees the need to review more testing.  And Gerald Bracey doesn’t even have to see the results to know that our education system is doing a great job.  And when I look at the inkblot I see a pudgy guy with a beard and male-patterned baldness laughing.

(edited for clarity)

7 Comments | research reports, school accountability | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

No Consumer Left Behind

December 8, 2008

The news is reporting today that the Republican (last time I checked) Bush Administration and Congressional Democrats are close to an agreement to bailout the auto industry.  The terms of the deal involve a $15 billion bridge loan and a federal oversight board.

It’s now becoming clear that rather than moving K-12 public education to look more like a competitive market, we are moving the competitive market to look more like K-12 public education.  To assist in those efforts (can’t nobody say JPGB never did nothing for the peoples), I would like to propose the No Consumer Left Behind act.  You don’t even need a new acronym!

Under the No Consumer Left Behind act we will provide a system of goals and assistance to ensure that all companies serve their consumers effectively.  No longer will we have stigmatizing terms like “bankruptcy.”  Instead, we will have “companies in need of improvement.” 

All companies will have to achieve profitability by 2014.  And they can define for themselves what “profitability” really means.  Each year they must make adequate yearly progress toward that goal.  If a company fails to make AYP they must offer their consumers the option to buy a different product that the same company sells.  After all we have to have choice!

Companies that are in need of improvement will also be provided with additional resources and professional development.  If we don’t help them, how else can they help their consumers?  We won’t call these additional resources a bailout or reward for failure.  Instead, we will call it technical assistance.  It’s just technical — like a technical foul.

We will also require all companies to employ “highly qualified” workers.  Highly qualified will generally be defined as whoever they currently employ.  Alternatively, highly qualified can be restricted to workers possessing union-approved credentials.

If a company fails to make AYP for several years, it will have to “restructure.”  But restructuring won’t be like the old bankruptcy restructuring, where you have to sell assets or layoff workers.  Instead, it can mean that you held some team-building workshops or hired a new CEO.  This new NCLB will be all about accountability.

And as you can tell from the title of the proposed law — we are doing all of this because we care about the consumer.  By focusing on companies in need of improvement, offering product choice within companies, providing additional resources to companies, requiring highly qualified workers, and redefining restructuring to mean essentially nothing, we are taking all of the steps necessary to help the companies — err, I meant consumer.

(HT: Bob Maranto)

5 Comments | economics | Tagged: , , , | Permalink
Posted by Jay P. Greene

The Humpty Dumpty Arkansas Courts

December 7, 2008

Courts claim to be in the business of interpreting the meaning of laws.  But the oddly limited or expansive meanings that are selectively applied to the words in those laws suggest that they are engaged in a completely different enterprise — namely, politics.  The idea that courts are just another political institution has long been held by political scientists, including myself.  We tend not to be hypnotized by the black robes, marble columns,  and Latin jargon into buying the notion that judges are some sort of special priesthood, immune from and indifferent to politics.  

Judges are just regular pols without the typical reelection pressures but also without the typical resources to advance their agenda.  Legislators have the power of the purse while executives have the power of the sword, but judges just have the power of their word.   The limitation on the power of judges is not the constraint of reelection, but the constraint of having to convince the other branches and the public to do what they say.  Cultivating the image of a disinterested priesthood enhances the power of judges to get others to do what they say.  But if the judges demand too much, they undermine their priestly image and erode their future power. 

Judges have been in a particularly strong position to get others to do what they say for the last five decades.  Early in the civil rights struggle our democratic institutions failed us, protecting obviously unjust and illiberal practices.  After initially siding with these illiberal forces (see Dred Scott or Plessy), the Courts detected a shift in elite opinion and joined forces with those elites to consolidate a new, progressive coalition.    The Courts could rightly take credit for having helped rescue us from the failure of our democratic institutions. 

Because they were instrumental in civil rights,  judges accumulated a considerable amount of political capital and popular goodwill.  And they’ve been spending that political capital ever since.   The civil rights era gave the Courts the role as guardians of our liberal virtue.  So, it’s hard to suggest that the Courts have overstepped their bounds, usurped the power of other branches, or arbitrarily interpreted the law without being accused of opposing the liberal virtue that Courts are supposed to protect.  Past critics of over-reach by Court included segregationists, so if you criticize judicial over-reach today on some other topic you must also be a segregationist.

This is especially true in Arkansas, where the memories of desegregation battles at Little Rock’s Central High School are particularly painful.  You cannot criticize Arkansas Courts for over-stepping their bounds or abusing their authority without being accused of being Orval Faubus — and there is no worse political insult in Arkansas.  The problem with immunity from legitimate criticism is that Arkansas Courts are especially unaccountable for judicial over-reach or arbitrariness. 

The most salient recent example of this is the action of the state Supreme Court in the Lake View school funding case.  The state constitution does  say that the state must “maintain a general, suitable and efficient system of free public schools.”  But who knew that general, suitable, and efficient meant that there was a specific dollar amount that had to be spent on every student in Arkansas?  And who knew that that amount had to increase by at least the rate of inflation every year?  I doubt that the authors of the Arkansas Constitution knew that general, suitable, and efficient meant all of these things, but the members of the Arkansas Supreme Court sure did.  And they figured out how much the legislature needed to spend per pupil and for school infrastructure by appointing Special Masters, who convened public meetings, received testimony from interested parties, and wrote a report summarizing their recommendations. 

Of course, there already exists a body for holding public meetings, receiving testimony from interested parties, and deciding upon the appropriate levels of public spending — it’s called the legislature.  With the appointment of Special Masters the Arkansas Supreme Court clearly usurped the legislature’s power.  And the Special Masters showed no restraint in determining spending priorities for the state — a power reserved by the Constitution for the legislature.  They declared: “[School districts] should have the means to meet the challenge if the State remains committed to the all-important practice of funding education first.”  Where in the state Constitution does it say that education has the first priority on resources? 

Some have argued that the responsibility to fund education first is implied by having education as the only policy area specifically mentioned in the Constitution.  I’m sorry to say that these people have never read the Arkansas Constitution.  It also specifically mentions a number of other policy areas, including the need for an agriculture, mining, and manufacturing policy.  Specifically, it says that the legislature must pass laws to “foster and aid the agricultural, mining and manufacturing interests of the State.”  If the Court and its Special Masters see the words general, suitable, and efficient as meaning that education must be supported as the first priority and at a specific, ever-increasing amount of spending, why haven’t they interpreted “foster and aid” to mean that the legislature must provide specific subsidies to agriculture, mining, and manufacturing?

Clearly we have a Humpty Dumpty Court.  The words mean what they want them to mean.  General, suitable, and efficient have expansive meanings if it suits their purposes while foster and aid mean essentially nothing.  Only judges, as the special class of high priests, possess the magical glasses that allow them to read between the lines of the Constitution to see that one phrase implies the moon while the other implies bupkis.

And now the Arkansas Supreme Court is at it again.  They are currently hearing arguments on whether a state law exempting state contracts in excess of $5 million from competitive bidding violates the state Constitition.  A plain reading of the text would suggest that it does.  The Constitution states: “All contracts for erecting or repairing public buildings or bridges in any county, or for materials therefor shall be given to the lowest responsible bidder, under such regulations as may be provided by law.” 

But Circuit Judge Jay Moody ruled that the state law did not violate the Constitution because he interpreted the provision as only applying to contracts from county governments — not contracts made by the state government and its agencies.  I’d like you to re-read the constitutional provision and ask yourself whether this is the most reasonable interpretation of the language.  Doesn’t the phrase “in any county” seem to describe the location of public buildings and bridges, emphasizing that the bidding requirement applies in all parts of the state, not the government agency engaging in the contracting?

We don’t know how the state Supreme Court will rule on the matter, but figuring that out requires a political, not a linguistic analysis.  They can and will interpret it in any way the see fit to advance their interests.  The words can mean just about anything they want them to mean.  “The question is which is to be master — that’s all.”

UPDATE:  The Arkansas Supreme Court interpreted the cluase as applying only to county contracts and upheld the state law.  The decision can be found here.

7 Comments | politics, school funding | Tagged: , , , | Permalink
Posted by Jay P. Greene

Violating the Denominator Law

December 2, 2008

Sean Corcoran, who is guest blogging for the blogger formerly known as Eduwonkette, may have to go to education research jail because he violated the Denominator Law today.  For those of you unfamiliar with the Denominator Law from my previous post on the topic (and ignorance of the law is no excuse) it is: “No one should be allowed to highlight numerators without also presenting denominators.  That is, it is often misleading to describe a big number without putting that number in perspective.”

So, Sean is all worried about private donations to public schools creating or exacerbating inequities in funding.  He references a report about California (and it had better be peer-reviewed or the blogger formerly known as Eduwonkette will throw a fit) that finds: “contributions to California school foundations rose from $123 million in 1992 to $238 million in 2001.”  He does helpfully add that $238 million only amounts to $40 per pupil.  But he doesn’t fully comply with the Denominator Law because he fails to point out that $238 million only represents .4% of the $52.2 billion in total public school revenue in California in 2001.

It’s not the average amount of private giving in California that really worries him.  What concerns him is that these donations are concentrated in wealthy areas: “Of course—as Brunner and Imazeki point out—these contributions are far from evenly distributed. Donations are strongly related to family income, and in some cases they are quite high, at more than $250 to $500 per student. (You can read about the $3.3 million education foundation in Santa Monica-Malibu Unified School District here).”

Here, your honor, is where he flagrantly breaks the Denominator Law.  He suggests that $250 to $300 per pupil, as illustrated by $3.3 million in private giving to public schools in Santa Monica, is “quite high.”  Without a denominator, it’s hard to judge how high $3.3 million in Santa Monica really is. 

Let me help.  According to the School Matters web site operated by Standard and Poor’s, Santa Monica has 12,191 students.  The private contributions Corcoran mentions amount to $271 per pupil — within his $250 to $300 range.  But total revenue for Santa Monica public schools amounted to $11,062 per pupil as of 2006.  Private contributions of $271 amount to only 2.4% of total revenue — not exactly “quite high.”

And this private giving hardly accounts for resource differences between Santa Monica and the average district in California.  According to School Matters the average district in CA had total revenue of $9,553 as of 2006, $1,509 less than in Santa Monica.  If Santa Monica received $271 in private donations compared to $40 for the average California district, the extra $231 could only account for about 15% of the extra resources Santa Monica possesses. 

If this is the worst case that folks can muster, it hardly seems like private giving is a significant contributor to resource inequities.  We only gain this appropriate perspective when we comply with the Denominator Law — so be sure to follow the law out there.

4 Comments | money myth | Tagged: , , , , | Permalink
Posted by Jay P. Greene

Replication, The True Test of Research Quality

December 2, 2008

When people can’t argue the facts, they argue peer review.  That’s been my experience when I’ve released non-peer reviewed reports.  Without peer review, folks wonder, how can we know whether to trust these results?

The reality is that even with peer review people still need to wonder whether to trust results.  Peer-review is by definition irresponsible — by which I mean that the reviewers have no responsibility.  By being anonymous, reviewers offer their opinions on the merit of research without any meaningful consequence to themselves.  Many reviewers do a laudable job, but there is nothing to stop them from using their reviews to advance findings they prefer and block findings they dislike regardless of the true merit of the work.  Peer-review is often little more than the anonymous committee vote of a panel composed of some mix of competitors and allies.  It is about as reliable as the Miss Congeniality vote at a beauty contest.  Do we really think she’s the nicest contestant or did the other contestants voting anonymously have ulterior motives for burying her with faint praise?

The true test of research quality is replication.  Science doesn’t determine the truth by having an anonymous committee vote on what is true.  Science identifies the truth by replicating past experiments, applying them to new situations, to see if the results continue to hold up. 

I’m pleased to say that several pieces of my work have been successfully replicated.  By successful replication I mean that the basic findings are upheld.  Replicators almost always make new and different choices about how to handle data or run an analysis.  The question is whether the same basic conclusion is found even when those different choices are made.

The evaluation I did with Paul Peterson and Jiangtao Du of the Milwaukee voucher experiment was successfully replicated by Cecilia Rouse.  The evaluation I did of the Charlotte voucher program was successfully replciated by Josh Cowen.  My study of of Florida’s A+ voucher and accountability program was successfully replicated three times — by Raj Chakrabarti; Rouse, et al; and West and Peterson.  And my graduation rate work has been successfully replicated by Rob Warren and Chris Swanson.

The interesting thing is that every one of my studies above was initially released without peer review.  And every one of them was attacked for being unreliable because they were not peer reviewed.  When they were all later published in peer reviewed journals (except the grad rate work) and successfully replicated I don’t remember ever hearing anyone retract their accusations of unreliability. 

(edited for typos)

4 Comments | research reports, school accountability, vouchers | Tagged: , | Permalink
Posted by Jay P. Greene

In Defense of the BCS

December 1, 2008

Barack Obama has his finger on the pulse of American public opinion.  So when the president-elect came out in support of an 8 team college football playoff to replace the current BCS-selected match-up of the top two teams, he was endorsing a view held by 97.4% of all football fans.  This stat comes from the same source that found that 73.8% of all statistics are made up on the spot.

I, however, am among the 2.6% that prefers the current BCS method.  Why? — because an 8 team playoff solves virtually none of the supposed injustices of a BCS-selected championship game and because playoffs create significant, new problems.

The main injustice that a playoff is supposed to prevent is the exclusion of worthy teams from competing in the post-season for the national championship.  The current system uses a formula combining coach and journalist rankings of teams with computer models of team performance given the difficulty of their schedules to identify the top two teams in the country.  Those two teams then play for the national championship. 

“But what about the third ranked team?” opponents of this system ask.  Shouldn’t they have a chance to compete for the championship also?  This concern for injustice is compounded by disputes over whether the top two teams identified by the BCS really are the two best teams.  People become particularly passionate about this if their team is the one ranked 3rd (or even 4th, 5th, etc…).  And the fact that computer models have a hand in selecting the top two teams only fuels the technophobe football fan rage.  The intensity of opposition to BCS ratings is almost always inversely related to a person’s ability to do algebra (or even compute simple sums).

Moving to an 8 team playoff doesn’t really solve this perceived injustice.  Instead of arguing over whether the 3rd ranked team was unjustly excluded from competing in the post-season for the national championship, we’ll just argue about whether the 9th ranked team was unjustly excluded.  You have to draw the line somewhere.

In addition, there has to be some method for selecting the 8 teams.  If you don’t like relying on computer models and polls, try to describe a system that would more accurately identify the best teams.  Some have suggested providing guaranteed spots to the winners of 6 of the most competitive conferences with two additional teams selected at-large.  But it’s not hard to imagine the injustices that would flow from such a system.  Who gets to pick the 6 conferences?  Why shouldn’t the 7th conference have a guaranteed spot?  What if there are two top-notch teams in a conference?  How will we select the two at-large teams?  The bar arguments will never end no matter how we select teams.

The virtue of the BCS method of ranking is that it combines multiple reasonable methods into a single rating.  It incorporates the subjective judgment of experts as well as the dispassionate computer assessment of team schedules.  Sure, the BCS, like any rating system, will be imperfect.  But its methodology is reasonable and the rules are clearly stated in advance.

The only question remaining is why only have 2 teams in the post-season instead of 4 or 8 (or 16 for that matter).  I’ve already argued that drawing the line anywhere is somewhat arbitrary and would produce disputes and claims of injustice.  But others might respond that it is better to have more teams included in the post-season than fewer. 

The problem with expanding the post-season to include more teams in the national championship race is that it would require more games to be played.  You cannot add games to college football without a price.  Other than among advocates of the ginormous financial bailout, everyone understands that there is no such thing as a free lunch.  Extra college games come at a cost.

If we simply add two more games to the post-season to have an 8 team playoff, we are requiring players to have longer seasons with greater opportunities for injuries.  Remember that college football players are uncompensated young students (and free tuition hardly qualifies as fair compensation given how much revenue they generate).  If we make them play longer seasons, they run a significantly higher risk of suffering debilitating injuries that could ruin any hopes for a professional football career and/or turn them into life-long cripples.  Barack Obama and 97.4% of all football fans may not care about exploiting unpaid college kids for our entertainment, but I think there have to be limits.

I suppose we could instead shorten the regular season by two games to avoid making players extend their season.  But if we do that we will reduce the information from the regular season for determining who deserves to be in the playoffs.  We’ll also deprive the vast majority of college football programs and their fans of two games and the revenue those games produce.  Again, there is no free lunch.

People wonder why college football is the only major sport without a playoff.  But college football is different from other sports.  Football is so brutal that it can only be played once a week and even then the probability of serious injury increases dramatically with each additional game.  We can expect the pros to play longer and run those risks because, well, their pros.  They are paid (although not nearly enough — but that is a story for another day), while college athletes are virtually unpaid (and that is an injustice that should also be corrected — but that is also a story for another day).  I’d rather have a bunch of bar arguments over whether the 3rd ranked team was unjustly excluded from the championship game than significantly increase the exploitation of college football players.

12 Comments | college sports | Tagged: , , | Permalink
Posted by Jay P. Greene

« Previous Entries
Next Entries »