Just About Everything is Endogenous

September 30, 2009

A common technique in analyses of education policies (and popularized in the book, Freakonomics) has suffered a setback recently.  The technique attempts to correct for endogeneity, which occurs when your dependent variable is causing one of your independent variables rather than simply the other way around.

It’s probably best to explain this with an example.  Let’s say you want to know how the number of police officers in a city affects the crime rate.  In this example the dependent variable is the crime rate and the independent variable is the number of police officers.  That is, you are trying to explain how the size of the police force causes crime rates to be high or low.

The trouble is that the causal arrow also goes in the other direction.  The crime rate affects the size of the police force because cities with a lot of crime may decide to hire a lot of police officers.  So, the number of police officers is endogenous to the crime rate.  

That endogeneity could produce some odd results if we didn’t do anything to correct it.  We might find that the number of police officers causes crime rates to be higher when it might really be the case that the size of the police force reduces crime but high crime rates cause larger police forces.

This kind of problem comes up quite often in econometric analyses in general and in particular in evaluations of education policies.  So, it was a great a thing that University of Chicago economist James Heckman developed a technique for unravelling these circular relationships and correcting for endogeneity bias.  Basically, the technique uses some exogenous variable to predict the independent variable without bias.

Again, it’s probably easiest to explain with an example.  If we can find something that predicts the number of police officers that has nothing to do with the crime rate, then we can come up with an unbiased estimated of the number of police officers.  We can then use that unbiased estimate of how many police officers there would be (independent of the crime rate) to predict the crime rate.  In theory the technique works great.  Heckman won the Nobel Prize in economics for developing it.

The tricky part is coming up with a truly exogenous instrument (something that predicts the independent variable but has no relationship with the dependent variable).  The only obviously exogenous instrument is chance itself.  An example of that kind of instrument can be found in analyses of the effect of using a voucher on the student achievement of students who actually attend a private school when the vouchers are awarded by lottery.  Those analyses use whether a student won the lottery or not to predict whether a student attended a private school and then used that unbiased estimate of whether a student attended a private school to predict the effect of private schooling on student achievement. 

Whether a student won the lottery is purely a matter of chance and so is completely unrelated to student achievement, but it is predictive of whether a student attends a private school.  It is a perfectly exogenous instrument.

The problem is that other than lotteries, it isn’t always clear that the instruments used are truly exogenous.  Even if we can’t think of how things may be related, they may well be.

A perfect example of this — and it is one that raises questions about how exogenous all instruments other than lotteries truly are — was recently described in the Wall Street Journal having to do with date of birth.  The date during the year when babies are born has long been thought to be essentially random and has been used as an exogenous instrument in a variety of important analyses, including a seminal paper in 1991 by Josh Angrsit and Alan Krueger on the effects of educational attainment on later life outcomes. 

Since states have compulsory education laws require that students stay in school until a certain age, babies born earlier in the year reach that age at a lower grade and can drop out having attained less education.  By comparing those born earlier in the year to those born later, which they believed should have nothing to do with later life outcomes, they were able to make claims about how staying in school longer affected income, etc…

But new work by Kasey Buckles and Daniel Hungerman at the University of Notre Dame suggests that the month and day of birth is not really exogenous to life outcomes.  As it turns out, babies born in January are more likely to be born to unwed, less educated, and low income mothers than babies born later in the year.  The difference is not huge, but it is significant.  And since this variable is not exogenous, perhaps some or all of the effect of attainment Angrist and Krueger observed is related to this relationship between date of birth and SES, not truly attributable to attainment.

And if birth order is not random when we all assumed it was, what other instruments in these analyses are also not truly exogenous but we just don’t know how yet?  It’s a potentially serious problem for these analyses.

6 Comments | methodology | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Schoolhouses, Courthouses, and Statehouses

August 9, 2009

The new book from Rick Hanushek and Alfred Lindseth, Schoolhouses, Courthouses, and Statehouses, is a remarkably comprehensive and accessible review of K-12 education reform strategies.  It’s a must-read for education policymakers, advocates, and students — at both the graduate and undergraduate levels.  Even experienced researchers will find this to be an essential reference, given its broad sweep and extensive citations.

The book basically makes four arguments.  First it establishes how important K-12 educational achievement really is to economic success and how far we are lagging our economic competitors in this area.  Second, it demonstrates the dominance and utter failure of input-oriented reform strategies, including across-the-board spending increases and class-size reductions.  Third, it describes how the court system has perpetuated failed input-reform strategies after having bought intellectually dishonest methods of calculating how much spending schools really need.  And fourth, it makes the case for reform strategies that involve “performance-based funding,” including merit pay, accountability systems, and choice.

None of these arguments is original to this book.  But to the extent that others have made these arguments, they have drawn heavily on Rick Hanushek’s research.  In this book you get to hear it directly from the source and you get to hear it all so persuasively and completely.

If I have any complaint about the book it is that they are too restrained in their criticisms of the methods by which adequate school spending has been determined and the “researchers” who have developed and profited from those methods.  These fraudulent analyses have justified court decisions ordering billions of dollars to be taken from taxpayers and blown ineffectively in schools.  And the quacks promoting these methods have made millions of dollars in consulting fees in the process.

Those methods include the “professional judgment approach,” which essentially consists of gathering a group of educators and asking them how much money they think they would need to provide an “adequate” education,  Naturally, they need flying saucers, ponies, and a laser tag arena to ensure an adequate education. 

Another method is the “evidence-based approach,” which selectively reads the research literature to identify what it claims are effective educational practices.  It then sums the cost of those practices while paying no attention to how many are really necessary for an adequate education or whether any of them are really cost-effective.

There is also the “successful schools approach,” which looks at how much money a typical successful school spends and calls for all schools to spend at least that much.  This of course ignores the fact that many successful schools spend less than the typical amount and are still successful.  One would have thought it impossible for them to be successful with less money than that deemed necessary to succeed. 

And lastly, there is the “cost-function approach.”  This approach takes the conventional finding that higher spending, controlling for other factors, has little to no relationship with student achievement, and then turns that finding on its head.  It does this by switching  the dependent variable from student achievement to cost.  The question then becomes: how much each unit of achievement contributes to school costs.  Switching the dependent variable does nothing to change the lack of relationship between spending and achievement.  If you hide behind enough statistical mumbo-jumbo you can hope that the courts won’t notice that there is still virtually no relationship between spending and achievement controlling for other factors.

The Hanushek and Lindseth book lays all of this out (see especially chapter 7), but they are remarkably restrained in denouncing these approaches and the people who cynically profit from them.  I don’t think we should be so restrained.  The promoters of this snake oil are often university professors with sterling national reputations.  They’ve cashed in those reputations to market obviously flawed methods.  We shouldn’t let them do this without paying a significant price in their reputation.

The University of Southern California’s Larry Picus, and the University of Wisconsin’s Allan Odden, are both past presidents of the well-respected American Education Finance Association.  They shouldn’t be able to sell the “evidence-based approach” to 5 states for somewhere around $3 million without people pointing and laughing when they show up at conferences.

I know that Rick Hanushek and Alfred Lindseth are too professional and scholarly to call these folks frauds, but I’m not sure what else one could honestly call them.  Rick comes close in his Education Next article on these school funding adequacy consultants, entitled, “The Confidence Men.”  But in this book,perhaps with the tempered emotions of his co-author,  he adopts a more restrained tone.  Perhaps this is all for the best because the book maintains the kind of scholarly temperament that strengthens its persuasiveness to those who would be more skeptical. 

This has been a great year for education reform books.  Schoolhouses, Courthouses, and Statehouses joins Terry Moe and John Chubb’s Liberating Learning, released earlier this summer, as members of the canon of essential education reform works.

3 Comments | methodology, school accountability, school funding | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Why Random Assignment is Important

July 2, 2009

Bill Evers has an excellent post over on his Ed Policy blog about how unreliable observational studies can be and how important it is to test claims with random-assignment research designs. 

Observational studies (sometimes called epidemiological or quasi-experimental studies) do not randomly assign subjects to treatment or control conditions or use a technique that approximates random-assignment (like regression discontinuity).  Instead they simply compare people who have self-selected or otherwise been assigned to receive a treatment to people who haven’t received that treatment, controlling statistically for observed differences between the two groups.  The problem is that unobserved factors may really be causing any differences between the two groups, not the treatment.  This is especially a problem when these unobserved factors are strongly related to whatever led to some people getting the treatment and others not. 

The solution to this problem is random assignment.  If subjects are assigned by lottery to receive a treatment or not, then the only difference between the two groups, on average, is whether they received the treatment.  The two groups should otherwise be identical because only chance distinguishes them.  Any differences between the two groups over time can be attributed to the treatment with high confidence.

If you don’t believe that research design makes a big difference, consider this table that Bill Evers provides on how much results change in the field of nutrition when random assignment (or clinical) studies are done to check on claims made by observational studies:

If we want to avoid the educational equivalent of quack medicine, we really need more random-assignment studies and we need to give the random-assignment studies we already have significantly greater weight when forming policy conclusions.

As I’ve written before, we have 10 random-assignment studies on the effects of vouchers on students who participate in those programs. Six of those ten studies show significant academic benefits for the average student receiving a vouchers and three studies show significant academic benefits for at least one major sub-group of students.  One study finds no significant effects.  

I believe that there are more random-assignment studies on vouchers than on any other educational policy and there are certainly more studies with positive results.  The depth of positive, rigorous studies on voucher participant effects is worth keeping in mind each time some new observational or (even descriptive) study comes out on school choice, including the most recent report from Florida.  Our opinion shouldn’t be based entirely on the latest study, especially if it lacks the rigorous design of several earlier studies.

3 Comments | methodology, vouchers | Tagged: , | Permalink
Posted by Jay P. Greene

The Professional Judgment Un-Dead

March 25, 2009

It’s time we drive a stake through the heart of “professional judgment” methodologies in education.  Unfortunately, the method has come back from the grave in the most recent Fordham report on regulating vouchers in which an expert panel was asked about the best regulatory framework for voucher programs.

The methodology was previously known for its use in school funding adequacy lawsuits.  In those cases a group of educators and experts was gathered to determine the amount of spending that is required to produce an adequate education.  Not surprisingly, their professional judgment was always that we need to spend billions and billions (use Carl Sagan voice) more than we spend now.  In the most famous use of the professional judgment method, an expert panel convinced the state courts to order the addition of $15 billion to the New York City school system — that’s an extra $15,000 per student.

And advocates for school construction have relied on professional judgment methodologies to argue that we need $127 billion in additional spending to get school facilities in adequate shape.  And who could forget the JPGB professional judgment study that determined that this blog needs a spaceship, pony, martinis, cigars, and junkets to Vegas to do an adequate job?

Of course, the main problem with the professional judgment method is that it more closely resembles a political rather than a scientific process.  Asking involved parties to recommend solutions may inspire haggling, coalition-building, and grandstanding, but it doesn’t produce truth.  If we really wanted to know the best regulatory framework, shouldn’t we empirically examine the relationship between regulation and outcomes that we desire? 

Rather than engage in the hard work of collecting or examining empirical evidence, it seems to be popular among beltway organizations to gather panels of experts and ask them what they think.  Even worse, the answers depend heavily on which experts are asked and what the questions are. 

For example, do high stakes pressure schools to sacrifice the learning of certain academic subjects to improve results in others with high stakes attached?  The Center for Education Policy employed a variant of the professional judgment method by surveying school district officials to ask them if this was happening.  They found that 62% of districts reported an increase in high-stakes subjects and 44% reported a decrease in other subjects, so CEP concluded that high-stakes was narrowing the curriculum.  But the GAO surveyed teachers and found that 90% reported that there had not been a change in time spent on the low stakes subject of art.  About 4% reported an increase in focus on art and 7% reported a decrease.  So the GAO, also employing the professional judgment method, gets a very different answer than CEP.  Obviously, which experts you ask and what you ask them make an enormous difference.

Besides, if we really wanted to know about whether high stakes narrow the curriculum, shouldn’t we try to measure the outcome directly rather than ask people what they think?  Marcus Winters and I did this by studying whether high stakes in Florida negatively impinged on achievement in the low-stakes subject of science.  We found no negative effect on science achievement from raising the stakes on math and reading.  Schools that were under pressure to improve math and reading results also improved their science results.

Even if you aren’t convinced by our study, it is clear that this is a better way to get at policy questions than by using the professional judgment method.  Stop organizing committees of selected “experts” and start analyzing actual outcomes.

5 Comments | methodology, school accountability, vouchers | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Looking Abroad for Hope

November 5, 2008

hope

HT despair.com. Looking for a Christmas idea to suit the new reality? Why not a despair.com gift certificate - ”For the person who has everything, but still isn’t happy.”

(Guest post by Greg Forster)

Looking around for something to give me hope this morning, I find the best place to turn (for today, at least) is outside the U.S. Specifically, I turn to the recently released study in Education Next by Martin West and Ludger Woessmann finding that around the world, private school enrollment is associated with improved educational outcomes in both public and private schools, as well as lower costs.

Well-informed education wonks will say, “duh.” A large body of empirical research has long since shown, consistently, that competition improves both public school and private school outcomes here in the U.S., while lowering costs. And the U.S. has long been far, far behind the rest of the world in its largely idiosyncratic, and entirely irrational, belief that there’s somthing magical about a government school monopoly.

And private school enrollment is an imperfect proxy for competition. It’s OK to use it when it’s the best you’ve got. I’ve overseen production of some studies at the Friedman Foundation that used it this way, and I wouldn’t have done that if I didn’t think the method were acceptable. However, that said, it should be remembered that some “private schools” are more private than others. In many countries, private school curricula are controlled – sometimes almost totally so – by government. And the barriers to entry for private schools that aren’t part of a government-favored “private” school system can be extraordinary.

That said, this is yet another piece of important evidence pointing to the value of competition in education, recently affirmed (in the context of charter schools, but still) by Barack Obama. Who I understand is about to resign his Senate seat – I guess all those scandals and embarrasing Chicago machine connections the MSM kept refusing to cover finally caught up with him.

Leave a Comment » | competitive effects, methodology, money myth, politics, research reports, vouchers | Tagged: , , , , | Permalink
Posted by Greg Forster

Educating Journalists about Education Science

July 16, 2008

(Guest post by Greg Forster)

Don’t worry, this post is definitely not a continuation of the recent big dustup about 1) whether it’s naughty for scholars to provide journalists with accurate information about their work; and 2) whether it’s naughty for anonymous bloggers to argue that scholars’ motives are relelvant to their credibility, but bloggers’ motives aren’t relevant to theirs (which reminds me of Pat Moynihan’s quip about the Supreme Court cases, since overturned, holding that government can’t subsidize private school books but can subsidize classroom equipment such as maps; Moynihan asked, “What about atlases?” – books of maps? What about scholars who are bloggers? Or bloggers who write about scholarly studies? Once you start legitimizing ad hominem arguments, where do you stop?).

But I would like to expand on a comment that Eduwonk made during said dustup, which deserves more attention and has significance well beyond the issues that were at stake in that squabble. The comment got lost in the exchange because it was somewhat tangential to the main points of contention.

He wrote:

Not infrequently newspapers get snookered on research and most consumers of this information lack the technical skills to evaluate much of the work for themselves.   As education research has become more quantitative — a good thing — it’s also become less accessible and there is, I’d argue, more an asymmetry to the information market out there than a fully functioning marketplace of ideas right now.  In terms of remedies there is no substitute for smart consumption of information and research, but we’re not there yet as a field.

We are living in the first golden age of education research, brought on by the advent of systematic data collection, which every other field of human endeavor began undertaking a long time ago but which education is only getting around to now because it has been shielded from pressure to improve thanks to its protected government monopoly. Given the explosion of new information that’s becoming available, educating journalists about quantitative research is a huge problem. Jay is right that there is a marketplace of ideas. There really can’t help but be one; the idea some people seem to have that we can forbid people who own information from spreading it around as much as they want is silly. But just because there’s a market doesn’t mean there’s a perfect market, and Eduwonk is right that markets require informed consumers to function well. The current state of methodological ignorance among journalists does hinder the market of ideas from functioning as well as it should. (I’ll bet Jay would agree.)

As it happens, the same subject came up this morning in a completely different context, as my co-workers and I struggled to figure out the best way to present the findings of an empirical study we’re coming out with so that journalists will be able to follow them. And I wasn’t there, but I hear this topic also came up at a bloggers’ panel at the recent conference of the Education Writers’ Association.

Here at the Friedman Foundation, this has been a topic of great importance to us for some time, since exposing the bad and even bogus research that’s used to justify the status quo is one of our perennial challenges. We took a stab at composing a journalist’s guide to research methods. It went over well when we first distributed it (at last year’s EWA, if memory serves). But it’s necessarily very basic stuff.

Eduwonk is also right about journalists having been snookered by lousy research, and I think that has had both good and bad effects. The good news is that I’ve noticed a clear trend toward greater care in reporting the results of studies (not at propaganda factories like the New York Times, of course, but at serious newspapers). In particular, we’re seeing journalists talk about studies in the context of previous studies that have looked at the same question. Of course, we have a long way to go. But we’re on the way up.

On the bad side, however, I have also noticed a greater reluctance to cover studies at all. Part of that is no doubt due to the increase in volume. I’m young, but even I can remember the heady days of 2003 when any serious empirical study on the effects of a controversial education policy (vouchers, charters, high-stakes testing) would get at least some coverage. Now it’s different, and (to echo Eduwonk) that’s a good thing. But I think it’s extremely unlikely that this is the only factor at work. Junk science has poisoned the well for serious research. No doubt that was part of its intended purpose (although of course the motives of those who produce it have no relevance to its scientific merts or lack thereof).

My hope is that journalists will soon realize they’re getting left behind if they don’t learn how to cover the research accurately. Their job is to go where the news is. If the news is in quantitative research - and that is in fact where a lot of it is – they’ll have to learn how to get there.

Also, the changing media landscape will help. The old idea that journalists must be neutral stenographers with Olympian detachment from all the issues they cover is an artifact of the mid-20th-century role of the media as oligarchic gatekeeper, and is rapidly dying out. As “news” increasingly includes coverage by people who are actively engaged in a field, even as advocates, we can expect the news to be increasingly provided by people with greater amounts of specialized knowledge. (By the way, the old idea of the scholar as detached Olympian stenographer is equally an artifact of vanished circumstances, and will probably be the next thing to go; see the Our Challenge to You statement on the inside cover of any empirical study published by the Friedman Foundation for our views on the relationship between advocacy and scholarship.)

An optimistic view, yes – but since my optimism on other subjects has been triumphantly vindicated over the past year, even when the conventional wisdom said to head for the hills, I think I’ll let it ride.

Leave a Comment » | journalism, methodology | Tagged: , , | Permalink
Posted by Greg Forster

Eduwonkette Apologizes

July 8, 2008

I appreciate Eduwonkette’s apology posted on her blog and in a personal email to me.  It is a danger inherent in the rapid-fire nature of blogging that people will write things more strongly and more sweeping than they might upon further reflection.  I’ve already done this on a number of occasions in only a few months of blogging, so I am completely sympathetic and un-offended.

One could argue that these errors demonstrate why people shouldn’t write or read blogs.  In fact some people have argued that ideas need a process of review and editing before they should be shown to the public.  These people tend to be ink-stained employees of “dead-tree” industries or academia, but they have a point: there are costs to making information available to people faster and more easily.

Despite these costs the ranks of bloggers and web-readers have swelled.  There are even greater benefits to making more information available to more people, much faster than the costs of doing so.  People who read blogs and other material on the internet are generally aware of the greater potential for error, so they usually have a lower level of confidence in information obtained from these sources than from other sources with more elaborate review and editing processes.  Some material from blogs eventually finds its way into print and more traditional outlets, and readers increase their confidence level as that information receives further review.

Of course, the same exact dynamics are at work in the research arena.  Releasing research directly to the public and through the mass media and internet improves the speed and breadth of information available, but it also comes with greater potential for errors.  Consumers of this information are generally aware of these trade-offs and assign higher levels of confidence to research as it receives more review, but they appreciate being able to receive more of it sooner with less review.

In short, I see no problem with research initially becoming public with little or no review.  It would be especially odd for a blogger to see a problem with this speed/error trade-off without also objecting to the speed/error trade-offs that bloggers have made in displacing newspapers and magazines.  If bloggers really think ideas need review and editing processes before they are shown to the public, they should retire their laptops and cede the field to traditional print outlets. 

We have a caveat emptor market of ideas that generally works pretty well.

So it was disappointing that following Eduwonkette’s graceful apology, she attempted to draw new lines to justify her earlier negative judgment about our study released directly to the public.  She no longer believes that the problem is in public dissemination of non-peer-reviewed research.  She’s drawn a new line that non-peer-reviewed research is OK for public consumption if it contains all technical information, isn’t promoted by a “PR machine,” isn’t “trying to persuade anybody in particular of anything,” and is released by trustworthy institutions.

The last two criteria are especially bothersome because they involve an analysis of motives rather than an analysis of evidence.  I defended Eduwonkette’s anonymity on the grounds that it doesn’t matter who she is, only whether what she writes is true.  But if Eduwonkette believes that the credibility of the source is an important part of assessing the truth of a claim, then how can she continue to insist on her anonymity and still expect her readers to believe her.  How do we know that she isn’t trying to persuade us of something and isn’t affiliated with an untrustworthy institution if we don’t know who she is?  Eduwonkette can’t have it both ways.  Either she reveals who she is or she remains consistent with the view that the source is not an important factor in assessing the truth of a claim.

No sooner does Eduwonkette establish her new criteria for the appropriate public dissemination of research than we discover that she has not stuck to those criteria herself.  Kevin DeRosa asks her in the comments why she felt comfortable touting a non-peer-reviewed Fordham report on accountability testing. That report was released directly to the public without full technical information, was promoted by a PR machine, comes from an organization that is arguably trying to persuade people of something and whose trustworthiness at least some people question.

So, she articulates a new standard: releasing research directly to the public is OK if it is descriptive and straightforward.  I haven’t combed through her blog’s archives, but I am willing to bet that she cites more than a dozen studies that fail to meet any of these standards.  Her reasoning seems ad hoc to justify criticism of the release of a study whose findings she dislikes.

Diane Ravitch also chimes in with a comment on Eduwonkette’s post: “The study in this case was embargoed until the day it was released, like any news story. What typically happens is that the authors write a press release that contains findings, and journalists write about the press release. Not many journalists have the technical skill to probe behind the press release and to seek access to technical data. When research findings are released like news stories, it is impossible to find experts to react or offer ‘he other side,’ because other experts will not have seen the study and not have had an opportunity to review the data.”

Diane Ravitch is a board member of the Fordham Foundation, which releases numerous studies on an embargoed basis to reporters “like any news story.”  Is it her position that this Fordham practice is mistaken and needs to stop?

3 Comments | methodology, research reports | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

What Does the Red Pill Do If I Don’t Take It?

June 19, 2008

 

(Guest Post by Matthew Ladner)

The hidden highlight from the Evaluation of the DC Opportunity Scholarship Program: Impacts After Two Years report is buried in the Appendix, pp. E-1 to E-2:

Applying IV analytic methods to the experimental data from the evaluation, we find a statistically significant relationship between enrollment in a private school in year 2 and the following outcomes for groups of students and parents (table E-1):

• Reading achievement for students who applied from non-SINI schools; that is, among students from non-SINI schools, those who were enrolled in private school in year 2 scored 10.73 scale score points higher (ES = .30)^2 than those who were not in private school in year 2.

• Reading achievement for students who applied with relatively higher academic performance; the difference between those who were and were not attending private schools in year 2 was 8.36 scale score points (ES = .24).

• Parents’ perceptions of danger at their child’s school, with those whose children were enrolled in private schools in year 2 reporting 1.53 fewer areas of concern (ES = -.45) than those with children in the public schools.

• Parental satisfaction with schooling, such that, for example, parents are 20 percentage points more likely to give their child’s school a grade of A or B if the child was in a private school in year 2.

• Satisfaction with school for students who applied to the OSP from a SINI school; for example, they were 23 percentage points more likely to give their current school a grade of A or B if it was a private school.

I’m trying to figure out why the impact of actually using the voucher program isn’t actually the focus of this study, and in fact is presented in an appendix. Instead all the “mixed” results are studying the impact of having been offered a scholarship whether the student actually used it or not.

I’m going to walk way out on a limb here and predict that the impact on test scores of being offered but not using a voucher will be indistinguishable from zero. If this were a medical study, we would have a group of patients in a control and experimental group offered a drug, some of them choose not to take it, but we ignore that fact and measure the impact of the drug based on the results of both those who took it and those who didn’t. Holding the pill bottle can’t be presumed to have the same impact as taking the pills.

We’ve all been told that exercise is good for our health. Should we judge the effectiveness of exercise on health outcomes by what happens to those who actually exercise, or by the results for everyone that has been told that it is good for you?

This shortcoming has been corrected in the Appendix, but that is getting very little attention. On page 24 the evaluation reads:

Children in the treatment group who never used the OSP scholarship offered to them, or who did not use the scholarship consistently, could have remained in or transferred to a public charter school or traditional DC public school, or enrolled in a non-OSP-participating private school.

So in the report’s main discussion, the kids actually attending private schools have to make gains big enough to make up for the fact that many “treatment” kids are actually back in DCPS. As it turns out, several subsets of students do make such gains, but that’s not the point. The point is we ought to be primarily concerned with whether actual utilization of the program improves education outcomes and with systemic effects of the program. We should indeed study who actually uses this program, and who chooses not to and the reasons why (very important information), but this sort of analysis seems to belong in the appendix rather than the other way around.

Receiving an offer of a school voucher doesn’t constitute much of an education intervention, and it seems painfully obvious that the discussion around this report is conflating the impact of voucher offers with that of voucher use. The impact of voucher use is clear and positive.

12 Comments | methodology, research reports, vouchers | Tagged: , | Permalink
Posted by matthewladner

The SAT and College Grades

June 18, 2008

(Guest post by Larry Bernstein)

Yesterday, the College Board released a study of the predicative power of the SAT to estimate a student’s college freshman year grade point average. A Bloomberg article condemned the results because of the relative ineffectiveness of the new SAT to predict college grades. The predictive power of the SAT is trivially improved by the addition of the new essay exam which adds test time and is costly to grade.

I think this should come as no surprise, and it shows the general limitations of using standardized tests to predict college grades. One of the key points made in the study is that high school grades are a better predictor versus the SAT. High school grades need to be included with the SAT to best estimate GPA. 

In my 1985 Wharton undergraduate statistics class, each student was required to create a regression research project. By chance, I chose to research predicting my classmate’s college GPA. I used 20 variables, including the SAT score, and I found only 5 variables with statistical significance: SAT score, number of hours studied, Jewish or Gentile, Wharton or other school such as the college of arts and sciences, and raised in the Northeast or elsewhere.

Similar to the national studies, in my survey of 100 fraternity brothers the SAT score did a mediocre job of predicting college GPA as a single variable. The key variable in my study was the number of hours studied. You would be surprised by the variance in Ivy Leaguers’ study habits. My survey asked students to estimate the number of hours as 1-10, 10-20, 20-30, or 30-40.  My favorite response was: “Is this per semester?” I assumed the student would realize it was per week! Work habits and effort played a critical role in estimating college GPA. Obviously, the college placement office will have difficulty estimating this variable, though difficulty of course load and number of AP classes might help.

The rest of the variables seem obvious. It is much more difficult to get into Wharton than the other programs at Penn. So it is no surprise that Wharton students were running circles around the non-Wharton students, even adjusting for SAT scores and hours studied. In addition, it is much more difficult to get into Penn from the NE than from other areas of the country.

Very few of the Jews were jocks. Needless to say my college fraternity had plenty of sample problems.

2 Comments | higher education, methodology, research reports | Tagged: , , , , | Permalink
Posted by Greg Forster

Strawman — er, I mean — Strawperson

May 22, 2008

The American Association of University Women released a report this week attempting to debunk concerns that have been raised about educational outcomes for boys.  The AAUW report received significant press coverage, including articles in the WSJ and NYT. 

But the AAUW report simply debunks a strawman — er, I mean — strawperson.  The report defines its opponents in this way: “many people remain uncomfortable with the educational and professional advances of girls and women, especially when they threaten to outdistance their male peers.”  Really?  What experts or policymakers have articulated that view?  The report never identifies or quotes its opponents, so we left with only the Scarecrow as our imaginary adversary.

Once this stawperson is built, it’s easy for the report to knock it down.  The authors argue that there’s no “boy crisis” because boys have not declined or have made gradual gains in educational outcomes over the last few decades.  And the gap between outcomes for girls and boys has not grown significantly larger. 

This is all true, as far as it goes, but it does not address the actual claims that are made about problems with the education of boys.  For example, Christina Hoff Sommers’ The War Against Boys claims: ”It’s a bad time to be a boy in America… Girls are outperforming boys academically, and girls’ self-esteem is no different from boys’. Boys lag behind girls in reading and writing ability, and they are less likely to go to college.”  Sommers doesn’t say that boys are getting worse or that the gap with girls is growing.  She only says that boys are under-performing and deserve greater attention. 

Nothing in the new AAUW report refutes those claims.  In fact, the evidence in the report clearly supports Sommers’ thesis.  If we look at 17-year-olds, who are the end-product of our K-12 system, we find that boys trail girls by 14 points on the most recent administration of the Long-Term NAEP in 2004 (See Figure 1 in AAUW).  In 1971 boys trailed by 12 points.  And in 2004 boys were 1 point lower than they were in 1971. 

In math the historic advantage that boys have had is disappearing.  In 1978 17-year-old boys led girls by 7 points on the math NAEP, while in 2004 they led by 3 points.  (See Figure 2 in AAUW)  Both boys and girls made small improvements since 1978, but none since 1973.

Boys also clearly lag girls in high school graduation rates.  According to a study I did with Marcus Winters, 65% of the boys in the class of 2003 graduated with a regular diploma versus 72% of girls.  Boys also lag girls in the rate at which they attend and graduate from college.  While boys exceed girls in going to prison, suicide, and violent deaths.

It takes extraordinary effort by the AAUW authors to spin all of this as refuting a boy crisis.  They focus on how the gap is not always growing larger and that boys are sometimes making gains along with girls.  They also try to divert attention by saying that the gaps by race/ethnicity and income are more severe.  But no amount of spinning can obscure the basic fact that boys are doing quite poorly in our educational system and deserve some extra attention.

To check out what other bloggers are saying on this report see Joanne Jacobs, and just this morning, Carrie Lukas in National Review Online.

3 Comments | methodology, research reports | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

« Previous Entries
Next Entries »
Follow

Get every new post delivered to your Inbox.

Join 940 other followers