Question for Sara Mead

(Guest Post by Matthew Ladner)

I saw a documentary on Napoleon’s Egyptian campaign a few years ago. After a completely nasty setback, Napoleon retreated in defeat back to Cairo, but then ordered a victory parade to be held before fleeing the country entirely.

Watching Fordham’s pre-school event online, I can’t help but think that pre-k advocates are trying to do the same thing with Oklahoma: pretend its a victory, when in fact it looks more like their Waterloo.

I watched the Fordham Foundation pre-school event online yesterday. I was especially taken by Sara Mead’s claim that universal preschool could lead to dynamic changes in K-12, and that disadvantaged kids in Oklahoma’s pre-k program made larger gains than other students.

The biggest problem for universal pre-k advocates, in my view, is that the academic gains associated with Pre-K programs fade out. Consider the blue line in the chart below-4th grade NAEP scores from Oklahoma. In 1998, Oklahoma adopted a universal pre-k program.

FL vs. OkI assume that Ms. Mead has a basis to say that disadvantaged children make bigger gains under the Oklahoma pre-k program. The more important question is whether those gains are sustained over time.

Based upon the NAEP scores, Oklahoma’s program looks like a dud, increasing all of one point between 1998 and 2007.

The best one can try to spin out of the Oklahoma situation is scores might have actually dropped in the absence of the program, but now you are really grasping at straws. I seriously doubt that anyone who voted for this program in 1998 could be anything other than disappointed.

The red line, Florida, shows what can be done with a vigorous effort to improve K-12 schools. Florida’s low-income children improved by 23 points between 1998 and 2007.

Florida voters created a universal pre-k program, which was implemented as a voucher, but none of those students had reached the 4th grade by 2007.

Mead would likely argue, and I think she did at the event, that Pre-K and K-12 reform aren’t mutually exclusive, and I agree. It seems fair to ask however: is Pre-K a waste of time as an education improvement strategy? If not, why are the Oklahoma results so dreadfully unimpressive?

About these ads

13 Responses to Question for Sara Mead

  1. Alsadius says:

    If Oklahoma adopted the program in 1998, then that means that the first cohort through would have been, assuming pre-K started in the 1998-1999 school year, 2003-2004. Thus, between 2003(no mandatory pre-K) and 2005(mandatory pre-K) the numbers jumped 5 points, fully reversing a past decline, and then held steady in 2007.

    This is obviously a wholly inadequate amount of data, but from what’s here, the program seems to have been a modest success. For that matter, the 2.5 points per year of implied growth in scores is exactly the same as Florida managed from 1998-2007. Obviously, a one-time change isn’t sustainable, but from this data mandatory pre-K is as effective as two years of the best reforms in the country.

    Unless I’m missing something, criticizing the program on the basis of this data is completely unwarranted.

  2. matthewladner says:

    Alsadius-

    What you describe is well within the range of white noise on a 500 scale point exam. A one point gain between 1998 and 2007 constitutes a failure to move the needle.

  3. Alsadius says:

    But 1998 is an irrelevant data point – it’s several years before the effects of a mandatory pre-K program will hit a 4th grade test. 2002 and 2003 are immediately before, and 2005 and 2007 are immediately after, and the difference between the two is 5.5 points, not 1. That’s certainly a noticeable difference, even if it’s not earth-shaking.

    Again, this is not enough data to judge the merits of this program by – even neglecting the massive concerns about how correlation is not causation, the missing years are an issue, and the fact that I don’t know whether the program started in 1998-1999 or 1999-2000 adds some uncertainty. But this data is certainly not enough to dismiss the program with. If you remove the implicit argument that a legislative decision to add pre-K should show up in 4th grade test results in the same year, and thus remove the 1998 data point, Oklahoma’s results are in the same ballpark as Florida’s(+6 points in 5 years, instead of +9), the state you like to hold up as the gold standard of education improvement.

    I’m not from Oklahoma, don’t know the legislative history of this program, or really much of anything about it. But your arguments here are setting off alarm bells in my head – the only notable difference between good Florida and bad Oklahoma is in the 1998-2002 interval, which is exactly the interval in which the program you’re using this data to attack should have no impact whatsoever. Add in what looks like cherry-picking of years(seriously, what kind of progression is 98, 02, 03, 05, 07?), and an insistence on the single most irrelevant data point as being the one that is supposed to prove your point, and this starts to look like exactly the sort of dishonest research I’ve seen slagged around here many times.

    I’ve been reading JPGB from the beginning, and I generally like what you guys have to say quite a bit. I’m not some random troll coming in to call you a bad researcher, and I’m not some Oklahoma congressman sockpuppeting to defend my pet project – I’m just confused by what you’re trying to say here, and think that either you or I have missed the point really badly. So let’s get to the root of this. One, why are you including 1998 data in your above chart? Two, why are you excluding the data for 2004 and 2006 from it? Three, why do you not have a “Here’s the first pre-K cohort” marker on the chart, or at least in the notes? And four, how big with the before-and-after discontinuity have to be for you to call the program a success?

  4. matthewladner says:

    Alsadius-

    The chart presents the available NAEP reading data. In recent years they have been going once every two years, so the next one will be 2009. There was no 4th grade reading NAEP given in 2004 or 2006.

    The first wave of kids to reach the 4th grade was probably in 2003-2004 kids assuming it passed in May of 1998. If that is the case, and assuming the 2003 NAEP testing took place in the spring of 2003 for the 2002-03 cohort, then I agree that it would be better to use the 2003 year as the baseline.

    A five point gain is still not at all impressive matched against the rhetoric of pre-school advocates, and of course it could have causes other than pre-school. We are supposed to believe that these programs are going to return benefits many times greater than the cost when in fact they struggle to document any impact on 4th grade reading scores.

    Now, if Oklahoma put this program in place in 1998, and if a few years later they had their free and reduced lunch Hispanic children outscoring 15 statewide averages, I can promise you that I would convert over to the cause. As it stands, this looks pretty unimpressive.

  5. Alsadius says:

    Oh, I assumed it was every year – that’s how all the tests around here are structured, and I sort of assumed everyone did it the same way. That certainly addresses that point.

    As for the program itself, I’ll certainly agree that it underperformed the claims of its advocates – that’s pretty much a null hypothesis in politics anyways. For that matter, I’d generally agree that it probably wasn’t worth the money, assuming it costs as much as similar programs I’ve seen elsewhere. In practice, these programs tend to amount as much to subsidized daycare as they do to education policy.

    It’s not that I object to being harsh – even the good numbers are a 1% increase in test scores for the low-scoring groups of children, supposedly those most positively impacted, which isn’t a huge change even if it is noticeable. My complaint was that your use of a 1998 baseline resulted in criticism that was unduly harsh, and likely unwarranted. If you want to attack it as spending hundreds of millions to boost scores 5 points, that’s a fair complaint. But calling it a 1 point boost is just being sneaky, and in fairly obvious fashion. You’ll do better basing your criticisms on solid ground than you will basing them on flashier numbers with poor grounding.

  6. The Minnesota Kid says:

    I don’t understand how you can claim “fade-out” from aggregated state-level data. You really need to look at individual student-level data to determine (1) program impact and, assuming that, (2) fade-out of that impact. Pre-K is universally available in Oklahoma but participation is not mandated. The fourth graders in Oklahoma who take the NAEP every other year (from 2003 on) will be comprised of universals and non-universals (especially when you consider in-migration) in some unknown and variable proportions. Add to this compositional problem the dastardly Ecological Inference Problem and the bottom line is that you can’t reliably judge the impact or fadeout of a pre-K program based on aggregate fourth-grade NAEP scores, even if you do align the years properly (which you must and initially didn’t). These data really can’t and shouldn’t contribute to such an analysis.

  7. Patrick says:

    I looked at NAEP data for Oklahoma for all students in 4th grade on the reading test.

    In 1992 the score was 220 in 2007 it was 217.

    The problem is that the 1992 test allowed accommodations, 2007 did not. However the 1998 test has results for both accommodations and no accommodations the result is 220 and 219 respectively (not a big deal).

    This doesn’t suggest that pre-k is a resounding success (don’t the “Fade-out” studies actually look at the student level data?). Personally, I think the pre-school idea is just a scam to increase adult employment. Children learn very little in preschool and nothing that they do learn isn’t relearned in Kindergarten and first grade. The most important thing I remember about preschool (besides playing He-Man and She-Ra with my playmate at the time) was adults deciding whether or not kids were potty trained well enough to attend kindergarten.

    I can’t even believe this was taken as a serious idea in the first place.

  8. Patrick says:

    Additionally, Oklahoma’s cumulative net migration from 98 to 07 was just 16,553. They have about 3 million people in the state. Considering they don’t have hundreds of thousands moving in and out every year (that would just be odd) I don’t think migration would really have a big impact.

    Maybe in Nevada where our net migration is considerably larger than the net migration of all 38 states with top public universities according to U.S. News and World Report (which suggests to me that research universities are probably overrated as engines for state economies).

  9. Patrick says:

    *Oklahoma has 3.6 million residents as of July 2008.

  10. matthewladner says:

    Alsadius-

    I don’t think there is anything sneaky about including 1998, if for no other reason that it was the year before the Florida reforms. If you watch the Fordham video, you’ll see Ms. Mead positing that universal pre-k could instigate positive reforms in K-12, rather than serve as a rival for other types of reform. It doesn’t seem to have happened in Oklahoma.

    Minnesota-

    The fade out problem has been established with individual level data. Oklahoma now has the highest pre-school attendance rate in the country, and is spending a great deal of money on the program, so it seems perfectly reasonable to me to track their 4th grade reading scores over time to see if there is any evidence of improvement.

    As it stands, the improvement among FRL kids between 2003 and 2007 was precisely one point higher than the national average (4 points).

  11. Alsadius says:

    Matthew – In the Florida case, yes, it’s relevant, especially given the dramatic results. However, using it as the baseline for Oklahoma was wrong.

    As for Mead’s commentary, I wasn’t trying to defend her. I haven’t had a chance to watch the video, but from your account her thesis seems overblown, at best.

  12. Early compulsory attendance at school is counter-indicated. States which compel attendance at age 7 have higher 4th and 8th grade NAEP Reading and Math scores than States which compel attendance at age 7. Even non-compulsory pre-K poses a threat, from “moral hazard”, with the taxpayers pushing one institutional option, the State (government, generally) tilts the field against whatever other options (part-time work for mom, grandparents, neighbors) parents might select.

    Questions:
    1) Will this universal pre-K employ only non-parents?
    2) Is a low client-to-staff ratio desirable?

    3) If “no” to question #1 and “yes” to question #2, why not leave kids with mom?

  13. Higher at 7 than at age 6.
    Sorry for the typo.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,567 other followers