Florida’s Grammar Controversy

May 24, 2012

(Guest Post by Lindsey Burke)

In the most recent administration of the Florida Comprehensive Assessment Test (FCAT) – the state’s criterion-referenced assessment of student achievement – Florida students were asked to pay a little more attention to punctuation, grammar, and spelling in order to get a passing grade on the writing assessment. FCAT cut scores were to reflect that, with proficiency status awarded to those students who could meet the requirements of the new grammar-sensitive assessment.

This rather trivial change has set off a firestorm in the Sunshine State, which just released this year’s FCAT scores, graded under the more rigorous standards.

In 2011, a whopping 81 percent of Florida’s fourth graders scored a 4 or better on the writing portion of the FCAT. Just 27 percent of the youngsters scored proficient under the more rigorous standards this year. Eighth and tenth graders saw similar declines.

The dramatic drop prodded the state board of education to revise the cut scores downward, temporarily dropping the passing mark from 4 to 3 (out of a possible 6 points).

Over the past decade, Florida has made dramatic gains in academic achievement. Florida skyrocketed from 5th worst in reading performance on the NAEP in 1998 to 8th best by 2007, significantly increased the number of students who take and pass AP exams, and began to narrow the achievement gap between white and minority students (Black and Hispanic students in Florida had twice the reading gains of the national average from 1998 to 2009). But evidence suggests that progress the Sunshine State had begun to taper out, with students plateauing on recent assessments.

Keen to ensure student achievement continued apace, Florida proactively raised the rigor of the FCAT – something they’ve done every other year or so since Gov. Jeb Bush’s A-PLUS plan was implemented. According to Commissioner Gerard Robinson, the board of education “asked scorers to grade essays more strictly, with an eye to punctuation, grammar and the quality of word choice and relevance.”

As Florida reels under the draconian requirements of – gasp! – punctuation awareness in a writing assessment, there’s a lesson to be learned for federal and state policymakers eager to adopt national standards and tests.

The backlash against Florida’s efforts to improve the rigor of the FCAT begs the question: what is the correct level of rigor for the 46 states that have adopted Common Core national standards that will not elicit similar reactions? We have yet to learn where the Common Core central planners will set their cut scores, or how they plan to go about setting passing marks on which both Alabama and Massachusetts will agree.

It is a cautionary tale for national standards proponents. Much of Florida’s success over the past decade can be attributed to the state continuously improving its standards and tests. With rigid national standards in place, that flexibility would be lost. And if mistakes are made in the standards, they’re here to stay.

Florida will likely succeed, as it has over the past few years, at striking the right balance on the FCAT. But being able to define what Florida students should know and be able to do, and crafting standards and tests to reflect that, will be lost if the state goes through with Common Core adoption.

Florida strengthened state tests to make sure kids could spell, apply punctuation, and grasp other grammar concepts. These are nuances the state will no longer be able to enjoy come 2014, when national standards and tests are to be fully implemented. The Sunshine State wants to continue its march to the top of the NAEP, and has been working to strengthen standards to achieve that goal. But that ability will soon be lost, which is the ultimate lesson that should be gleaned from the FCAT controversy.

(edited to fix a typo)


Here’s Why Victory Looks Like This

March 7, 2012

(Guest post by Greg Forster)

Jay points to the way Democrats and progressives are now saying all the same things we’ve been saying for a decade, but acting like they thought of them, and remarks that this is What Victory Looks Like.

He’s right, and here’s why. To large extent, you have to let people “steal” your ideas in order to get victory. It’s not just a price we need to be willing to pay if necessary. It’s always necessary.

Major reform of a cultural system has to start with ideas and practices germinating outside the core institutions of that system. If major reform were welcome inside the core institutions, it wouldn’t be necessary in the first place. The incubators of reform can’t be seen as fringe groups – this is why organized libertarianism has had much less influence than its intellectual seriousness and devotion of financial resources might lead you to expect. But the reform incubators are never going to be inside the core, either. You need something that’s a happy medium between credibility and independence.

Now, for a long time in America, the Democratic party and the progressive ideological movement have been the “core” institutions governing education. When you ask the American people whom they trust to do the right thing about education, they overwhelmingly say Democrats and progressives. That makes them the core.

The key to victory is to get the core groups to adopt the ideas that incubated in institutions outside the core. The greatest challenge is that the core groups want to defend their “core” turf against outsiders. They want to keep control of the core, and they can’t do that if they admit that outsiders have superior ideas. The solution is to get the core groups to co-opt (i.e. steal) the ideas and pretend they thought of them.

So you’re never going to get (very many) Democrats and progressives saying, “Why, yes, as a matter of fact the conservatives were right about education all along!” Admitting that would require them to sacrifice their status as the cultural core institutions of American education. Instead they’re going to say, “What American schools need are good, liberal, progressive ideas like choice, competition, and accountability.”

That’s what victory looks like.


The Dark Days of Educational Measurement in the Sunshine State Ended in 1999

February 8, 2012

(Guest Post by Matthew Ladner)

Over on the Shanker Blog of the American Federation for Teachers, Matthew DiCarlo writes a thoughtful but ultimately misguided post A Dark Day for Education Measurement in the Sunshine State.

DiCarlo is obviously very bright, but a few critical misinterpretations have led him astray. DiCarlo demonstrates that family income is highly correlated with student test scores in Florida. No surprise- the same is true everywhere.

Having demonstrated this, DiCarlo develops a critique of Florida’s school grading system. The Florida school grading system carefully balances overall performance on state exams with academic growth over time. Specifically, the formula weights student proficiency on state exams as 50% of a schools grade, 25% on the growth of all students, and the final 25% on the growth of students who scored in the bottom quartile on last year’s exam.

The last bit is the clever part of the formula. By double weighting the gains of students who are behind, they become the most important children in the building. Only the bottom quartile from last year’s test count in all three categories.

DiCarlo goes into the devilish details about how the state determines these gains, and concludes that some of the gains measures don’t actually measure academic growth but actually effectively measure academic proficiency. The use of proficiency levels in determining gains is critical because students are taking a higher grade level assessment with more rigorous content.  If a student achieves a proficient score on the eighth grade FCAT and then again on the ninth grade FCAT, the student is performing at a higher level because the content is more difficult.  Florida’s system does not provide credit for a learning gain for students performing Advanced in one year but Proficient the next year.

DiCarlo has failed to appreciate that the mastery of more challenging academic material from one grade to the next itself constitutes a form of academic growth.

The 9th grade student has now studied the mathematics curriculum of both 8th and 9th grade and has demonstrated  proficiency of the 8th grade material and  proficiency of the 9th grade material. Given the valid system of testing, we can feel assured that the 9th grader knows more about math than he or she knew as an 8th grader. The growth in this case is staying on track in a progressively more challenging sequential curriculum.

The Florida system, in essence, makes use of proficiency levels in order to give definition to gains and drops as meaningful. There of course is no “correct” way to structure such a system, and if 100 different people examined any given system they would likely have 500 different suggestions for improvement to match their preferences.

DiCarlo’s notion of “fairness” seems to have distracted him from a far larger and more important issue: the utility of the Florida grading system, seen best at the school grading level, has improved student achievement for all students.

If you go back as far as the FCAT data system will take you for results by Free and Reduced lunch eligibility for 3rd grade reading, you’ll find that in 2002 48% of Florida’s free and reduced lunch students scored FCAT 3 or better. In the most recent data available from 2010, 64% scored FCAT 3 or better. That is an enormous improvement in the percentage of students scoring at grade level or better.

In 2002, 60% of all Florida students scored Level 3 or above, and in 2010, 72% scored Level 3 or above. Free and reduced lunch eligible kids in 2010 outperformed ALL kids in 2002 by 4 percentage points. That’s real progress.  And the free and reduced lunch eligible children overtake the 2002 general population averages in a large majority of grades tested.

The same pattern can be found in Florida’s NAEP data. For instance, in 1998, 48% of Florida’s free and reduced lunch eligible students scored “Below Basic” on the NAEP 8th grade reading test. In 2011, that number had fallen to 35%. If an “unfair” system helps to produce a 27% decline in the illiteracy rate among low-income students, I’d like to order up a grave injustice.

The “Dark Days of Education Measurement in Florida” in my view were before school grades. Academic failure lied concealed behind a fog of fuzzy labels, and Florida wallowed near the bottom of the NAEP exams. Back when there was little transparency and even less accountability, far more students failed to acquire the basic academic skills needed to succeed in life. While perhaps a lost golden age for educators and administrators wishing to avoid any responsibility for academic outcomes, it was a Dark Age for students, parents and taxpayers.

Ironically, DiCarlo has decried a system which has weakened the link between family income and academic outcomes demonstrated in his post. Yes it is still strong in Florida, but it used to be much, much stronger.

Finally, can one truly complain about the “fairness” of a system providing more than ten times as many A/B grades as D/F grades? If anything, the Florida school grading system has grown too soft in my view (see chart above).

I’ve read enough of DiCarlo’s work to know that he is a thoughtful person, so I hope he will examine the evidence for himself and reconsider his stance. I don’t have any reason to think that the Florida system is perfect. I don’t think a perfect system exists, and I suspect that there are some changes to the Florida system that DiCarlo and I might actually agree on.

It seems however difficult to argue that the Florida system hasn’t been useful if one gives appropriate weight to the interests of students, parents and taxpayers to balance those of school staff.


Something Rotten in the State of NAEP?

November 10, 2011

(Guest Post by Matthew Ladner)

So if you measure the learning gains for children with disabilities on the four main NAEP exams for the entire period all 50 states and the DC have participated, you get the information in the above chart. Last week, the Bluegrass Institute’s Richard Innes alerted me in the comments and by email about fishy exclusion rates for children with disabilities and English Language Learners. I had only casually examined the exclusion rates, but having examined them more closely, I’m concerned.

The 2011 NAEP included standards for inclusion, which include 95% of all students selected for testing, including 85% of students with disabilities or classified as English Language Learners. One might possibly infer that some states were playing games and tricks with excluding such students in the past, and that simply listing the rates wasn’t doing the trick. This year, they listed expected standards and provided the gory details in an Appendix. On the conference call regarding the results, the NAEP team took pains to note this innovation.

So, as you can see, half of the states in the Top 10 gainers for children with disabilities just so happen to be states that violated the inclusion standards on one or more NAEP exam. Hmmm. Moreover, some of them didn’t just barely miss these standards, but instead chose to commit violence against them.

Maryland led the nation in gains among children with disabilities….or did they? Maryland’s inclusion rate for children with disabilities on the 4th grade reading test in 2011: 31%, which though completely pathetic actually beat the 30% rate for children with disabilities on the 8th grade reading test. The ELL rates were almost as bad.

The only other state to sink into the 30s? That would be second place Kentucky, which also excluded an enormous number of ELL students from NAEP examination. The math exams were better than the reading, but lo and behold- there is Maryland again falling below inclusion standards. Maryland failed to meet the 95% overall inclusion standard on 3 out of the 4 exams in 2011.

I have run the numbers for gains among children who are neither disabled nor ELL, and something real and positive is happening in Maryland: scores are up. It is however obvious that the NAEP created these standards for a reason, and have invited people to make up their own minds about whether to throw a skeptical flag in the air.

I’m throwing my flag. I don’t know if it explains all of the gains in Maryland and Kentucky, but it seems pretty obvious to me the results from those two states and perhaps others ought not to be considered comparable to the other states.

I’ve been told and I find it credible that these exclusions have only a small impact on the statewide numbers. Can we imagine however that very high exclusion rates for ELL students will not heavily bias the Hispanic number? Or that sky-high special ed exclusions won’t inflate a variety of subgroup scores? Or that excluding many of both of these subgroups won’t impact your Free and Reduced lunch eligible sample?

So given that the Congress mandated participation in NAEP as a part of NCLB, a mandate which all the federalist bones in my body find quite reasonable, perhaps it would be a jolly good idea for Congress to mandate minimum inclusion rates along with participation when reauthorization finally rolls around. Caesar’s wife must be above suspicion.


2011 NAEP: Florida Finally Hits a Wall

November 3, 2011

(Guest Post by Matthew Ladner)

The Florida Age of Public School Improvement hit a wall in the 2011 NAEP. This should not be terribly surprising, as Florida’s improvement seemed certain to plateau in the absence of additional reforms.

Governor Jeb Bush relentlessly pursued a dual strategy- transparency with teeth from the top down, parental choice from the bottom up. Together these reforms drove improvement in the public schools for a number of years.  Accountability measures included school grading (A-F) and earned promotion in the early grades. Parental choice measures included Opportunity Scholarships for children attending F rated schools, the nation’s first special needs voucher program (McKay Scholarships), the nation’s largest scholarship tax credit program (Step Up for Students), a decent charter school law and the nation’s most robust system of digital learning. Florida lawmakers also attempted to thoughtfully incentivize success.

Governor Bush took office in 1999 and left office in 2007. It would be nice if these efforts could indefinitely push progress forward, but there have been plenty of bumps and problems along the way. In 2006, the Florida Supreme Court rendered a logic-free ruling abolishing Opportunity Scholarships (failing school vouchers) for private schools, and followed that up by ruling against a state authorizer for charter schools. Tax-credits, McKay and digital learning continued to incrementally advance, but not at an earth-shattering rate.

The larger problem may have come in the top down measures. The chart below presents the distribution of district and charter school grades, with one line being the A/B grades and the other D/F grades. The dotted lines represent instances when the state board raised school grading standards.

The setting of these standards represents far more of an art than a science. Set them far too high and disaster follows (this happened in Arizona). Set them too low, and you remove the tension in the system needed to drive improvement. Even after the last increase in grading standards, more than 10 times as many Florida schools received A/B grades as D/F grades.

Florida’s policymakers raised standards four times, and last year (wisely) put in an automatic trigger to raise standards by a preset amount when a certain ratio of schools get A or B grades. In addition, a fresh set of reforms passed the Florida legislature in 2011, revamping teaching and increasing charter school and digital learning options.

Just as it is impossible to exactly pinpoint how much of what caused the gains, it is likewise impossible to say exactly what made them stall. Note however that one of the favorite explanations of the anti-reform crowd, the pre-school, finally saw the advent of children old enough to have participated in the program and age into the 4th grade NAEP sample. I hope that someone is carefully studying variation in participation and corresponding trends in FCAT data, but the results at the aggregate level thus far seem underwhelming.

Plenty of other things, however, have been going on- including the collapse of a housing bubble, cutbacks in public school funding (including of some of the incentive funding programs) and a variety of other very bad things. My advice to Florida policymakers: roll up your sleeves and get back at it. Despite the enormous amount of progress seen on NAEP (and no one loves celebrating it more than me) too great of a gulf lies between a state system awarding ten times as many top grades as low grades but still  suffering from large minorities of students scoring below basic on the NAEP exams.

Governor Bush has consistently said for years that success is never final, and reform is never finished. The 2011 pause in progress demonstrates that he called it correctly.  Moving the needle on student learning on a meaningful scale and at a sustained basis represents one of the greatest public policy challenges of our times. Governor Bush has passed the torch to a new generation of Florida reformers, and they must now find new ways, and fine-tune the old ways, to push academic progress forward.

Edited for typos


Some Thoughts in Advance of NAEP ’11 Release

October 27, 2011

(Guest Post by Matthew Ladner)

NAEP is going to release the 2011 Reading and Mathematics results on November 1st. I thought it would be interesting to boldly make some predictions in advance. Here’s my first one: the 2011 results won’t be all that different from the 2009 results.

I know, I’m going waaaaay out on a limb here, but that’s my prediction and I am sticking to it.

While a number of states have engaged in far-reaching reforms, the vast majority of these efforts still lie in the implementation stage. Possible exceptions in my mind include Washington D.C., Louisiana and Florida.

For DC, the 2011 NAEP will constitute the first plausible check on the tenure of Michelle Rhee. DCPS began making substantial math and reading progress in the mid 1990s, with huge gains but with scores still low. Assuming normal lags between changes and impacts, I believe that the 2009 NAEP arrived a bit early. I’ll be very interested to see what happens with the 2011 scores. Washington DC is also experiencing gentrification, so I will look at the free and reduced lunch numbers.

Louisiana will be a very interesting case, as some important statewide reforms still remain in the implementation phase, but where New Orleans has been in serious reform mode since 2005. I’ll take a look at the trend in urban numbers.

Florida of course enjoyed a steady increase in NAEP scores since 1998. Florida lawmakers also instituted a fresh set of far-reaching reforms in 2011, but the verdict on those will come years down the road. Governor Crist failed to pursue far-reaching reforms of his own, and vetoed some of those that reached his desk. Florida’s scores may rise again, but I won’t be surprised if they hit a plateau.

The Great Recession may also make this NAEP a little less incremental that usual. It will be interesting to see what happens to scores in the “Sand States” with the greatest property crashes (Arizona, California, Florida and Nevada) in addition to other states with acute economic distress like Michigan.

I will look with some interest at Arizona’s scores. Not only is the state face down on the economic canvass, with house building and flipping having been signature industries before the pop, it is possible that the infamous SB 1070 may lead to the illusion of progress in Hispanic scores. To the extent that the already partially overturned SB 1070 convinced undocumented families to leave Arizona, it may create the appearance of academic improvement.

Outside of that, I’ll be looking for pleasant surprises. Tell me what you are interested in seeing from the 2011 NAEP in the comments.


What’s Going Right in Waconda?

October 24, 2011

According to the Global Report Card that Josh McGee and I developed, tiny Waconda, Kansas is one of the top-performing school districts in the United States.  Other than being the home to what residents claim is the world’s largest ball of twine (pictured above), one might not think that there was anything exceptional about this rural, farm community in north central Kansas.

But in 2007 the average student in Waconda performed better than 91% of students in our 25 country comparison group in math achievement.  If we relocated Waconda to Finland, the average student in Waconda would outperform 88% of the students in Finland in math.

A reporter for Yahoo News was curious about what they were doing right in Waconda.  Here is what she found:

So why are Waconda kids–65 percent of whom live in poverty–doing so well? And can other schools follow their lead?

The Waconda district comprises four small towns–Cawker City, Downs, Glen Elder and Tipton–and seven schools spread over 411 square miles. Most people in the area work in agriculture or in manufacturing.

The district’s superintendent of seven years, Jeff Travis, told Yahoo News that after years of high test scores, the community expects its students to excel. Most years, he added, no one drops out of high school. The district won 14 state Governor Achievement Awards and one national “Blue Ribbon Award School” over the past four years.

“It’s a tradition now, and they expect themselves to do well,” Travis said. “Like a ball team that continues to win because of a tradition, we have an academic tradition.”

Still, the community doesn’t quite seem to get how exceptional they are. “Everybody’s pretty happy [but] nobody understands how big a deal it is,” he said.

Travis says the students’ high level of achievement is even more extraordinary given that 65 percent of them qualify for free or reduced federal lunches, an indication that they live in poverty. High poverty schools are often dogged by low test scores and high dropout rates. Many educational observers indeed blame the nation’s sky-high child poverty level for the country’s comparatively low performance in math.

One theory Travis has is that Waconda school kids have no sense that they’re materially deprived. “North Central Kansas is rural, and urban poverty is kind of different [from] rural poverty,” he said. “A lot of our people don’t even understand that they’re living in poverty.” According to state data, most of the students are white, and no kids need English language learning classes.

About 10 percent of the students in the school district are foster kids, Travis says. “We just [have] a lot of adults that care about kids, so it’s been a popular thing for parents to take in foster children.”

But Josh adds a useful note of caution at the end of the reporter’s piece:

One of the Global Report Card’s authors, Josh McGee, says the small size of Waconda schools may have skewed the results slightly, since randomness has a greater impact on a smaller sample size. Most of the best-performing school districts in his ranking were small, and many of them were also made up of charter schools. You can read more about his methodology here.

We may not be able to generalize much from the success in Waconda, but it is a fun and impressive story.

Meanwhile, coverage of the Global Report Card continues to stream in.  For an updated list of media for the Global Report Card (with links), click here.


Josh McGee on the Global Report Card

October 18, 2011

Check out Josh discussing the Global Report Card at this Fordham Institution event on “The Other Achievement Gap” focusing on the disappointing performance of some of our “best” students and school districts.

Josh starts at around 12:30 in the video.

Also check out the updated list of coverage on the Global Report Card.


Updated Reporting on the Global Report Card

October 9, 2011

Coverage of the Global Report Card continues to roll in.  Here is a current list:

Global Report Card Results and Article

Education Next

Global Report Card Web Site

Methodological Appendix

Op-eds

Sacramento Bee

Hartford Courant

The Oklahoman

Austin American Statesman

Atlanta Journal Constitution

Interviews

Wall Street Journal (video)

Education Next (video)

Education Next (podcast)

Dallas Morning News (Q&A)

Choice Media.TV (video)

News

Dallas Morning News (subscription required, although a version can be read here)

Arkansas Democrat Gazette (subscription required)

Roll Call (article by Morton Kondracke)

Education Week

Yahoo News

Atlanta Journal Constitution

Time Magazine

KSN-TV

Richmond Times-Dispatch

United Press International

East Valley Tribune (Arizona)

TC Palm

Milwaukee Journal Sentinel

St. Pete Times

Maryland Gazette

Hawaii Reporter

Delaware News-Journal

Kansas Reporter

School Library Journal

My Fox DFW

Dallas Observer

Market Watch

Blogs

Education Next

Cato@Liberty

Joanne Jacobs

Mackinac Center

Illinois Rising

Ed is Watching 

Gotham Schools

Fordham’s Education Gadfly

Flypaper

Bacon’s Rebellion

The Locker Room

The Western Wrangler

Choice Remarks

TPE Post

Missouri Education Watchdog

Whiteboard Advisors

Jorge Werthein

The Caisson

School House Wonk

School Finance 101

Criticism

The last blog post contained some criticisms about whether the assumptions for the analysis were reasonable.  Josh McGee replied in the comment section of that post.  And NCES Commissioner, Jack Buckley, told Education Week that “The methodology in this report is highly questionable.”  This assessment is a little strange because what we did was similar to what the U.S. Department of Education has done in several past reports linking international test results to state NAEP results.  (See for example this.)  We just bring the results down to the district level.  If ours is highly questionable, then the U.S. Department of Education’s own efforts must also be questionable.

(UPDATED 12-19-11)


American School Reform

September 28, 2011

“The spirit of enterprise, which characterizes the commercial part of America, has left no occasion of displaying itself unimproved.”

-Federalist 7

“In a state of disunion…that unequaled spirit of enterprise, which signalizes the genius of the American merchants and navigators, and which is in itself an inexhaustible mine of national wealth, would be stifled and lost, and poverty and disgrace would overspread a country which, with wisdom, might make herself the admiration and envy of the world.”

– Federalist 11

(Guest post by Greg Forster)

Jay, Rick Hess and Paul Peterson have all recently made really impressive scholarly contributions that all point to the same conclusion: “It’s not all about poor kids,” so it’s time to overcome “our achievement-gap mania” and get our “globally challenged” total population of kids – including the middle-class suburban white ones – “ready to compete.” Because it’s clear that they’re not, and it’s clear that they need to be, more than ever before.

This is a really encouraging development and a badly needed message for school reformers. There is no law of nature that says America will always be a flourishing and successful nation, and it will not in fact remain so unless we overcome our myopia and confront the mediocre performance of all our schools.

Raising the “floor” is important. But it’s much more important to get rid of the “ceiling” – the sense that in most schools we’re already good enough, the sense that we don’t need improvement. In fact, removing the ceiling will do more to raise the floor than any of our direct efforts to raise the floor.

Here’s my concern. As we move to confront the middle-class white suburbanites with the inadequacy of their schools, it’s important that the message not be “your school sucks and I can prove it.” Not that I hear Jay, Rick or Paul saying that; they’re not. But that will be the cariacature our enemies will deploy against us. We have to take proactive steps to preempt that tactic.

I think we can improve our message by grounding it in an affirmation of what’s best about America. America is an enterprise society; always has been. America was founded as the country that looked at Europe, clinging (bitterly) to the last remaining remnants of a thousand years of feudalism on the assumption that the basic ways of the world could never be changed, and said: “The old ways aren’t good enough. We can do better. We will plant our roots in the past, but our branches must grow upward.”

We can draw on that as we speak into suburban complacency. A tree that isn’t growing is dying; for nations as for forests, there is no comfortable plateau. Nations that seek comfortable plateaus, like those in Europe today, wither. Americans have never wanted a comfortable plateau; we want every generation to be more blessed than the last. However, the data in our schools show that our national future is clearly not being prepared for growth. But this is America. We don’t accept complacency. We don’t shrug our shoulders and accept decline. We know we can do better. And there are models of reform that can unlock our potential.

Grounding this new direction for school reform in the American culture as an enterprise society will keep us from descending into squabbling over whether we’re “anti-public schools” and keep everyone’s eye on the ball: the flourishing of our national future.