The Hubris of NCTQ’s Ed School Ratings

One of the bigger problems in education policy is hubris.  People regularly claim that they know what the right policies or practices are, and things would be better if only others would bend to their will.  The truth is that we know relatively little about effective education policies and practices.  This isn’t for lack of trying.  Despite considerable research effort and policy inquiry, we’ve found remarkably few “universal truths” about effective education.  Part of the difficulty is that knowing what works presupposes that there is a single, best way.  But  it appears much of what is effective in education is contingent on particular needs and circumstances and does not lend itself to broad declarations about the “right”  practices and policies.

Because the scourge of PLDD is endemic, however, we continue to hear claims that “We know what works.”   This was the traditional refrain of teacher union leaders, but now reformers have joined the hubris chorus.  The latest example of this is the ratings of Ed Schools issued by the National Council of Teacher Quality.  NCTQ claims to know what good teacher preparation programs should be doing and judges those programs against NCTQ’s vision of effective practices.

In particular, NCTQ identifies 18 standards by which it judges Ed Schools.  “Our standards for the first edition of the Teacher Prep Review” NCTQ assures us, ” are based on research; internal and external expert panels; the best practices of other nations and the states with the highest performing students; and, most importantly, what superintendents and principals around the country tell us they look for in the new teachers they hire.”

NCTQ describes the research basis for their standards in a lengthy document.  Yet, even according to their own description only 8 of the 18 standards are supported by “strong research.”  And in most of the 8 cases where they do claim to have strong research support, the research does not actually provide them with the strong support they assert.

For example, the “Early Reading” standard assesses whether “The program trains teacher candidates to teach reading as prescribed by the Common Core State Standards.”  None of the studies they cite actually examine the specific standard since none specifically examine what methods of teaching reading, if any, are actually prescribed by Common Core.  As is the case with all 18 standards in the NCTQ rating system, one has to make a series of leaps between the research cited and the actual standard being used to judge teacher prep programs.

In the case of early reading, the “strong research” they cite examines whether teachers are familiar with the “five components of effective reading instruction,” and whether teachers who are certified and have masters degrees are more likely to know those five components.  It turns out teachers are generally not familiar with the five components and are no more likely to know them if they are certified or have a masters.  That’s all very nice, but isn’t the “strong research” supporting the standard supposed to show that knowledge of the five components, which presumably have something to do with teaching “reading as prescribed by the Common Core State Standards,” actually lead to improved reading by students?  The strong research cited by NCTQ says it generally doesn’t: “This study also found no relationship between teachers’ knowledge of these components and their students’ reading growth – with the notable exception of third-grade students.”  This is typical of the “strong research” supporting 8 of the 18 standards by which NCTQ judges Ed Schools.

Standards 1 and 6 address whether teacher prep programs select “teacher candidates of strong academic caliber” and whether “teacher candidates have the broad content preparation necessary to successfully teach to the Common Core State Standards.”  In both cases the “strong research” on which these standards rely is a study by Boyd, et al examining the relationship between teacher characteristics and student achievement.  Let’s leave aside the fact that NCTQ acknowledges that research by Harris and Sass as well as Chingos and Peterson contradict their standard.  Even the Boyd, et al study they do cite does not specifically demonstrate that teachers from more selective programs or with more content training are more effective.  First, Boyd, et al are careful not to make the type of strong causal claims from their work that NCTQ does:

It is not easy to estimate how the achievement gains of students are affected by the qualifications of their teachers because teachers are not randomly sorted into classrooms. For example, if teachers in schools in which students perform best in math are more likely to be certified in math, one might be tempted to conclude that being certified to teach math contributes to higher student achievement. The causal relationship, however, may operate in the other direction; that is, more qualified teachers may be in schools where students perform well in math because they prefer to teach good students and because employers want to staff their courses with in-field certified teachers. Analysts need to be careful not to attribute the test-score gains associated with sorting to the attributes of teachers.

Beyond the fact that Boyd, et al would not make the strong causal claims from their work that NCTQ feels free to do, the Boyd, et al study examines a basket of teacher qualifications and does not claim to be able to distinguish accurately between teacher experience, selectivity of the college they attended, content knowledge, and other characteristics because “many of the measures of teachers’ qualifications are highly correlated with each other.”  In short, the Boyd, et al study is hardly the “strong research” in support of their standards that NCTQ claims it is.

Do we need more examples of how NCTQ misinterprets or stretches research to claim that their standards are supported by “strong research”?  Oh, how about one more…  Standard 13 is “Equity” and judges teacher prep programs based on whether “The program ensures that teacher candidates experience schools that are successful in serving students who have been traditionally underserved.”  The “strong research ” NCTQ cites for support of the claim “that entering teachers learn crucial methods of instruction and management through observation of and supervised practice in schools where staff are successfully teaching students living in poverty” is a piece by Ronfeldt.

Unfortunately, Ronfeldt’s study appears to make the opposite claim.  He finds that it is more important for student teachers to be trained in schools with low staff turnover that tend to have more advantaged students.  He concludes:

Should we place student teachers in “difficult-to-staff, underserved” schools to learn to teach? The main
findings of this study suggest otherwise – learning to teach in difficult-to-staff field placement schools is associated with lower teacher effectiveness and retention. Moreover, the results demonstrate that being trained in field placements with higher concentrations of poor, black, and lowest-achieving students has no significant effect on teacher retention or effectiveness.

I haven’t see this much unreliable citation of research since I read teacher union reports.

To be fair, NCTQ acknowledges that quality research on effective education practices is in short supply: “To the extent that high-quality research can inform how teachers should be prepared, NCTQ uses that research to formulate standards. Unfortunately, research in education that connects preparation practices to teacher effectiveness is both limited and spotty.”  But this lack of evidence does not prevent NCTQ from confidently declaring that they know what teacher prep programs should be doing and judging them on that basis.  If quality research is so limited, how does NCTQ know what everyone else is supposed to be doing?

And I’m sure that there is considerable room for improvement in teacher prep programs.  Many of NCTQ’s recommendations are probably sensible, even if they aren’t backed by “strong research.”  The problem is not so much that NCTQ is suggesting bad ideas as that they are claiming to know much more than they actually know.  And they are willing to boss around everyone else despite not knowing as much as they think.

Maybe we’d make more progress in improving teacher prep programs if we were more upfront about what we didn’t know and encouraged more experimentation and data-collection so that we can learn more.  And given that different circumstances may call for different practices, maybe we should be open to a variety of Ed School approaches rather than attempting to impose the one true way.


10 Responses to The Hubris of NCTQ’s Ed School Ratings

  1. […] expect, University of Arkansas's Jay P. Greene, one of the foremost Common Core foes has already trashed NCTQ's report for doing so (as well as for "hubris" in attempting to force ed schools toward one […]

  2. Greg Forster says:

    Maybe we’d make more progress in improving teacher prep programs if we were more upfront about what we didn’t know and encouraged more experimentation and data-collection so that we can learn more.

    But Jay, that would require us to permit, and even encourage, a diversity of approaches. How does that make us experts look smarter than everyone else?

  3. Jay,

    Just saw in this morning’s paper that a local school district super, Avossa, was an advisor on this NCTQ report. I know for a fact that almost as soon as he started he brought in Cambridge Education to do Quality Reviews on each of the schools. You may remember Cambridge was vital in foisting Michael Barber’s Global Citizenship vision on the UK and then Barber recommended them to Joel Klein for NYC in 2007. They are also involved in doing the MET Teacher Evaluations Gates is financing which use the Charlotte Danielson Framework (that dovetails nicely with the OBE Implementation Handbook she wrote back in the 80s).

    Every city I have read the Cambridge Quality Reviews for–Tampa, Charlotte, Fulton, NYC for starters, they come in and tell the teachers they cannot teach the content anymore. That they must use a child development,student-centered approach instead.

    I finally looked into the real definition of Quality and wrote a post explaining what Dewey envisioned as well as Cambridge and the accreditors.

    Everything I have read about the new moves in teacher accreditation are consistent with this definition of quality.

  4. […] or the ubiquitous U.S. News college rankings). Elsewhere, education scholar Jay Greene has raised fair concerns about the hubris of a group declaring what the “right” policies are when in actuality the […]

  5. Joy Pullmann says:

    So, Jay, would you say we just don’t know enough to rate education schools now, or would you recommend different/fewer criteria for doing so? And am I reading your last section right to say you agree that ed schools are generally awful?

  6. Hi Joy — I do not believe we know enough to tell ed schools what the right way to prepare teachers is. But I also suspect that we can never know enough to identify the one true way because there is no one true way. Just as there is no right way for all students to be educated (which is part of the case for choice), there is no right way to prepare teachers. We need choice and competition among ed schools just as we need it in K-12.

    And yes, I think many ed schools are academically weak. Some of this is because there is a cartel of accreditators and unions restricting the range of methods by which people can become teachers. Some of this is because K-12 schools are not sufficiently motivated by choice and competition to hire only the best trained teachers.

    But the wrong way to fix this is to try to impose on all ed schools a particular approach to preparing teachers.

  7. […] researchers across the ideological spectrum, including Linda Darling-Hammond, Bruce Baker, and Jay Greene, criticize the report for methodological flaws. One issue is the use of “document review” […]

  8. […] on teacher preparation programs in U.S. News is another good example of this, as well noted by Jay Greene.  There are things in that report that ring true to me (hardly anyone does a good job of preparing […]

  9. […] Council on Teacher Quality–I’m not a huge fan of Jay Greene, but his takedown of NCTQ’s ed school ratings was perfect. These guys aren’t just boring and predictable, […]

  10. […] out ranking ed schools on whether they meet NCTQ’s ideas of what makes ed schools effective. As I pointed out last year, NCTQ purports to have a strong research basis for claiming that ed schools should adhere to their […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s