Jay P. Greene's Blog

One of the bigger problems in education policy is hubris.  People regularly claim that they know what the right policies or practices are, and things would be better if only others would bend to their will.  The truth is that we know relatively little about effective education policies and practices.  This isn’t for lack of trying.  Despite considerable research effort and policy inquiry, we’ve found remarkably few “universal truths” about effective education.  Part of the difficulty is that knowing what works presupposes that there is a single, best way.  But  it appears much of what is effective in education is contingent on particular needs and circumstances and does not lend itself to broad declarations about the “right”  practices and policies.

Because the scourge of PLDD is endemic, however, we continue to hear claims that “We know what works.”   This was the traditional refrain of teacher union leaders, but now reformers have joined the hubris chorus.  The latest example of this is the ratings of Ed Schools issued by the National Council of Teacher Quality.  NCTQ claims to know what good teacher preparation programs should be doing and judges those programs against NCTQ’s vision of effective practices.

In particular, NCTQ identifies 18 standards by which it judges Ed Schools.  “Our standards for the first edition of the Teacher Prep Review” NCTQ assures us, ” are based on research; internal and external expert panels; the best practices of other nations and the states with the highest performing students; and, most importantly, what superintendents and principals around the country tell us they look for in the new teachers they hire.”

NCTQ describes the research basis for their standards in a lengthy document.  Yet, even according to their own description only 8 of the 18 standards are supported by “strong research.”  And in most of the 8 cases where they do claim to have strong research support, the research does not actually provide them with the strong support they assert.

For example, the “Early Reading” standard assesses whether “The program trains teacher candidates to teach reading as prescribed by the Common Core State Standards.”  None of the studies they cite actually examine the specific standard since none specifically examine what methods of teaching reading, if any, are actually prescribed by Common Core.  As is the case with all 18 standards in the NCTQ rating system, one has to make a series of leaps between the research cited and the actual standard being used to judge teacher prep programs.

In the case of early reading, the “strong research” they cite examines whether teachers are familiar with the “five components of effective reading instruction,” and whether teachers who are certified and have masters degrees are more likely to know those five components.  It turns out teachers are generally not familiar with the five components and are no more likely to know them if they are certified or have a masters.  That’s all very nice, but isn’t the “strong research” supporting the standard supposed to show that knowledge of the five components, which presumably have something to do with teaching “reading as prescribed by the Common Core State Standards,” actually lead to improved reading by students?  The strong research cited by NCTQ says it generally doesn’t: “This study also found no relationship between teachers’ knowledge of these components and their students’ reading growth – with the notable exception of third-grade students.”  This is typical of the “strong research” supporting 8 of the 18 standards by which NCTQ judges Ed Schools.

Standards 1 and 6 address whether teacher prep programs select “teacher candidates of strong academic caliber” and whether “teacher candidates have the broad content preparation necessary to successfully teach to the Common Core State Standards.”  In both cases the “strong research” on which these standards rely is a study by Boyd, et al examining the relationship between teacher characteristics and student achievement.  Let’s leave aside the fact that NCTQ acknowledges that research by Harris and Sass as well as Chingos and Peterson contradict their standard.  Even the Boyd, et al study they do cite does not specifically demonstrate that teachers from more selective programs or with more content training are more effective.  First, Boyd, et al are careful not to make the type of strong causal claims from their work that NCTQ does:

It is not easy to estimate how the achievement gains of students are affected by the qualifications of their teachers because teachers are not randomly sorted into classrooms. For example, if teachers in schools in which students perform best in math are more likely to be certified in math, one might be tempted to conclude that being certified to teach math contributes to higher student achievement. The causal relationship, however, may operate in the other direction; that is, more qualified teachers may be in schools where students perform well in math because they prefer to teach good students and because employers want to staff their courses with in-field certified teachers. Analysts need to be careful not to attribute the test-score gains associated with sorting to the attributes of teachers.

Beyond the fact that Boyd, et al would not make the strong causal claims from their work that NCTQ feels free to do, the Boyd, et al study examines a basket of teacher qualifications and does not claim to be able to distinguish accurately between teacher experience, selectivity of the college they attended, content knowledge, and other characteristics because “many of the measures of teachers’ qualifications are highly correlated with each other.”  In short, the Boyd, et al study is hardly the “strong research” in support of their standards that NCTQ claims it is.

Do we need more examples of how NCTQ misinterprets or stretches research to claim that their standards are supported by “strong research”?  Oh, how about one more…  Standard 13 is “Equity” and judges teacher prep programs based on whether “The program ensures that teacher candidates experience schools that are successful in serving students who have been traditionally underserved.”  The “strong research ” NCTQ cites for support of the claim “that entering teachers learn crucial methods of instruction and management through observation of and supervised practice in schools where staff are successfully teaching students living in poverty” is a piece by Ronfeldt.

Unfortunately, Ronfeldt’s study appears to make the opposite claim.  He finds that it is more important for student teachers to be trained in schools with low staff turnover that tend to have more advantaged students.  He concludes:

Should we place student teachers in “difficult-to-staff, underserved” schools to learn to teach? The main
findings of this study suggest otherwise – learning to teach in difficult-to-staff field placement schools is associated with lower teacher effectiveness and retention. Moreover, the results demonstrate that being trained in field placements with higher concentrations of poor, black, and lowest-achieving students has no significant effect on teacher retention or effectiveness.

I haven’t see this much unreliable citation of research since I read teacher union reports.

To be fair, NCTQ acknowledges that quality research on effective education practices is in short supply: “To the extent that high-quality research can inform how teachers should be prepared, NCTQ uses that research to formulate standards. Unfortunately, research in education that connects preparation practices to teacher effectiveness is both limited and spotty.”  But this lack of evidence does not prevent NCTQ from confidently declaring that they know what teacher prep programs should be doing and judging them on that basis.  If quality research is so limited, how does NCTQ know what everyone else is supposed to be doing?

And I’m sure that there is considerable room for improvement in teacher prep programs.  Many of NCTQ’s recommendations are probably sensible, even if they aren’t backed by “strong research.”  The problem is not so much that NCTQ is suggesting bad ideas as that they are claiming to know much more than they actually know.  And they are willing to boss around everyone else despite not knowing as much as they think.

Maybe we’d make more progress in improving teacher prep programs if we were more upfront about what we didn’t know and encouraged more experimentation and data-collection so that we can learn more.  And given that different circumstances may call for different practices, maybe we should be open to a variety of Ed School approaches rather than attempting to impose the one true way.

This entry was posted on Tuesday, June 18th, 2013 at 9:32 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.