What do you see in this picture? The new PISA results are out and education charlatans of every stripe are finding proof of their own preferred policy solution.
Dennis Van Roekel of the National Education Association sees addressing poverty as the solution: “The United States’ standings haven’t improved dramatically because we as a nation haven’t addressed the main cause of our mediocre PISA performance — the effects of poverty on students.” There is some evidence for this, but the OECD analysis finds that student socioeconomic status only explains 15% of the variance in test results. And according to the Wall Street Journal coverage, “Jack Buckley, the commissioner of the National Center for Education Statistics, noted that American students from families with incomes in the highest quartile did not perform as well as students with similar backgrounds in other countries.
According to the San Jose Mercury News, “‘None of the top-tier countries,’ said Randi Weingarten head of the American Federation of Teachers, ‘nor any of those that have made great leaps in student performance, like Poland and Germany, has a fixation on testing like the United States does.'” Except many of the big gainers, particularly in Asia, do fixate on testing.
Best Practices guru, Marc Tucker, was on NPR this morning saying something about how “what you will find among the top performers” is that they”provide more resources to kids who are harder to educate than kids who are easier to educate.” But in the San Jose Mercury News Tucker seems to suggest that more resources is not the solution when he asks, “Why are we not getting more bang for the buck?” And on NPR Tucker credited Singapore’s success to “not just more teachers but better teachers.” But the Wall Street Journal cites the OECD analysis, saying it “found a low connection between class size and test scores.” And in the country Tautology Land better teachers are the ones who produce better scores.
It is possible to do credible social scientific analyses of international test scores if you do something like a regression that systematically examines variation in performance within and across countries controlling for other variables. See for example work by Ludger Woessmann. But just eyeballing the top performers and making up stories about why they succeeded based on picking and choosing characteristics about them is pure quackery. As I’ve said before, best practices are the worst.
So, reach for your Duck Dynasty duck quacker and watch as folks make up stories about the picture above. Personally, I see a cute little dog.
Jay P. Greene's Blog 
I think all the commentary mostly points to low levels of statistical literacy of the media combined with their thirst for a good story. Russ Whitehurst’s critique of the Union City story is a larger tale about looking for causes while not understanding (or not wanting to use) elementary statistics. Causal sources of education outcomes will be complex when looked at across nations. But that’s a bad story.
Buried somewhere is the curious finding that America did fine on math in the TIMSS, especially after individual American states are taken out of the rankings (the US placed 12th in math in eighth grade, but 4 of the 11 higher ranking systems were Massachusetts, Minnesota, North Carolina, and Indiana). Maybe someone could reconcile these two.
Three things come immediately to mind (and I know this procedure won’t satisfy anyone’s thirst for rigourous social science): (1) the comparator jurisdictions are generally weaker in TIMSS than in PISA (no Shanghai in TIMSS, for example); (2) TIMSS, being centred at Boston College, is closer to U.S. exam practices than is PISA, centred at the OECD in Paris, which requires higher level thinking, especially application of knowledge; (3) TIMSS tests younger pupils (eighth grade as opposed to 15-year-olds, who will generally be in tenth or eleventh grade) — combined with the PIAAC data, the implication is that the longer our students remain in American state schools while foreign children remain in their schools, the farther behind American children fall, with the disturbing outcome that America’s young people comprise the least competent workforce to be found in any developed country.
I think you have “Stacked cards” to reach your conclusion that America’s young people are the least competent workforce. There is absolutely nothing in the data that would prove that. I believe the high school students show up differently because the CURRICULUM is different in these schools across the globe… Also, who is choosing the samples of the students who will take these tests and what courses have they had. I taught in MA schools for over 40 years and I never knew even one pupil who took these tests… maybe they restrict their sample (intentionally or perhaps unknowingly?) Samples of students, different texts/curriculum/programs means you are comparing apples and oranges. But no matter what you compare you cannot generalize to the statement about the competence of the workforce…. there is no predictive validity in any of these tests… even if there were it would be predicting what “some people on panels” believe to be important (using the standard setting but panels many of which do not contain any teachers like myself with the quality of experiences I have in Massachusetts.)
Jean, you obviously didn’t read about the PIAAC (Programme for the International Assessment of Adult Competencies, administered by the OECD — a companion test for PISA) results that came out two months ago. Read pages 75, 85, and 95 of the main report, in addition to all of the sampling methodology you’re interested in, and then reply with your questions or comments or both.
Thanks Jay. I have Tucker’s book and found it weak. The local regional metropolitan planning authority just released its economic competitiveness strategy obligating all school districts in the 10 county ATL area to “integrate best practices and innovative new programs into PreK-12 classrooms throughout the region.”
Forcing everyone to adopt the National Center on Education and the Economy’s focus.
Yeah that kind of mind arson will do wonders for true competitiveness.
I wish when people would identify these “authors” they would point out the ideology behind their views…quote: “in which Tucker stated that “the United States will have to largely abandon the beloved emblem of American education: local control.” He also dared write: “I propose to greatly strengthen the role of the state education agencies in education governance, at the expense of local control … [G]overnance roles of the local districts, as well as the federal government, would be significantly decreased. Independent citizen governing boards would be eliminated.”
The issue of local control is an experienced value in Massachusetts; yes, we have our faults… county systems such as Virginia operates would possibly offer some economies of scale. But, for whatever value you chooser (local , etc) there are tradeoffs to be made. It is important to know what a person like Tucker values before accepting all his viewpoints…. We need more realistic political systems that explain why we can’t get along and everything is polarized. I might share 8 values with a libertarian but when it comes to two other issues we totally disagree….. It is important to know what is valued (such as local control in Massachusetts) and accept that we cannot reach “Perfection” so there will be shortcomings and failures in whatever system is implemented. I think this is the “rorschach” that we need to investigate further and ignore the media pundits.
Reblogged this on The Echo Chamber.
[…] that being said, it’s also important to heed Jay Greene’s valuable prescription not to “impose a preferred policy prescription” based on which nations are registering what PISA scores. Better to stick with the statistically […]
Well, I was right. Exporting our children to Estonia for their K-12 education would have been good for students and taxpayers.
is this meant to be fatuous? do I detect irony? or should I read your comment literally
[…] course not. As Jay Greene so helpfully points out as everyone scrambles to cherry-pick data to press their agendas, just “eyeballing” […]
read Jay’s earlier articles on “selection on the dependent variable”….
The Israeli gains are obviously correlated with more enriched Iranian uranium. Makes kids study harder.
Or perhaps it was the positive motivation of wanting to do well in the year of Thanksgivukah.
Clearly these gains are a result of the increased geopolitical power of Israel from the commercialization of hydraulic fracturing by Al Copeland laureate George P. Mitchell.
The circle is now complete!
According to the Israeli press children who were tested in hebrew had a math results of 489 while children who were tested in Arabic scored 388 . That means that the gap between Arabs and Jews is 1 SD (100 point).
Consider that Ashkenazim/Mizrahim ratio among Hebrew testers is 50/50 , and that Ashkenazim typically score 0.75 SD (75 point) then Mizrahim (e.g on the Israeli SAT) , then Ashkenazim score in math is about : 489 + 75 * 0.5 ~ 521 .
Arabs declaring war on Israel and the US for that. …..
Reblogged this on This Got My Attention and commented:
Progress for some and others, including the USA, regress. The US throws money hand over fist at education and decline is what we have to show for it.
Nothing will help children learn as long as they live in broken homes, are dragged from pillar to post and made to be the one who tries to keep peace between parents. Child abuse must be addressed. No child can learn until he lives in a safe environment.
[…] That depends, of course, on what you think was really behind the the improvement. As this great blog post by Jay Greene (h/t Ben Wildavsky) points out, the release of these results every three years or so […]
[…] what we see in the PISA policy Rorschach test. Reject the latest effort to centralize standards and assessments and empower families with […]
[…] what we see in the PISA policy Rorschach test. Reject the latest effort to centralize standards and assessments and empower families with […]
[…] what we see in the PISA policy Rorschach test. Reject the latest effort to centralize standards and assessments and empower families with […]
[…] what we see in the PISA policy Rorschach test. Reject the latest effort to centralize standards and assessments and empower families with […]
[…] what we see in the PISA policy Rorschach test. Reject the latest effort to centralize standards and assessments and empower families with […]
[…] https://jaypgreene.com/2013/12/03/let-the-best-practices-rorschach-test-begin/ […]