By John Sener
The latest round of national self-flagellation generated by the recent announcement of the latest round of PISA international test scores (and neatly packaged by the US Department of Education as “PISA Day”1) has subsided for the most part. Unfortunately, the accompanying memes and their destructive power continue to reverberate through the American psyche, humming a depressingly familiar tune. Let’s listen more closely to what the song sounds like:
PISA Days are here again
Time to moan and gnash our teeth again?
Do our kids just stink at math again?
PISA days are here again…
Upon closer examination, the notion that we can determine much of anything useful about the state of American education or its economic impact through PISA or any other standardized test scores quickly falls apart like a wet answer sheet. As one critique has noted,2 standardized tests are less objective than is commonly believed: they don’t measure actual student achievement; their results are routinely distorted or misapplied for assessment purposes; they encourage students to become “superficial thinkers.” Instead, consider this list of traits from educational critic Gerald Bracey that we presumably want our learners to have: creativity, critical thinking, resilience, motivation, persistence, curiosity, endurance, reliability, enthusiasm, empathy, self-awareness, self-discipline, civic-mindedness, courage, compassion, resourcefulness, sense of beauty, sense of wonder.3 What do they have in common? Standardized tests measure NONE of them.
Still, as a recent New York Times editorial4 bemoaned, don’t PISA results demonstrate that US kids stink — er, “lag behind” in math, and that “even [US] gifted students can’t keep up” with the rest of the world in math and science? There are several answers to this question, including: “No,” “So what?” and “That’s actually a good thing.” One reason is that PISA results purport to represent the achievement of all American 15-year-old students (on a low-stakes, largely meaningless snapshot of an event, but that’s another line of critique). In reality, analyses of math test results that take socioeconomic factors into account have found that US students are doing just fine5 or even scoring near the top.6 Not that the results really matter; as another report noted, the US (or any large country for that matter) doesn’t need all of its students doing math and science at a high level, but rather only a “sufficient number of highly educated workers.”7 So in fact, any STEM workforce shortage could be met (at least in theory) by adopting one of the NYT editorial’s proposed solutions: more focus on boosting achievement of gifted students in math and science (which could, among other things, drag down the US’s overall PISA scores). Indeed, this may be a good thing, for as University of Oregon researcher Yong Zhao has found, countries with high PISA math scores actually score lower on measures of “perceived entrepreneurial capabilities.”8 As Zhao argues in his book World Class Learners: Educating Creative and Entrepreneurial Students,9 teaching students to think like entrepreneurs (resourceful, flexible, creative, and global) is different from preparing students to be homogenous, compliant, and standardized workers for mass employment (and guess what’s excellent preparation for that outcome?). Nevertheless, the chorus bemoaning US PISA test results remains loud:
Altogether shout it now
There is no one
Who can doubt it now
Chinese, Poles & Finns will rout us now
PISA days are here again
Actually, there are all sorts of reasons to doubt PISA results. As the Brookings Institution’s Tom Loveless has pointed out,10 the PISA results for China are heavily skewed since the Chinese government only allowed the release of results for Shanghai, an area which is highly unrepresentative of the country as a whole. Yet that hasn’t stopped the major media outlets from shouting out the results, for example “China: The World’s Most Clever Country?”11 (BBC), “US Teens Lag in Global Education Rankings as Asian Countries Rise to the Top”12 (USNews/NBC News), and “U.S. Students Get Stuck in Middle of the Pack on OECD Test”13 (Bloomberg).
As for the Poles, we can see how their PISA prowess (they’ve been kicking the US’s PISA ranking butt since 2006 now14) has already translated into having Europe’s most dynamic economy in 201315 — surely the result of their improved education system? Or not — as the article notes, “The story of the Polish miracle is a testament to the importance of prudent policymaking — but it’s just as much about luck.”
Perhaps the most revealing PISA results are those from Finland, which has dropped from its previous top-ranked perch to 5th in science, 6th in reading, and 12th in math. As Education Week’s Rick Hess has noted,16 it’s silly to think that Finland’s schools “fell apart between 2009 and 2012” — just as silly as it was to think that “Finland had cracked the code of educational excellence.” As this post-mortem of Finland’s fall from PISA grace17 illustrates, the actual picture is more complicated. The variety of explanations offered — other countries got better, Finland got complacent, they were never really that good to begin with, Finnish students “have forgotten how to work” — demonstrates how PISA and other standardized tests are really more Rohrschach than rigorous assessment.
To their credit, many major news outlets are also starting to report on how the results are being questioned18 and are calling for a somewhat more nuanced interpretation of the results.19 Even here, however, most everyone seems to buy in to the notion that the PISA results mean something really important for the US, and that the news is not good:
Our country’s future is gone
Unless we can be number one —
Obsession with the US’s PISA rankings echoes the American craze over college rankings, or for that matter BCS football rankings, where being near the top is all that matters, and being Number One is the ultimate aim. Conveniently ignored are two not-so-minor details: there is no demonstrable association between national test scores and national success20 and the US has never scored number one or even near the top of international assessments.21 This hasn’t stopped US Secretary of Education Arne Duncan from proclaiming the “brutal truth” that relative changes in countries’ PISA rankings represents a “threat of educational stagnation and complacency,” showing that “our students are basically losing ground” and “running in place, as other high-performing countries start to lap us,” and reflecting a “reality [that] is at odds with our aspiration to have the best-educated, most competitive workforce in the world.”22
In reality, the use of rankings to compare countries encourages a false precision about their meaning. Danish statistician Svend Kreiner from the University of Copenhagen has written a paper that shows how PISA rankings fluctuate wildly depending on which test questions are used to compile the results.23 In the 2006 reading rankings, Canada could have been positioned anywhere between 2nd and 25th, Japan between 8th and 40th, and the UK between 14th and 30th because PISA uses an “inappropriate” mathematical model to scale up its results. As Professor Kreiner noted, “It is meaningless to try to compare reading in Chinese with reading in Danish” because of the varying levels of difficulty among languages.
Instead, the use of comparative international rankings seems to have the opposite effect: an inevitable deluge of sloppy, and ultimately shamefully imprecise, thinking. So, for instance, the aforementioned New York Times editorial equates performance on standardized tests with national innovation, as do Secretary Duncan’s aforementioned remarks: “Applied math and science skills of the kind measured by PISA are essential to propelling innovation and maintaining international competitiveness.” Almost makes you long for the old days when giants of innovation such as Bill Gates, Steve Jobs, and their like wowed us with their PISA test scores, doesn’t it? Except, of course, that we know nothing about their PISA scores. In fact, the odds are that they didn’t even take the test. Which is as it should be — since we know that innovation comes from a different source and is not something that tests like PISA are even designed to measure.
PISA days are here again
The cries for change are clear again
So, let’s test our kids to death again
Secretary Duncan’s remarks also included a not-so-implicit call for college entrance exams like those in Singapore, China, Germany, and other “high-performing countries.” Just what we need — even more tests and assessments! Because apparently we can never weigh the calf often enough; as I found while researching my book, one Milwaukee teacher reported having to give over 150 assessments during the 2008-09 school year — to her kindergarten students.24
Recently I attended an event where a reporter talked about her experiences covering local education issues. The audience discussion made it abundantly clear that test scores have become our societal shorthand for educational crisis. It was as if the populace is under a collective spell — low test scores are bad, high test scores will fix things.
At least the aforementioned New York Times editorial proposes different solutions — more government money for gifted students, “accelerated learning” (aka more Advanced Placement courses), early college admission, even “psychological testing” — anything to keep academically challenging the “brightest students” before they “lose steam and check out” — as if the problem were still producing and stoking “little engines that could” — improve their test scores, that is.
So get used to hearing this song over, and over, and over — despite the fact that one could fill books (for instance this one25) that analyze the shortcomings of standardized tests. Yet the most curious thing about this situation is not that people believe in the power of test scores despite all rational argument to the contrary. There is clearly something deeper going on — on a more emotional level, perhaps. More on that in the next post…
1 PISA Day: Learning Beyond the Rankings website.
2 Valerie Strauss, The Myths of Standardized Testing. Washington Post, 4/15/11.
3 Gerald Bracey, No Child Left Behind: A Threat to National Security. Huff Post, 4/9/07.
4 New York Times editorial, Even Gifted Students Can’t Keep Up: In Math and Science, The Best Fend for Themselves. New York Times, 12/14/13.
5 Diane Ravitch, Stephen Krashen: Our PISA Scores are Just Right. Diane Ravitch’s Blog, 12/9/12.
6 Brandon Weber, Five Myths about Our Schools That Fall Apart When You Look Closer. Upworthy, 12/3/13. [starting at 1:18]
7 Richard Rothstein and Mark Conroy, PISA Day — An Ideological and Hyperventilated Exercise. The Economic Policy Institute blog, 12/1/13.
8 Yang Zhao, Numbers Can Lie: What TIMSS and PISA Truly Tell Us, If Anything? Education in the Age of Globalization blog, 12/11/12.
9 Yang Zhao, World Class Learners. Education in the Age of Globalization blog, 2012 (n.d.).
10 Thomas Loveless, PISA’s China Problem. Brookings Institution blog, 10/9/13.
11 Sean Coughlan, China: The World’s Most Clever Country? BBC News Business, 5/8/12.
12 Daniel Arkin, US Teens Lag in Global Education Rankings as Asian Countries Rise to the Top. NBCNEWS/AP, 12/3/13.
13 Oliver Staley, U.S. Students Get Stuck in Middle of the Pack on OECD Test. Bloomberg, 12/3/13.
14 Programme for International Student Assessment, Wikipedia, last modified 12/15/13.
15 Stephan Faris, How Poland Became Europe’s Most Dynamic Economy. Bloomberg BusinessWeek, 11/27/13.
16 Rick Hess, 7 Reasons I Don’t Care about the PISA Results. Education Week, 12/4/13.
17 Adam Taylor, Finland Used to Have the Best Education System in the World — What Happened? Business Insider, 12/3/13.
18 Valerie Strauss, Four Lessons on New PISA Scores — Ravitch. Washington Post, 12/3/13.
19 Los Angeles Times editorial, Are America’s Students Falling Behind the Rest of the World? Los Angeles Times, 12/9/13.
20 Keith Baker, Are International Test Scores Worth Anything? Phi Delta Kappan, October 2007.
21 Diane Ravitch, My View of the PISA Scores. Diane Ravitch’s Blog, 12/3/13.
22 Arne Duncan, The Threat of Educational Stagnation and Complacency. US Department of Education, 12/3/13.
23 William Stewart, PISA Rankings Are ‘Utterly Wrong.’ TES Connect, 7/19/13.
24 John Sener, The Seven Futures of American Education (2012), p.78.
25 Philip Harris, B.M. Smith, J. Harris, The Myths of Standardized Testing: Why They Don’t Tell You What You Think They Do, Rowman & Littlefield (2011).
Filed under: Uncategorized |