By John Sener
[Note: This is the second in a three-part series, “PISA Days Are Here Again.” See part 1 and 3.]
The latest round of national self-flagellation generated by the recent announcement of the latest round of PISA international test scores (and neatly packaged by the US Department of Education as “PISA Day”1) has subsided for the most part. Unfortunately, the accompanying memes and their destructive power continue to reverberate through the American psyche, humming a depressingly familiar tune. Let’s listen more closely to what the song sounds like:
PISA Days are here again
Time to moan and gnash our teeth again?
Do our kids just stink at math again?
PISA days are here again…
Upon closer examination, the notion that we can determine much of anything useful about the state of American education or its economic impact through PISA or any other standardized test scores quickly falls apart like a wet answer sheet. As one critique has noted,2 standardized tests are less objective than is commonly believed: they don’t measure actual student achievement; their results are routinely distorted or misapplied for assessment purposes; they encourage students to become “superficial thinkers.” Instead, consider this list of traits from educational critic Gerald Bracey that we presumably want our learners to have: creativity, critical thinking, resilience, motivation, persistence, curiosity, endurance, reliability, enthusiasm, empathy, self-awareness, self-discipline, civic-mindedness, courage, compassion, resourcefulness, sense of beauty, sense of wonder.3 What do they have in common? Standardized tests measure NONE of them.
Still, as a recent New York Times editorial4 bemoaned, don’t PISA results demonstrate that US kids stink — er, “lag behind” in math, and that “even [US] gifted students can’t keep up” with the rest of the world in math and science? There are several answers to this question, including: “No,” “So what?” and “That’s actually a good thing.” One reason is that PISA results purport to represent the achievement of all American 15-year-old students (on a low-stakes, largely meaningless snapshot of an event, but that’s another line of critique). In reality, analyses of math test results that take socioeconomic factors into account have found that US students are doing just fine5 or even scoring near the top.6 Not that the results really matter; as another report noted, the US (or any large country for that matter) doesn’t need all of its students doing math and science at a high level, but rather only a “sufficient number of highly educated workers.”7 So in fact, any STEM workforce shortage could be met (at least in theory) by adopting one of the NYT editorial’s proposed solutions: more focus on boosting achievement of gifted students in math and science (which could, among other things, drag down the US’s overall PISA scores). Indeed, this may be a good thing, for as University of Oregon researcher Yong Zhao has found, countries with high PISA math scores actually score lower on measures of “perceived entrepreneurial capabilities.”8 As Zhao argues in his book World Class Learners: Educating Creative and Entrepreneurial Students,9 teaching students to think like entrepreneurs (resourceful, flexible, creative, and global) is different from preparing students to be homogenous, compliant, and standardized workers for mass employment (and guess what’s excellent preparation for that outcome?). Nevertheless, the chorus bemoaning US PISA test results remains loud:
Altogether shout it now
There is no one
Who can doubt it now
Chinese, Poles & Finns will rout us now
PISA days are here again
Actually, there are all sorts of reasons to doubt PISA results. As the Brookings Institution’s Tom Loveless has pointed out,10 the PISA results for China are heavily skewed since the Chinese government only allowed the release of results for Shanghai, an area which is highly unrepresentative of the country as a whole. Yet that hasn’t stopped the major media outlets from shouting out the results, for example “China: The World’s Most Clever Country?”11 (BBC), “US Teens Lag in Global Education Rankings as Asian Countries Rise to the Top”12 (USNews/NBC News), and “U.S. Students Get Stuck in Middle of the Pack on OECD Test”13 (Bloomberg).
As for the Poles, we can see how their PISA prowess (they’ve been kicking the US’s PISA ranking butt since 2006 now14) has already translated into having Europe’s most dynamic economy in 201315 — surely the result of their improved education system? Or not — as the article notes, “The story of the Polish miracle is a testament to the importance of prudent policymaking — but it’s just as much about luck.”
Perhaps the most revealing PISA results are those from Finland, which has dropped from its previous top-ranked perch to 5th in science, 6th in reading, and 12th in math. As Education Week’s Rick Hess has noted,16 it’s silly to think that Finland’s schools “fell apart between 2009 and 2012” — just as silly as it was to think that “Finland had cracked the code of educational excellence.” As this post-mortem of Finland’s fall from PISA grace17 illustrates, the actual picture is more complicated. The variety of explanations offered — other countries got better, Finland got complacent, they were never really that good to begin with, Finnish students “have forgotten how to work” — demonstrates how PISA and other standardized tests are really more Rohrschach than rigorous assessment.
To their credit, many major news outlets are also starting to report on how the results are being questioned18 and are calling for a somewhat more nuanced interpretation of the results.19 Even here, however, most everyone seems to buy in to the notion that the PISA results mean something really important for the US, and that the news is not good:
Our country’s future is gone
Unless we can be number one —
Obsession with the US’s PISA rankings echoes the American craze over college rankings, or for that matter BCS football rankings, where being near the top is all that matters, and being Number One is the ultimate aim. Conveniently ignored are two not-so-minor details: there is no demonstrable association between national test scores and national success20 and the US has never scored number one or even near the top of international assessments.21 This hasn’t stopped US Secretary of Education Arne Duncan from proclaiming the “brutal truth” that relative changes in countries’ PISA rankings represents a “threat of educational stagnation and complacency,” showing that “our students are basically losing ground” and “running in place, as other high-performing countries start to lap us,” and reflecting a “reality [that] is at odds with our aspiration to have the best-educated, most competitive workforce in the world.”22
In reality, the use of rankings to compare countries encourages a false precision about their meaning. Danish statistician Svend Kreiner from the University of Copenhagen has written a paper that shows how PISA rankings fluctuate wildly depending on which test questions are used to compile the results.23 In the 2006 reading rankings, Canada could have been positioned anywhere between 2nd and 25th, Japan between 8th and 40th, and the UK between 14th and 30th because PISA uses an “inappropriate” mathematical model to scale up its results. As Professor Kreiner noted, “It is meaningless to try to compare reading in Chinese with reading in Danish” because of the varying levels of difficulty among languages.
Instead, the use of comparative international rankings seems to have the opposite effect: an inevitable deluge of sloppy, and ultimately shamefully imprecise, thinking. So, for instance, the aforementioned New York Times editorial equates performance on standardized tests with national innovation, as do Secretary Duncan’s aforementioned remarks: “Applied math and science skills of the kind measured by PISA are essential to propelling innovation and maintaining international competitiveness.” Almost makes you long for the old days when giants of innovation such as Bill Gates, Steve Jobs, and their like wowed us with their PISA test scores, doesn’t it? Except, of course, that we know nothing about their PISA scores. In fact, the odds are that they didn’t even take the test. Which is as it should be — since we know that innovation comes from a different source and is not something that tests like PISA are even designed to measure.
PISA days are here again
The cries for change are clear again
So, let’s test our kids to death again
PISA Days…are…here…a…gain!
Secretary Duncan’s remarks also included a not-so-implicit call for college entrance exams like those in Singapore, China, Germany, and other “high-performing countries.” Just what we need — even more tests and assessments! Because apparently we can never weigh the calf often enough; as I found while researching my book, one Milwaukee teacher reported having to give over 150 assessments during the 2008-09 school year — to her kindergarten students.24
Recently I attended an event where a reporter talked about her experiences covering local education issues. The audience discussion made it abundantly clear that test scores have become our societal shorthand for educational crisis. It was as if the populace is under a collective spell — low test scores are bad, high test scores will fix things.
At least the aforementioned New York Times editorial proposes different solutions — more government money for gifted students, “accelerated learning” (aka more Advanced Placement courses), early college admission, even “psychological testing” — anything to keep academically challenging the “brightest students” before they “lose steam and check out” — as if the problem were still producing and stoking “little engines that could” — improve their test scores, that is.
So get used to hearing this song over, and over, and over — despite the fact that one could fill books (for instance this one25) that analyze the shortcomings of standardized tests. Yet the most curious thing about this situation is not that people believe in the power of test scores despite all rational argument to the contrary. There is clearly something deeper going on — on a more emotional level, perhaps. More on that in the next post…
__________
1 PISA Day: Learning Beyond the Rankings website.
2 Valerie Strauss, The Myths of Standardized Testing. Washington Post, 4/15/11.
3 Gerald Bracey, No Child Left Behind: A Threat to National Security. Huff Post, 4/9/07.
4 New York Times editorial, Even Gifted Students Can’t Keep Up: In Math and Science, The Best Fend for Themselves. New York Times, 12/14/13.
5 Diane Ravitch, Stephen Krashen: Our PISA Scores are Just Right. Diane Ravitch’s Blog, 12/9/12.
6 Brandon Weber, Five Myths about Our Schools That Fall Apart When You Look Closer. Upworthy, 12/3/13. [starting at 1:18]
7 Richard Rothstein and Mark Conroy, PISA Day — An Ideological and Hyperventilated Exercise. The Economic Policy Institute blog, 12/1/13.
8 Yang Zhao, Numbers Can Lie: What TIMSS and PISA Truly Tell Us, If Anything? Education in the Age of Globalization blog, 12/11/12.
9 Yang Zhao, World Class Learners. Education in the Age of Globalization blog, 2012 (n.d.).
10 Thomas Loveless, PISA’s China Problem. Brookings Institution blog, 10/9/13.
11 Sean Coughlan, China: The World’s Most Clever Country? BBC News Business, 5/8/12.
12 Daniel Arkin, US Teens Lag in Global Education Rankings as Asian Countries Rise to the Top. NBCNEWS/AP, 12/3/13.
13 Oliver Staley, U.S. Students Get Stuck in Middle of the Pack on OECD Test. Bloomberg, 12/3/13.
14 Programme for International Student Assessment, Wikipedia, last modified 12/15/13.
15 Stephan Faris, How Poland Became Europe’s Most Dynamic Economy. Bloomberg BusinessWeek, 11/27/13.
16 Rick Hess, 7 Reasons I Don’t Care about the PISA Results. Education Week, 12/4/13.
17 Adam Taylor, Finland Used to Have the Best Education System in the World — What Happened? Business Insider, 12/3/13.
18 Valerie Strauss, Four Lessons on New PISA Scores — Ravitch. Washington Post, 12/3/13.
19 Los Angeles Times editorial, Are America’s Students Falling Behind the Rest of the World? Los Angeles Times, 12/9/13.
20 Keith Baker, Are International Test Scores Worth Anything? Phi Delta Kappan, October 2007.
21 Diane Ravitch, My View of the PISA Scores. Diane Ravitch’s Blog, 12/3/13.
22 Arne Duncan, The Threat of Educational Stagnation and Complacency. US Department of Education, 12/3/13.
23 William Stewart, PISA Rankings Are ‘Utterly Wrong.’ TES Connect, 7/19/13.
24 John Sener, The Seven Futures of American Education (2012), p.78.
25 Philip Harris, B.M. Smith, J. Harris, The Myths of Standardized Testing: Why They Don’t Tell You What You Think They Do, Rowman & Littlefield (2011).
Filed under: Uncategorized |
John Mark Walker: "If educational communities can continue to push platform integration and content portability, in the future, students may be able to design their own personalized degrees from smaller, modular chunks that cross institutional barriers" (
Richard Koubek
Judith McDaniel
Tim Fraser-Bumatay: "Although the format leaves us far-removed physically, the online forum has its own sense of intimacy" (Judith McDaniel, "
Ryan Kelly: "For me to be able to work with people clear across the country for an extended period of time opened me up to new things" (Judith McDaniel, "
Daniel Herrera: "As a Mexican American, I know that words of identity are powerful; so to discuss white privilege with my professor and classmates in a face-to-face class would have been terrifying and impossible" (Judith McDaniel, "
Cathy Gunn: "Traditional methods for effecting change at my institution aren’t getting us even to a trickle yet, let alone to thinking about or planning for a wave!" (
Educational Technology and Change Journal 
I don’t wish to detract from the overall thrust of the article but must point out that one comment, only peripherally related, does not fit.
“…any STEM workforce shortage could be met …”
Just as the gnashing of teeth over PISA is exaggerated so is the STEM workforce shortage. Much of this so-called shortage comes from companies lobbying for more H-1B visas. Their real motive is to suppress the wages of their technology workforce.
Some of the shortage is already removed due to off-shoring. At lower and middle levels of software and other remotely performable engineering jobs, companies can send their work to other countries. Even if productivity is lower, they save money because salaries are much less than half of those in the U.S. even after the middleman costs are considered. In simple terms, you can buy two programmers in India for less than the cost of one in the U.S.
However, remote management issues make it better to import the people instead of exporting the work. Thus, we see the push for more H-1B visas and cries of STEM workforce shortages.
The other part of this equation is the reluctance of today’s employers to train workers. When I graduated from college, OJT was common, often expected by both employer and employee. Despite skyrocketing profits, companies remain reluctant to train their new hires and seek those who require no training.
I’ll also echo John’s comments about the validity of the PISA tests for measuring the abilities of students to function well in today’s workplace. The most desirable skills are not measured. This fact does not mean that we’re training our students well, just that the PISA measurement doesn’t answer that question. To the extent that students with deeper understanding of a topic tend to outperform those who merely memorize it, it may act as a substitute. That extent is unclear. Still, PISA should not be totally ignored and swept aside. Data are always valuable even if those interpreting it come to different conclusions.
The most significant data from from the analysis of correlations with poverty. This high (negative) correlation illustrates that poor students receive a poor education, on average. Our somewhat steady absolute scores on PISA suggest that we’re not getting poorer as a nation, not a wonderful conclusion. Our decline in relative position indicates that other nations are gaining in wealth — if you take this particular correlation seriously. It’s not clear how much this all matters.
One thing is clear, and we don’t require any PISA results to figure it out. We are wasting lots of young minds by providing them with an inferior education. The scores from schools with less than 10% lunch assistance show that we can teach well (if you allow that PISA has any correlation with teaching excellence at all).
We “merely” have to spread this good teaching to all schools.
Instead of worrying about other countries, we must focus on our own and fix the problems we can see clearly. The results will come nicely.
Harry,
Thanks for your comments. Re the STEM shortage, many observers agree with you that H-1B visas are a significant factor but still insist that the shortage is real, for instance this article I found but did not include in the piece: http://www.qualitydigest.com/inside/quality-insider-article/us-stem-workforce-shortage-myth-or-reality.html. It’s on my to-do list to research this issue in more detail in the near future for a client. In the meantime, I’m withholding judgment.
Re your second comment, I’ll respond through the third and final article in this series (which I hope to finish shortly)…
John
The very acronym, STEM, has such a variety of meanings that a “STEM shortage” is hard to quantify. What is a “technology job”? Automotive repair has become one of late.
However, there is a more insidious side of this debate. The perceived shortage of STEM graduates is being used as a hammer to reshape K-12 curricula — yes, even kindergarten — by adding computer programming. These people insist on forcing software engineering on our tots because they claim that young people must learn these skills at a very early age and provide no hard statistics to back up these spurious claim.
I will provide just one counterexample, all that should be needed — myself. I learned computer programming on my own as a freshman at Caltech when I was seventeen years old, hardly anywhere near being in elementary or middle school. Much later, I made an excellent living from my skills for many years.
I had a friend some years after college who had been a defense industry engineer and wished to learn software. He absolutely could not comprehend what it was all about and ended up very frustrated by the entire effort. We should not force computer programming, no matter how nicely robed, on students. Computer programming, and more broadly software engineering, requires certain mental skills and personality traits.
A good computer programmer must be able to deal with the complex and abstract syntax and semantics of a programming language. This person must be able to visualize layers of logic and keep track of the flow of logic. The most minor errors cause failure meaning that programmers must be very focused on detail and be very precise. Extreme attention to detail is a personality trait not usually associated with the creativity and design capabilities required in software engineering. Could it be that a shortage of these sorts of people is just a natural part of the range and variety of humans? Might it be true that pushing more people into software will merely lower the quality and productivity of the overall set of programmers?
It may be hard to comprehend, but I have seen productivity ranges for software developers around 100-to-1. When I used to pay attention to these things, the industry norm for writing software was around ten tested, debugged, and commented lines of software per normal working day. My typical output was 100 lines, and I knew others similarly productive. On the other hand, it was not hard to find those with well below those ten lines per day.
Will we be doing our schools and companies a service by creating more of the few lines per day programmers? Realize that much of the reason for this low number is the time required to debug and fix erroneous code. Having spent two years fixing other people’s code, I can attest to how remarkably common errors are even in released code. (Of course, I find them in my own programs as well. In software as in life, no one is perfect!)
Just suppose that we did pump out many more low-level computer coders. Other countries are doing the same and very assiduously marketing them here in the U.S. at very low cost. Is it fair to train people without talent here to compete in a global market with those whose work can be sold at less than half the cost of a local worker? Especially when those foreign workers are probably more capable, and certainly more willing to work long, hard hours, than local workers?
Don’t get me wrong. The U.S. has produced the most high-quality software engineers of any country as I view the global landscape. However, asking lots more of our citizens to chase this dream when they’ll only end up frustrated makes little sense to me.
In this analysis, I have focused on computer software. STEM consists of much more. It’s just that the largest STEM gaps seem to be in software according to what I read. Let’s pick any other STEM discipline, for example chemical engineering. This is a truly tough area to learn with an incredible list of difficult courses to achieve a bachelor’s degree and an engineering certificate. I have some close second-hand experience here because that’s what my daughter did at MIT. She graduated with an A average.
The frustration with the job market at the time (1997) was intense. In essence, the only jobs she could find were laboratory technician jobs, the sort you’d expect for 2-year degrees, not for the exalted MIT combined B.S. and engineering certificate (with a very high grade average). So, she spent a year working for me and then went to law school and ended up making much more than nearly all chemical engineers.
I suspect that the STEM shortage is very spotty and often exaggerated. I also suspect that many college graduates not in fields listed as STEM could be trained to be very capable in some area in which companies truly do have a deficit.
Every young person should be able to reach some degree of self-fulfillment. Today, a very large percentage of our workforce are unhappy in their jobs. One aspect of self-fulfillment in employment is matching abilities to job requirements. The rush to push everyone into STEM careers runs directly counter to that goal.
But again, it depends on your definition of STEM — coming full circle back to my beginning. Is the graphic artist working to design websites a STEM worker? What about the product designer who creates the cases for smart phones? Is the writer who creates the text and images in training documents a STEM worker?
By basing a set of statistics on questions asked of TRs (talent recruiters) about STEM, you are setting yourself up for some biased results. Few TRs really understand what STEM is or what those people actually do on a day-to-day basis. They often have trouble interpreting the job descriptions that hiring managers hand them.
Can we do a better job of teaching math and science so that fewer students are turned off by these fields? Yes — most emphatically. Would that change help our STEM “crisis”? Maybe, maybe not. Will pushing STEM down to kindergarten help? I truly cannot believe that.
Instead, we must stop destroying our young children’s inherent interest in counting things, understanding their world, and creating artifacts. These drives are very basic and must be stamped upon rather heavily to squash. Our schools have been doing a great job of that and have have been helped and encouraged by a large number of people who simply don’t like science because it tells the truth or helps ordinary people to think better.
I continue to believe that the STEM shortage is mostly illusion with just enough reality to make it appear entirely real. The best lies contain some truth. Most of the proposed fixes to this perceived shortage are ill-considered and ill-advised.
By ignoring any STEM shortages and even ignoring this newly minted acronym (new in relative terms), we can focus on improving education in general. Better education will automatically create more graduates ready for many careers, including those considered as STEM.
Life is not always simple. PISA is flawed but has a story to tell. An editorial in today’s New York Times does not use the PISA scores directly but rather to pivot to the real story. See http://www.nytimes.com/2013/12/18/opinion/why-students-do-better-overseas.html.
The opinion neatly explains three reasons why we have problems with our education system using three countries as examples and contrasts.
Finland has a much better teacher education system.
Canada has a much better mechanism for funding education.
Shanghai has decided to make education for all truly equal.
A few quotes from the article will help to explain.
“… thanks to a national decision made in 1979. The country decided to move preparation out of teachers’ colleges and into the universities, where it became more rigorous.”
An recent report rated “teacher preparation programs ‘an industry of mediocrity,’ rating only 10 percent of more than 1,200 of them as high quality. Most have low or no academic standards for entry.”
When my son we emergency credentialed, he had to take teacher college classes every semester. According to him, they were a joke and without any intellectual rigor at all.
It’s not our teachers’ fault that they get substandard education. Too many of the teacher colleges, especially the for-profit ones, are just in the business of printing diplomas and collecting tuition.
Canada does not use property taxes to fund education — at all. In California, where I live, the schools get state money apportioned according to need. Sounds good, right? However, my community receives one of the lowest per-pupil fundings in the state but still maintains very high levels of achievement with over 90% of high school graduates going to college and scoring third in the state on its standardized tests. Just a couple of miles away, another school system languishes. The difference is the education foundation here that collects over $1,000 from every family who joins and has well over a million dollars a year to add to the school budget. It’s population of 35,000 has plenty of homes without children and lots trying to move here to take advantage of the schools.
California, as a state, ranks 35th in the nation in per-pupil spending. You couldn’t it here as a new math/science building is about to be erected. The state-supplied per-pupil spending is less than the surrounding poorer districts. One nearby chronically underachieving district gets one-third more that my district. It received $10,586 per student and scored an average of 679. My district received $6,363 for the same time period and scored 926. The amount received does not match the actual spending.
Finally, there’s Shanghai deciding that a mind is too valuable to waste and providing great education to every child, including immigrants.
[…] list of the top performing countries in education is based on a survey and methodology which is highly […]