Beware of Predictions

“In climate research and modeling, we should recognize that we are dealing with a coupled non-linear chaotic system, and therefore that long-term prediction of future climate states is not possible.” IPCC Third Assessment Report, 2001.

Despite this admission, the International Panel on Climate Change (IPCC) continues to have confidence in a scenario for the future which is so disturbing that nations are being asked to take drastic action to mitigate the climate “catastrophe” that awaits us. While actual evidence suggests that the climate is undergoing natural cyclical change and that the “man-made” impacts are small, policy makers are determined to act on models and scenarios from “experts” which remain unsupported by compelling “real world” evidence.

Indeed, the expert “consensus” position is based on selective use of evidence, some of it from peer reviewed journals and some not, and expert group-think. Psychologists understand this phenomenon and have developed a thorough understanding of just how wrong experts can be.

Phillip Tetlock author of Expert Political Judgement^[1] and a Professor of Psychology at Penn State University, provides strong empirical evidence for just how bad we are at predicting. He conducted a long-running experiment that asked nearly 300 political experts to make a variety of forecasts about dozens of countries around the world. After tracking the accuracy of about 80,000 predictions over the course of 20 years, Tetlock found:

That experts thought they knew more than they knew. That there was a systematic gap between subjective probabilities that experts were assigning to possible futures and the objective likelihoods of those futures materializing … With respect to how they did relative to, say, a baseline group of Berkeley undergraduates making predictions, they did somewhat better than that. How did they do relative to purely random guessing strategy? Well, they did a little bit better than that, but not as much as you might hope …

The psychologist Daniel Kahneman, who won the Nobel Prize in Economics for his work on decision-making, has looked at the issue of “experts” and why they often get things wrong. In his book Thinking Fast and Slow^[2] he points to several aspects of their psychology as factors, but highlights two in particular: the illusion of understanding and the illusion of validity. These are primary causes of experts getting it wrong.

The illusion of understanding refers to the idea that the world is more knowable than it actually is. In particular, experts believe that they have an in-depth and insightful understanding of the past and this enables them to better understand the future. They use what Kahneman refers to as the WYSIATI rule – “what you see is all that there is” and this provides the basis for their confidence.

For example, it must be the case that high levels of government indebtedness (levels of debt to GDP ratio above 90% is the most recent version of this^[3]) stifle the economy and reduce investor and entrepreneurial confidence according to some notable economists. Or it is obvious that human generated C02 is the major cause of climate change according to some climatologists. Both of these understandings are based on a particular view of historical data and “facts” and an extrapolation of these views into the future.

The views exist independently of the evidence to support them. Just as financial advisers are confident that they are successful in predicting the future behaviour of stocks, so macro-economists are confident that their views of austerity have the weight of history behind them. Those committed to the view that human produced CO2 is the primary cause of climate change are not deterred by evidence that it may not be or that climate change has stalled for the last seventeen years.

Experts are sustained in their beliefs by a professional culture that supports them. Austerians  (those who believe that austerity is the only way) have their own network of support, as do the Keynesians who oppose them. Anthroprocene climatologists who believe that man is the primary cause of global warming have their own network of support among climate change researchers and politicians while the skeptical climate scientists also have their support networks. All remain ignorant of their ignorance and are sustained in their belief systems by selected use of evidence and by the support of stalwarts. These supportive networks and environments help sustain the illusion of validity. It is an illusion because evidence which demonstrates contrary views to those of the “experts” are dismissed and denied – the expert position, whatever it may be, is valid simply because they are expert.

Indeed, using Isaiah Berlin’s 1953 work on Tolstoy (The Hedgehog and the Fox), Austerians and anthropocenes are “hedgehogs” – they know one big thing, they know what they know within a coherent framework, they bristle with impatience towards those who don’t see things their way and they are exceptionally focused on their forecasts. For these experts a “failed prediction” is an issue of timing, the kind of evidence being adduced and so on – it is never due to the fact that their prediction is wrong. Austerians who look at the failure of their policies in Europe, for example, suggest that the austerity did not go far enough; anthroprocene climatologists see the lack of warming over the last seventeen years as proof that they are right, it is just that the timing is a little out. Even the climatologist trapped in thick ice in the Antarctic in December 2013 who set out to study the thinning ice-cap claims he just went to the wrong place – “climate change is happening and the ice is melting” he says, as he is lifted off the thick ice by helicopter.

Tetlock’s work, cited above, is a powerful testimony to these two illusions – understanding and validity. His results are devastating for the notion of “the expert”. According to Kahneman, “people who spend their time, and earn their living, studying a particular topic produce poorer predictions than dart throwing monkeys”.

Tetlock observes that “experts in demand were more overconfident that those who eked our existences far from the limelight”. We can see this in spades in both economics and climate change. James Hanson, recently retired from NASA and seen to be one of the worlds leading anthroprocene climatologists, makes predictions and claims that cannot be supported by the evidence he himself collected and was responsible for. For example, he suggested that “in the last decade it's warmed only about a tenth of a degree as compared to about two tenths of a degree in the preceding decade” – a claim not supported by the data set which he was responsible for. This overconfidence and arrogance comes from being regarded as one of the leading climate scientists in the world – evidence is not as important as the claim or the person making it. Hanson suffers from the illusion of skill.

Kahneman recognizes people like Hansen. He suggests

“…overconfident professionals sincerely believe they have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion.”  

There are other psychological features of the expert that are worthy of reflection. For example, how “group think” starts to permeate a discipline such that those outside the group cannot be heard as rational or meaningful – they are referred to as “deniers” or “outsiders”, reflecting the power of group think. The power of a group (they will claim consensus as if this ends scientific debate) to close ranks and limit the scope of conversation or act as gatekeepers for the conversation. Irving Janis documented the characteristics of group think in his 1982 study of policy disasters and fiascoes^[4]. He suggests these features:

1. Illusion of invulnerability –Creates excessive optimism that encourages taking extreme risks. We can see this in the relentless pursuit of austerity throughout Europe.
2. Collective rationalization – Members discount warnings and do not reconsider their assumptions. We see this in relation to both climate change and austerity economics.
3. Belief in inherent morality – Members believe in the rightness of their cause and therefore ignore the ethical or moral consequences of their decisions. Austerians appear to willfully ignore the level of unemployment and the idea of a lost generation of youth workers, especially in Greece and Spain. Anthropecene climate researchers generally present themselves as morally superior.
4. Stereotyped views of out-groups – Negative views of “enemy” make effective responses to conflict seem unnecessary. Climate “deniers” commonly face suggestions that they be prosecuted or punished in some way^[5].
5. Direct pressure on dissenters – Members are under pressure not to express arguments against any of the group’s views. This has occurred in climate change research community, since grants appear to favour those who adopt the view that man made CO2 is the primary cause of climate change.
6. Self-censorship – Doubts and deviations from the perceived group consensus are not expressed.
7. Illusion of unanimity – The majority view and judgments are assumed to be unanimous. This is especially the case in “consensus” (sic) climate change science and amongst austerians.
8. Self-appointed ‘mindguards’ – Members protect the group and the leader from information that is problematic or contradictory to the group’s cohesiveness, view, and/or decisions.

- all of these characteristics can be seen to be in play in the two examples used throughout this chapter – economics of austerity and made man global warming.

There is also the issue of the focusing illusion. Kahneman sums this up in a single statement: “nothing in life is as important as you think it is when you are thinking about it”. “Government debt is the most important economic challenge facing society today” says a well known economists, or “climate change is a life and death issue” says US Secretary of State, John Kerry.  Neither of these statements are true for anyone unless they are obsessive.

Society faces a great many challenges. Much will depend on our own preoccupations and what focus one take for the concerns you have. Some are more concerned about the future of Manchester United or Chelsea football clubs than they are about debt, deficits or climate change. The illusion is that one person’s focus is, by definition, better than another’s simply because they are expert in this field.

Nassim Taleb makes a very compelling argument against forecasting in several of his books, most notably in The Black Swan^[6]. He explains that we can make use of very short-term guesses or predictions, but long-term forecasts are nothing more than pure guesswork. We are guilty of ascribing far too much predictability to the truly unpredictable. It is very common for our human brains to believe we are recognizing patterns that are only a random sequence of events. Experts have tried to overcome our human fallacies with tools such as quantitative modeling. However, even these models play only on our biases. We believe that models that have accurately predicted the future in the past are likely to predict the future going forward. But that is no more true than believing me when I tell you that a coin will land heads up just because I accurately predicted it would do so the last ten times.

So beware of predictions, especially those made by experts. New years eve and day are the prime season for prediction. You have been warned.

---

[1] Tetlock, P. E(2006) Expert Political Judgment – How Good is It? How Can we Know? New Jersey: Princeton University Press.

[2] Kahneman, D (2011) Thinking Fast and Slow. Toronto: Doubelday Canada

[3] Reinhart, C. and Rogoff, K. (2013) This Time Its Different – Eight Centuries of Financial Folly. New Jersey: Princeton University Press.

[4] See Janis, Irving L.  (1982).  Groupthink: Psychological Studies of Policy Decisions and Fiascoes.  Second Edition.  New York: Houghton Mifflin.

⁶ It has been suggested that those who deny climate change is caused by human activity should be put to death. See http://joannenova.com.au/2012/12/death-threats-anyone-austrian-prof-global-warming-deniers-should-be-sentenced-to-death

[6] Taleb, N ( 2010) The Black Swan – The Impact of the Highly Improbable.  New York: Random House.

School Choice, PISA and Policy

The educational neo-conservative ideologues are out in force in the US, UK and Canada and they are using their interpretation of the OECD PISA results as a smokescreen to justify their take on what needs to happen to improve school system performance and outcomes. Whether or not the PISA data supports their propositions, they make their claims anyway.

The latest example appeared in the Globe and Mail where Paul Bennett (a consultant ) argues a case for school choice being a “solution” to Canada’s education “problem” (see here).

An analysis of the PISA data sets shows that Bennett’s arguments are without foundation (setting aside some errors of fact in his writing). Writing in The Guardian using his in-depth analysis of the PISA data, Pasi Sahlberg of Harvard University makes clear:

School choice does not improve the performance of education system. School choice and competition between schools are related to greater levels of segregation in the education system. That, in turn, may have adverse consequences for equity in learning opportunities and outcomes. Indeed, successful education systems do better than those that have expanded school choice. All successful school systems have a strong commitment to maintain their public schools and local school control. PISA 2012 data show that the prevalence of charter and free schools with related competition for students have no discernible relationship with student learning.

and that what the PISA data actually shows is the failure of the neo-conservative market agenda for education (see here).

So rather that attend to the noise of Bennett and his cohort, pay attention to those who look to understand what the evidence really says.

JEFF JOHNSON'S PISA ENVY

The headlines have past and we can now settle down again and get back to the real work of helping students learn, engaging with communities in support of that learning and supporting our teachers who, each day, work hard to ensure that high quality learning takes place.

The headlines on Tuesday were focused on the OECD assessments of students – the results of the Program of International Student Assessment (PISA). The analysis of the 2012 results suggests that Canada was among the high-level achievers, though performance of the country's 15-year-olds in math has declined, with a 14-point dip in the past nine years. Manitoba and Alberta have really seen the greatest decline in math scores. Performance in reading has remained relatively stable, and there has also been a decline in science performance - dipping from an average of 534 in 2006 and 529 in 2009.

The good news for Canada, especially Alberta, is that we do well on measures on equity – access to and success in education: we are not a post-code edutocracy. Many other countries (like the US and Britain) are. Also good news is that the global education reform movement countries (GERM) do especially poorly when compared to those which pursue equity as the cornerstone of their educational strategy.

Now before we get excited, a note of caution. For an international test to work, all students tested have to answer the same questions, or at least questions of similar difficulty and intention. In one obvious sense, they don't: the questions are translated into different languages which, according to one Norwegian academic, "results in rather strange prose" in his country and for several others. Danish academics, when they analyzed the 2006 Pisa tests, found that eight of the 28 reading questions were deleted from the final analysis in some countries. Moreover, about half the students participating that year weren't tested on reading at all. The OECD, which runs PISA, says it calculates "plausible values" for the missing scores, and this is a standard statistical device. But it's a hard idea for most of us to get our heads round, and many statisticians dispute its validity, suggesting that the results are nonsensical and meaningless.

Further, a study by Keith Baker published in 2007 showed that there is no relationship between a nation’s economic productivity and its test scores. Also, test scores did not bear any relationship to quality of life or democratic institutions or creativity. Indeed, the U.S. “clobbered the world,” with more patents per million people than any other nation even though it has poor scores on PISA and other measures. Baker concluded that a focus on such things as PISA is an unhealthy and distracting basis for educational policy.

The PISA results often produce “PISA envy” amongst Ministers of Education, who jockey for insight and explanations on the “big PISA day” every few years. Our own Alberta Minister, Jeff Johnson, is no exception. He says he is looking around the world to understand how we can improve math education and continue to improve overall standards and he is already convinced that the issue is teacher quality. Conveniently, given that he already has plans to act on this “issue”, he sees this as the major task for 2014.  In part he is right – we already have excellent teachers who now need the right conditions of practice and support so as to be able to fully utilize their skills. Rather than “transform teacher quality and teacher education” we need a genuine partnership at the level of the school and the district  between students, teachers, administrators and parents to ensure we have great schools for all students in all areas of Alberta. This is not what Johnson has in mind. He intends to tinker and change.

But we shouldn’t let league tables, PISA or standardized tests drive what we do. We should let teachers and in-school administrators, supported by district colleagues, drive what we do. Ministers of education should ensure an adequate supply of resources, appropriate over-arching conditions of practice, an agreed curriculum framework and then get out of the way. Schools is where learning takes place and schools are the place that makes a difference. Give our schools back to the teachers and stand by to be amazed at what they can do.

Science and Public Policy

How should science shape public policy?

This is not a theoretical question. It is in fact a very practical one and is at the heart of the debate taking place at COP19 – the Climate Change conference taking place in Poland. The issue on the table is deceptively simple. Africa, small island states and others are seeking compensation from the developed world for the impact climate change “is having” and “will have” on their States. The sums in question are in the billions of dollars. Africa alone estimates that it requires $20 - $30 billion annually to 2035 and then significantly more, perhaps up to $60 billion a year.

The call for cash is based on the assumption that extreme weather events are “a result of” climate change. Super-storm Sandy, the Hiyan typhoon and other such events are cited as examples of the consequences of high levels of CO2 emissions.

The problem is that the scientists most closely involved in understanding extreme weather events do not see a direct link between climate change and extreme weather or between levels of CO2 and extreme weather. Indeed, the body which governments use to establish the “state of science” – the International Panel on Climate Change (IPCC) – has explicitly said in both 2012 and 2013 that it cannot establish any clear link between climate change and the levels of man-made CO2 in the atmosphere and extreme weather events. Other researchers who have worked on this file for their careers, such as Roger Pielke Jnr from Colarado,  also agree with this conclusion (see here). While there are always some who will say that there is a link – James Hansen and Michael Mann can be relied upon to make such leaps – these are the new deniers of science.

Another rationale for this call for cash is the threat of rising ocean levels. Yet there is disputed science here. A paper published recently in Global and Planetary Change finds global sea level rise has decelerated by 44% since 2004 to a rate equivalent to only 7 inches (17.8 cm) per century – well within the boundaries of natural events. According to the authors, global mean sea level rise from 1993-2003 was at the rate of 3.2 mm/yr (± 0.4 mm/yr), but sea level rise "started decelerating since 2004 to a rate of 1.8 ± 0.9 mm/yr in 2012."  Given that the science showing a link between climate change and sea level rises are not crystal clear, is this a sound basis for the redistribution of substantial sums of scarce funds?

A part of the rationale for the redistribution is the invocation of the precautionary principle. But this is a “weaponized” version of this principle and  the scientific community is now speaking out against this “weaponised” use of the precautionary principle.

For example, eighty-one of the world’s leading toxicologists recently signed a letter to the EU Chief Scientific Advisor expressing their concern at the EU’s lack of proper scientific procedures in assessing potential endocrine disruptors and their impact on health – yet the EU is seeking to ban the use of endocrine disrupting chemicals (EDCs). These are present in a large number of everyday items, such as food packaging, cosmetics and pesticides. A small number of studies have suggested a possible connection between EDCs and the decline of sperm counts, hormonal changes in women and the increase in certain cancers. The signatories emphasized the importance of using the best science to find a sensible, rational way of setting policies. But the EU is not interested in “science” just in looking to be doing the right thing (a.k.a. as political correctness).

That is what is happening in Poland. On the basis of “being seen to do the right thing”, countries are contemplating the creation of a $100 billion a year fund to compensate states for the impacts of climate change, despite the lack of strong and compelling scientific evidence that the connections being made between extreme weather events or sea level rise and climate change.

The transfer of wealth from rich to poor has always been on the agenda in these conversations, starting with the Earth Summit in Rio in 1992. Maurice Strong, the Canadian who was Secretary-General of this summit, said at its opening “we should consider new taxes, user charges, emission permits, citizen funding all based on the polluter-pays principle”. Also involved here is the requirement that such funds should be managed through “global government” organizations, such as the UN. Indeed, a document leaked in 2010 makes it clear that wealth transfer and global governance are explicit goals of the UN (see here).

So is this all about science? No. Is this all about politics and political correctness? Yes. Are science and political correctness connected? What is occurring is the hijacking of science by politicians – including President Obama and others. Is this desirable? No. Is this good for science? No. We should refine our critical skills and our sensitivity to science as a practice so that we can challenge the politics of wealth transfer by challenging the claimed links to science – they simply are not there.

Sea Level Rises and the Precautionary Principle

Sea level rises are said to be the most serious consequence of climate change. According to the draft IPCC’s fifth assessment draft leaked recently, sea levels are likely to rise by between 29 and 82 centimeters by the end of the century, compared to 18-59 centimeters anticipated in the 4^th Assessment report published in 2007 – the IPCC thinks things are getting worse. Satellite measurement shows that sea levels have risen (app) 6 cm since 1990. If we see a 6 cm rise every 15 years (app) we will see a rise of (app.) 34 cm by 2100.

A paper published recently in Global and Planetary Change finds global sea level rise has decelerated by 44% since 2004 to a rate equivalent to only 7 inches (17.8 cm) per century. According to the authors, global mean sea level rise from 1993-2003 was at the rate of 3.2 mm/yr (± 0.4 mm/yr), but sea level rise "started decelerating since 2004 to a rate of 1.8 ± 0.9 mm/yr in 2012."  The IPCC’s fourth assessment looks more likely to be the “best case” at this time rather than the more recent IPCC analysis.

Some suggest that these estimates are far too low. National Geographic said this in its November 2013 issue:

“Climate scientists now estimate that Greenland and Antarctica combined have lost on average about 50 cubic miles of ice each year since 1992—roughly 200 billion metric tons of ice annually. Many think sea level will be at least three feet [91 cm] higher than today by 2100. Even that figure might be too low.”

- the 91 cm figure being much higher than the IPCC anticipate.

The World Bank analysis also assumes a higher rate than the IPCC. A recent study of the likely impact of sea level rise on coastal cities suggests that the mean sea level will rise between 0.2 meters and 0.4 meters (19. 8 cm to 39.9 cm) by 2050. This both larger and faster than other estimates.

James Hansen, the doyen of climate change scientists, sees ocean level rise as even more alarming.  He has suggested that sea level rises could be between 15 and 20 feet – that is between 457 cm and 356 cm – since the IPCC models take inadequate account of ice sheet melting.

Not to be outdone, Ben Strauss from Climate Change Central in the US suggests in a recent study that by the end of this century, if global climate emissions continue to increase, the climate system may lock in 23 feet of sea-level rise." 23 feet is 701 cm – way beyond all consensus models. He basis this on a calculation of sea level rises linked to the level of CO2 emissions and the predicted temperature rises these levels of CO2 will cause.  Strauss sees the IPCC as “enormously cautious”.

Meantime, the ultimate skeptical ocean scientist is Nils-Axel Molner. He has been studying sea level rises for most of his career and he dismisses most of the concerns out of hand.  Indeed, he suggests that there is no discernable rise in sea level beyond that normally expected for the last fifty years.

So, what to do? If you are a policy maker you have an international body suggesting that sea level rise could be somewhere between 26 and 32 cm by the end of the century. Then you have respected scientists who claim “special expert knowledge” suggesting that this is by far too cautious and is more likely to be around 700 cm. Yet another suggests not to panick, since nothing is happening. What is clear is that we don’t know – the science is very unsettled.

One place has already made its decision. Kiribati – a small island in the Pacific – is seeking to buy land in Fiji so that it can relocate its population of 113,000. One member of this community is seeking asylum in Australia as a result of these developments. Other countries are seeking reparations from the developed world to pay for sea defenses – this is a major argument behind the $100 billion annual fund being demanded in Poland as part of this weeks annual climate change jamboree (COP19).

Outgoing New York Mayor Bloomberg, whose own team of scientists suggest that sea level rises in the New York area could be as much as 73 cm by 2050, has proposed an investment of just over $19 billion in strengthening  the sea walls and flood defenses of New York city. New York is just one of twenty five cities around the world to be investing in these strategies.

The precautionary principle seems to apply. If you have responsibility for a coastal city or a community in a floodplain, there is a need to plan and to take such precautionary measures as seem appropriate and affordable. While precaution can be expensive, dealing with the aftermath of flooding is even more expensive.

But the next time you hear a politician say that the science is settled, just smile. It isn’t.