"I knew it all along..." - avoiding hindsight bias after eruptions

“I knew it all along…” – as volcanologists, we need to be careful not to fall into the many traps that come from retrospectively looking at and indeed commenting on crises or catastrophes such as the recent eruption of Ontake.

There is a fantastic book you might want to read: Thinking fast and slow by Daniel Kahneman, which synthesises a huge body of research about how and why we make the decisions we make, particularly when it comes to risk and uncertainty. Many readers of this blog will be familiar with Kahneman’s papers, notably the 1976 “Heuristics and Biases” work with Amos Taversky. Others will be familiar with some of the work by his former PhD student Baruch Fischhoff on (among other things) risk communication*. I was planning on writing a short review of Thinking fast and slow from the perspective of what volcanologists can learn from cognitive psychology, but the eruption in Japan has got me thinking about one particular cognitive trap – the ‘hindsight bias’ or the ‘I knew it all along principle’, first investigated by Baruch Fischoff.

The key message is that as a group we must be very careful that when looking back at past eruptions, particularly when eyeballing monitoring data post-hoc, that we don’t make pronouncements about “missed warning signs” because we interpret things with the benefit of hindsight.

It turns out that it is very difficult for a human mind to reconstruct what we thought about something once we adopt a new belief about it. It leads us to believe that we understand the past, overstating the accuracy of the beliefs that we held (or would have held) at the time, as these are corrupted by what we now know. Kahneman suggests that if we are surprised by an unpredicted event, we adjust our view of the world to accommodate that surprise. Thus when we look back, we forget the state of mind or understanding that we had at the time, and simply think about what we know now.

What hindsight bias can do is lead us to interpret the quality of a decision (such as the recommendation for some kind of mitigative action) on whether the outcome was positive or negative, rather than whether or not the decision making process was sound. This bias leads us to a) overstate our expertise post-hoc, b) neglect the role of luck (or lack of it) in a particular outcome and c) suppress any memory of the effect that uncertainty will have had on our or other people’s interpretations/decisions.

Our natural tendency is to criticise decision making on risk issues when an outcome is negative, and neglect to recognise or praise decision-making when the outcome was good; this ‘outcome bias’ (a facet of hindsight) affects our interpretation of past events far more than we might realise. When considering what might happen at a volcano, a simplistic explanation is that we can consider the probability of an eruption happening given some monitoring signal [P(A|B)]. But, after an event has occurred…it’s quite different! It’s no longer an event that could happen (a chance or likelihood) but a certainty. So when we re-interpret past events, hindsight bias makes it very difficult for us in our present state of certainty, to acknowledge the attendant uncertainty before the eruption occurred. We find it very difficult to reconstruct or understand what our past belief would have been.

Kahneman suggests that these biases make it “almost impossible to evaluate a decision in terms of the beliefs that were reasonable when the decision was made”.

In fact, research suggests that the worse or more shocking a catastrophe is, the more acute hindsight bias becomes (think back to reactions in the aftermath of 9/11). This – in the case of Ontake – is reflected by language such as “failed to forecast” used in many** news articles.

So what does this mean for volcanologists in the wake of a tragedy such as the eruption of Ontake? Well, the first thing we should be aware of is that our opinions post-hoc, about what monitoring data may or may not have shown, or what decisions should or shouldn’t have been made, are prone to huge biases. So, we should be very careful what we voice about these events…particularly to the media! If we are going to retrospectively look at something, let’s do it in a robust and sensible way, such as the work by TheaHinks, Willy Aspinall and others on the 1976 eruption of Soufriére Guadeloupe.

Another point is that from afar – not being a Japanese volcanologist working on Ontake – the availability of information for us to be able make an informed opinion is surely very limited (what Kahneman refers to as the ‘availability bias’ or the ‘what you see is all there is to know principle’). So, just as we should be very cautious about talking about ‘missed signs’, we should also be aware that when we say things like ‘it’s impossible/very difficult to predict such eruptions’ or ‘there were no precursors’, our opinions are perhaps based on very sparse evidence (of course we can draw on other examples from other cases – but hopefully you get my point). In essence, maybe we could do with waiting for a little more information before passing comment.

Hopefully you get the idea that if you haven’t yet read Thinking fast and slow, then please do. It’s very difficult to overcome the various heuristics and biases that affect our opinions and decisions (even Kahneman admits to relentlessly struggling with this) …but being aware of them is an excellent first step.

 * Want to know more about the science of risk communication - read this excellent paper by Nick Pidgeon and Baruch Fischhoff 

** Not all articles/commentaries fall foul of the hindsight bias -  if you want to read some measured and not overly opinionated articles by volcanologists about the Ontake eruption – you might want to look here (Becky Williams) and here (Eruptions blog).

The ‘social’ geologist

“The social geologist - thinking about people and not just rocks” 

As a young undergraduate, I used to love going out into the field - swinging my hammer at any outcrop foolish enough to show itself to me. Whilst I wasn’t as hammer happy as some of my peers (“Jackhammer” Jake in particular), there seemed to be so much rock out there, and too many samples to collect. I can distinctly remember though, standing on a desolate beach in Somerset on a windy afternoon, being told we were not allowed to hammer at rocks here, as the sight was a SSSI (Site of Special Scientific Interest). This was probably my first introduction to geologists having an obligation to preserve things, even at small scales such as this. 

 As geologists we have a responsibility to do things sustainably, to preserve and in some cases protect, whilst making the most of the world that we live in and it’s resources. This obligation, or responsibility, doesn’t stop with the natural environment and it’s feelings or preservation, but also to the people that live in it. Can the people be more important than the resources or the scientific research? Of course they can and indeed they are - a fact that can often be overlooked. 

But why, as geologists, should we be bothered about people? Isn’t that what geographers or social scientists do? I remember having some disdain for “Jackhammer” Jake, famed for his rock smashing prowess, when he switched courses from geology to Environmental Geoscience. It was almost as if the word ‘Environmental’ made us ‘pure’ geologists shudder. Jackhammer however, was just ahead of the curve and spotted a trend that we should all take note of. It is impossible to divorce the world of science from the public’s impression of it, and so we really do need to keep people in mind when we plan our next research, exploration or extraction. I’m not calling for everyone to become social scientists,  not every geologist needs to go out and interview people (please don’t just go out and do a questionnaire to tick this box), but all of us need to at the very least, engage with those that do try to listen to or understand people (social scientists) and we need to keep an open mind. 

Half way into my MSc, I worked at the Montserrat Volcano Observatory in the role of scientific communication and outreach. I had gone out there to work at a volcano observatory, hoping to do some quantitative volcanic risk assessment work if the volcano started to erupt, but ended up learning profound lessons about the intersection of people and science. Heightened activity in December 2009 meant that we were no longer able to give sufficient warning to some people in a particular community there, and so the authorities took the decision to evacuate them. Residents there were quite unhappy with the decision to evacuate and in some cases angry. The decision was based on sound scientific evidence, so why didn’t people understand? I was perplexed, but just tried to continue acting professionally. Only later, having acquired some social science research training and returning to Montserrat, did I manage to talk to some of the residents and understand from their point of view why they were so unhappy with the evacuation. What may have seemed black and white to scientists, did not seem that way to the residents. You see, we are all people with emotions, feelings and opinions, but we sometimes forget this when we are being professional scientists. We then manage to divorce emotion from our thinking and in some cases decision making, instead relying on pure logic. Clearly from a practical point of view it is sometimes necessary to act with cold logic, but we should be aware of the other dimension, so that we can at least understand, even if it sometimes does not change the decisions we make. 

People, us scientists included, have opinions and feelings, based on a vast many things. Some opinions may seem ‘wrong’ scientifically, but are they any less valid? The ‘social geologist’ doesn’t need a social science degree… they need to stop and consider the opinions of others before they smash the rock with their hammer, sink a new well or evacuate a village. They are aware of the importance that people and their voices should have in decision making, and strive to be inclusive. If you read the news or articles in scientific journals, our world is facing an uncertain future, in terms of climate, competition for resources or increasing vulnerability to natural disasters. Scientists can’t answer and fix these issues on their own, so we need to include and listen to the public. We can no longer hide inside a lab or behind a computer screen but rather we should make concerted effort to engage with and communicate to people who’s lives we affect with our decisions.