BTIII: Assessing Uncertainty

I can't lie to you. I have been turning myself inside out trying to get a handle on risk evaluation in the aviation safety sphere for close to five years now and I still don't feel any closer to an answer. And I say "an" answer and not "the" answer. Since you are always assessing risk in terms of your objectives, there can and will be multiple approaches to assessing the risk of the same scenario depending on whether you are considering your safety, financial or legal objectives.

The Perpetual Problem?

The "problem" with aviation safety risk evaluation popped its head up again for me in a recent discussion. Without going into too much detail I was discussing the impact of an aerodrome defect with a non-aviation colleague.

We both identified safety as the key impact area and then our company process required us to assess the impact according to a scale (not quite a matrix ;)). We couldn't escape the top box, the highest level category, because as soon as the safety of an aircraft is called into question, you can't escape the possibility of complete disaster.

When pondering this problem, I keep coming back to the idea that aviation, from a safety perspective, is inherently perilous. You can't commit aviation without being "all in". As such, the risk-level question tends to end up as a probability continuum from negligible impact to catastrophe.

Alright, let's stop there. I'm pretty sure I've discussed this stuff before. So, let's take it as read that I am, essentially, only interested in the probability of the worst case.

That simplifies things, doesn't it? Unfortunately, my recent readings of Dekker and Taleb have primed me for skepticism when complex systems appear simple. In the last BT post I wrote, I did highlight that a bow-tie diagram is only ever a model of reality. I think it would be inappropriate to evaluate it using an approach more complex than the model itself.

So I won't be advocating a Baysian approach or a Markov analysis. Instead, let's embrace the uncertainty and evaluate the risk of something by not evaluating the risk of something1.

How to Murder an Analogy

When you want to see something in the dark, it is best not to look directly at it. Due to the biology of the eye, low light receptors are more prevalent in the area of the retina outside of the focal points. Therefore, you will better see an object in the dark if you aren't looking directly at it!

I'm proposing something similar. If you want to evaluate the risk of the bow-tie scenario, don't look at the top event - look around the top event.

Bow-Tie Overview

Around the top event, I consider there to be three primary things - threats, consequences and controls (including defeating factors and secondary line controls).

Therefore, I propose we assess a BT based on:

• our exposure to the threats;

• the criticality of the consequences; and

• the effectiveness of the controls.

Threat Exposure

Exposure is a common word in the risk management game and I really like it. As such, I think it is underused. What I like about it is the implicit idea that risk exists everywhere, at all times but that the context in which we are operating may vary.

If you take my boring predictable runway excursion BT example, those threats really do exist at all airports. All aircraft have the potential to carry out an unstable approach, all runways have the potential to be contaminated but not all contexts have the same exposure to these threats.

Runway Excursion Bow-Tie Model

Why not use probability or likelihood?

Well, probability tends to convey an air of accuracy and mathematical legitimacy which is rarely justified. Likelihood, not so much but it is tied often to an occurrence of a discrete event. Whereas, linguistically, for me at least, I find exposure better attuned to both discrete events and persistent conditions.

So, step one is to assess one's exposure to the identified threats.

Rethinking Influence

On the other side of the top event, let's look at the criticality of the consequences. In an earlier post, I had used the term influence to encompass the concepts of pathways and proximity of events to the final condition (absolute destruction). I've had a rethink and today, I'm going with criticality.

Think of the relationship between each consequence and the potential final outcome. Are there many ways this situation can go pear-shaped? Or is this consequence a LOL-cat's whisker away from disaster itself?

Step two is to assess the criticality of the outcomes.

Self-Critique

Once you've plugged the holes with your controls, identified new holes, plugged them up again and so on, you will need to sit back and criticality assess the effectiveness of those controls.

Without a BT diagram, this could get very hard but the diagrammatic approach can help and some software makes things even easier. Once you have your measure of effectiveness, I think you've got all you need to make an assessment of risk, all without actually assessing the top event.

Step three, assess the effectiveness of controls.

Short-Sheeted Again

How to actually assess exposure, criticality and effectiveness and how to put them together are questions I have not yet answered. But the brain matter is continually churning and as soon as I know (or think I know), I'll post it here.

1. I'm sorry. I've been reading a few obtuse academic texts lately and perhaps the language is rubbing off on me.