Industry Wide Analysis: The Authorisation Holder Performance Indicator

In 2011/2012, the Australian Civil Aviation Safety Authority (CASA) was redeveloping its surveillance procedures with the production of the CASA Surveillance Manual (CSM). There were a number of objectives attached to this project with taking a risk-based approach to surveillance chief among them.

The previous manual already had a method by which an authorisation holder (i.e. a certified air carrier, aerodrome, delegation or other civil aviation certificate holder) might be tagged as being of concern but there was little guidance provided to inspectors and managers as to how to appy these labels. The concept was known as Surveillance Posture and included three levels.

As a member of the Safety Systems Office (SSO), I was tasked with looking into this process and identifying a way of making it more risk-based, systematic and defensible. 

Building on the Work of Others

When I was assigned to this task, a questionnaire approach had already been developed but the questions were a little ad hoc, varied between industry sectors or hadn't yet been developed for some areas. Other states' Civil Aviation Authorities (CAAs) had also been working in this area with the New Zealand approach reviewed as part of our work. 

What I set out to do was to build on this work and, using the well-established Reason Model, develop a proforma set of questions that could be used for any authorisation holder type - airline, airport, maintenance organisation, manufacturer, etc.  I did this by building a large question bank from the existing questionnaires and started to identify common questions and underlying concepts. In was surprising to see the Reason Model fall out of this analysis so easily. There were a couple of gaps but they were easily filled in with targeted questions.

Assessment Algorithm

With a final set of sixteen questions spread over the four levels of the Reason Model, the next step of the process was to develop the assessment algorithm that would turn a completed questionnaire into a score that meant something in the real world.  This was done through trial and error with sense checking provided by colleagues in the SSO.

Using a set of archetypical operators, I manipulated variables across the question groups to arrive at indicative lists of surveillance postures for these hypothetical operators. Each operator through each variation of the algorithm was discussed to see if the result made any sense. 

As I was essentially systematising a process that used the knowledge and experience of inspectors to arrive a consistent conclusion, the algorithm needed to make logical sense to those same inspectors.

Validation

In order to deploy the new system, CASA needed to show that it did what we said it was going to do. We also needed to show inspectors what it was, how it did what it did and that those results supported their own assessments of operators. 

The validation program of the AHPI involved using a sample (approx 10%) of all operators across the Australian aviation industry and their assigned inspectors to test the system in small groups across all CASA regional offices and divisions.

These groups were first asked to assign a surveillance posture to the sample group. Thanks to CASA's policy of rotating inspectors periodically, each group usually consisted of more than one inspector familiar with the sample group operators. This allowed for moderation of the "analogue" approach to surveillance posture assessment. Then, inspectors were asked to assess their operators using the AHPI tool. 

Afterwards the results were calculated and compared to the traditional assessments of the operators. Where inconsistencies were noted, they were discussed with the assigned inspector to confirm their initial assessment. In these cases either the inspector revised their "analogue" assessment, altered answers to the individual questions or small tweaks were made to the algorithm.

Defence

The final hurdle of this project was to present and defend the new tool to the CSM Project Board. This group was overseeing the entire manual redevelopment project and, as senior leaders in CASA, were very demanding when it came to making changes and proposing new systems. 

At the next board meeting, I presented the development and validation of the AHPI.  The probing questions came thick and fast but I remember mostly ensuring that the power of the tool was not overstated. THe AHPI served a single purpose of providing inspectors with a way of categorising operators for surveillance intensity. The real work of any inspector remained and still remains auditing in the field to ensure that operators are complying with their regulatory obligations.

Epilogue 

Earlier this year, CASA revised the CSM and made changes to the Authroisation Holder Assessment process. It now appears that the AHPI is part of a larger assessment process and it is good to see the tool has lasted over five years so far.