AI-Powered Incident Report Summaries

Having mentioned this great new technology a few times, perhaps I should talk about what ChatGPT could do for you. Well, at it’s core, ChatGPT reads and it writes and that got me thinking about similar tasks I’ve had to do in the past. Reading incident reports and summarising them for my bosses came to mind almost at once. So, let’s test it out.

Quick Overview of Using ChatGPT

There are a bunch of ways to use ChatGPT but the best place to start is a standard web browser. After you visit https://chat.openai.com/, register for a free account and log in, you’ll be presented with introductory screen with an input box at the bottom. This is where you enter your “prompts”.

That’s just a fancy word to cover the text you give the AI model that will generate a “response”. It will usually include a question or a direction on what you want the response to be. Here are a few examples:

• Can you please tell me how long I should boil an egg?

• Please describe the process for completing a tax return in New York City.

• Should I ask Mary-Sue to marry me?

The AI will then generate a response (even to that last question). And then, and this is really cool, you can continue the conversation. It will remember your earlier prompts and its responses. So, if you don’t know where to get your W-2 form from, you just ask and the AI will let you know.

Here is a short video.

A quick example showing ChatGPT telling me all about the Wright brothers’ first flight

Prompt Sculpting

The key to getting good results out of ChatGPT is finding the best way to sculpt your prompt. There appear to be whole careers being forged in this space at the moment, but I enjoy a little experimentation. For example, you can ask the AI to write in a particular style, like journalistic or academic, in a particular format, such as in an outline or a script, or even like a particular author, say Tolkien or Dr Seuss. Tomorrow’s post using these factors for comedic purpose.

In our experiment today, I used a range of different approaches to arrive at a prompt format that should be reusable over and over again. We’ll talk about automation at the end but first, let me introduce you to some data.

Online Aviation Incident Reports

To assess ChatGPT’s ability to summarise incident reports, I need some incident reports. So, I headed over to the Aviation Safety Network’s Wikibase. They’ve got a lot of data and with some Python coding (and help from ChatGPT), I was able to pull together a little database of 263 accident and incident reports.

And while I wanted to test the system’s ability to digest different amounts of information, I was butting up against the maximum prompt length. I was also having trouble working out how to feed the data to the system. Even when I asked it, it suggested I put in a spreadsheet and email it (as far as I can tell, no such feature exists). It also suggested an ASCII table and while this worked some of the time, it was hit and miss with larger datasets.

In the end I feed it the incidents from the worst day in December (26 Dec, n=14), one by one. A little laborious but let’s touch on automation later.

Final Prompt

I won’t bore you with all the prompt variations that I tried. Because I tried a bunch of them. Getting it right can be really really really frustrating. The pay-off is that once you find a good prompt, you should get repeatably good results.

In the end, I went with the following:

Using all of the data I have provided, can you please write a summary of these aviation incidents in the style of an executive summary for my boss?

The summary should be at least four paragraphs long. Please write the dates in a regular format.

It should start with an introduction stating that this is a digest of recent aviation incidents and accidents as well as noting the number of event reports, fatal accidents and aircraft destroyed.

It should then detail serious events in a separate paragraph in order of significance. Significant incidents involve fatalities and total aircraft destruction. Less serious events can be noted with minimal detail in a single paragraph.

In a conclusion paragraph, please also note any trends, common details or significant issues.

The Results

Here is the first response:

Initially, I thought it wasn’t bad. Despite the work it took to get to this point (which is why I am a little behind today), I can see promise.

But We Aren’t There Yet

I ran these prompts in a multitude of variations with different data structures, etc. And probably the most striking responses were the ones that made blatant mistakes. Miscalculating the number of incidents, counting fatalities when there were none, and just ignoring some of the reports I fed to it.

But then sometimes, it would surprise you with apparent insight. For example, incident number 2 was not an aviation accident per se. It was a drone strike related to the Russian invasion of Ukraine. As such, the soldiers killed in the event are not listed as fatalities. But on a couple of occasions, the AI would still look at the event details and over-ride the more statistical aspects of the data.

All this suggests to me that these systems are getting very close. The next phase will be for people is specific niches to train these models on their specific needs. Like us in aviation safety, we will be able to train these models on how we structure data, how we talk about common activities and phenomena and what deductions we make from our analysis.

These systems will be able to take that training in and run away with it.

Automation

The above exercise was an experiment looking at a very small part of what could be a highly automated system. As system that reads the latest reports, whether internal or public, and provides a natural language overview for busy people to stay on top of things.

I’ve been experimenting with some of these ideas and have even initiated my own “bot” that likes to post about birds involved in wildlife strikes. I’ll be introducing you to them in the next week or so.

Header image: Lukas (via Pexels)