Aviation AI: What Evidence Should You Ask For?

Before trying artificial intelligence (AI) in aviation, ask what it is intended to do, where it has been evaluated and how your team can challenge or stop its proposal. A successful demonstration deserves a well-prepared trial; it does not establish suitability for operational use.

The conversation often starts with an appealing practical promise: find information faster, spot a discrepancy or prepare an analysis. For the team watching the demonstration, the challenge is turning that interest into questions the supplier can actually answer.

EASA’s starting point

The programme published for the EASA AI Days, on 9 and 10 September 2026 in Cologne, covered ways to establish the reliability of artificial intelligence (AI) and the conditions in which people use it. Those questions remain useful when preparing a trial with your team.

The EASA discussion paper, Proposed Issue 3, published on 3 June 2026, is presented as a proposal. It is not a general authorisation to use a tool. This article draws on that discussion framework without attributing conclusions to the conference.

The worksheet below is our suggestion for preparing a supplier discussion. It does not replace analysis of the particular system or its applicable requirements.

A demonstration worth exploring

Consider a fictional case. Nora coordinates digital-tool trials at an aircraft operator. A supplier demonstrates an assistant that brings maintenance information together and suggests avenues to examine. The presentation runs smoothly, the answers are readable and her colleagues immediately see the attraction.

Then a technician asks a simple question: what happens when a document is missing? The supplier replies that the user stays in control. Nora does not dismiss the project, but that answer is not enough to organise a trial.

She wants to know what her colleague will see, what they can check and how they will continue if a proposal sounds plausible but lacks support. That is where useful evaluation begins.

Describe a task, not a broad promise

Before discussing accuracy or time savings, write down the task you want the tool to perform. “Help maintenance” is too broad to organise a trial. “Bring together available information before a technician examines a file” gives you a clearer starting point.

In the fictional case, Nora specifies that the assistant prepares possible leads. The trial gives it no operational decision and changes none of the existing responsibilities. This boundary lets the team discuss the product without assuming a feature shown on screen suits every use.

Ask the supplier to describe the same task in their own words. If they describe something different, discovering the mismatch now is easier than finding it after sharing documents or arranging the team's time.

Ask what sits behind the results

A demonstration rarely answers every question a future user will have. Which documents were available? Which situations were absent? Did the people evaluating the answers already know the files?

You do not necessarily need another customer's confidential data. You need enough detail to understand whether the evaluation resembles your intended use. An attractive overall number is hard to interpret if easy cases and important errors are mixed together.

Nora asks for explained examples of failure, including how those failures were detected. She is not looking for a tool that claims never to be wrong. She wants a supplier who can show where the proposal becomes less reliable and what the user sees at that point.

For the document-retrieval side, our article on Korean Air's AI search examines the link between an answer and the records found. Here, the question comes earlier: do we have enough evidence to begin a trial of this tool?

Let the person responsible try the oversight

“Human validation” becomes useful when you can describe it on screen. Can the person see the necessary evidence? Can they open the relevant document? Do they understand what remains uncertain? Can they continue without using the proposal?

In our example, Nora invites a technician to work through an incomplete fictional file. She asks them to explain what they would do next. If they must guess why the assistant suggested a particular lead, an extra “I checked” box will not solve the problem.

The session also reveals workload. A tool that requires a fresh investigation for every answer may move the effort rather than reduce it. That is not always visible in a presentation run by its designer.

Plan the stop and the update before starting

A reassuring trial is not one without stopping conditions. Agree what triggers a pause: an out-of-scope proposal, an inaccessible essential source or behaviour the team cannot explain, for example.

Also specify how work resumes during the pause. In Nora's case, the team returns to its usual method. Reaching the supplier should not be necessary to make that return possible.

Finally, ask how changes will be announced. A different tool version, model or document collection may justify repeating selected cases. Agree what you will retain to understand the original trial result after the product has moved on.

A worksheet that makes the discussion concrete

Use this preparation aid with the people who will actually conduct the trial. It is an editorial suggestion, not a list of EASA requirements.

• The task: what the tool prepares and what it does not decide.
• The situations: documents and difficulties representative of intended use, including cases not yet evaluated.
• The results: evidence supplied and questions it leaves open.
• The person: who examines the proposal and what they need to challenge it.
• The stop: the signal that pauses the trial and the working method that resumes.
• The next step: what gets checked after a change and who decides whether to continue.

In Nora's fictional file, one line remains unanswered: the supplier has not demonstrated behaviour when documents are missing. The team keeps that as a prerequisite for the trial. It treats neither the missing answer as proof of danger nor a commercial promise as validation.

Leave with a decision colleagues can understand

You do not have to finish the meeting with a purchase or a rejection. A limited trial can be a sensible next step when its purpose, boundaries and responsibility are clear.

The useful result fits into a sentence colleagues who missed the meeting can understand: we will proceed with this task and these people after receiving this evidence. Or we will wait because this question remains unanswered.

That is practical preparation for discussions about aviation AI: arrive with a specific use and leave with evidence to examine, rather than a general impression of how good the demonstration looked.