In this edition of Ask an Expert, Fiona Morgan, founder of AI Fully Informed, explains how insurers can identify automation’s influence on customer decisions and assess potential privacy disclosure obligations.

Q: If a customer withdraws a claim after an automated message, was it really their decision?

A: It might have been their choice, but I would want to understand what led them to make it.

In my presentation, I used a fictional example of a home insurance claim following storm damage.

The customer uploads photographs, and a computer tool estimates the repair cost. If that estimate comes in below the policy excess, the amount the customer contributes towards a claim, the system automatically sends them a message telling them this.

The customer might then think, “There’s no point claiming,” and withdraw.

That looks like a customer decision. But what if the computer underestimated the damage? The customer could abandon a claim that would otherwise have resulted in a payment. That’s where we need to look more closely at the role the technology played.

The automated decision-making changes I discussed require businesses covered by the Australian Privacy Act to describe certain uses of computer programs in their privacy policies. The question isn’t simply whether a computer makes the final decision. It also matters whether it does something substantially and directly related to making a decision that could reasonably be expected to significantly affect someone’s rights or interests.

I wouldn’t give a definitive legal answer about that withdrawal. I’m an AI product specialist, not a privacy lawyer. What I would do is document the estimate, the automated message and the potential outcome, so a lawyer can assess how they fit together.

For brokers, there’s a similar issue before a customer even buys cover. A broking platform might automatically filter insurers according to the client’s occupation and postcode. The broker chooses from the remaining options, but the system has already decided which insurers never see the risk. If that affects the cover or price available to an individual, it deserves attention.

My advice is to look beyond the obvious acceptance or rejection. Ask what information customers receive, which options they are shown and what happens if they do nothing.

That means mapping the process as it actually operates and talking to the people who use it. A customer’s final choice, or a staff member’s final approval, doesn’t, by itself, tell you how much influence the computer had.