Understanding hallucination rates in ambient documentation

This article is part of a series exploring implementation lessons from Gold Coast Hospital and Health Service's 16-week evaluation of ambient documentation across 7,499 consultations. For the full analysis and all implementation lessons, see our complete article.

Two findings that seem contradictory

One safety risk highlighted in the evaluation is incorrect or unsupported content making it into the clinical record. Two findings in the paper help quantify that risk, but on the surface, they may seem hard to reconcile.

In staff surveys, 47% of respondents reported observing hallucinations at least once during the trial, and 16% reported observing potential bias (43% response rate).

In a separate structured quality assessment of a small sample of note pairs, outputs were rated highly on "freedom from hallucination" (4.83/5) and "freedom from bias" (5.0/5).

What these numbers mean

These results are capturing different things.

The survey reflects whether clinicians ever noticed something concerning during routine use across many consultations. These survey findings may be subject to self-selection bias, as individuals with personal experience of hallucinations would likely be more motivated to participate, potentially leading to an inflated hallucination rate.

The structured assessment rates the quality of a small sample of reviewed outputs.

Both signals matter.

Understanding what "hallucination" means in practice

It also helps to unpack what "hallucination" means in practice. Clinicians may use the term whenever something in the draft note doesn't match their recollection of the encounter, doesn't fit the clinical context, or doesn't appear supported by what was said.

In that sense, a reported hallucination is a frontline safety signal - something doesn't look right, was noticed by a clinician, and reported.

However, that signal on its own doesn't explain why the concern appeared in the output.

What looks like a "hallucination" can arise for different reasons. From gaps or ambiguity in what was captured, to missing context, to errors in attribution or synthesis. Sometimes more than one factor is involved.

Building systems that catch errors

That's why it's imperative that ambient documentation tools make it easy for clinicians to flag concerns in the moment, and that teams have a structured quality assurance process to triage reports, investigate patterns, and minimise repeat issues over time.

Closing that loop matters too - sharing outcomes with clinicians so they understand what happened, why it happened, and how similar issues will be handled in future.

About this series: This article is part of a series based on independent, peer-reviewed research from Gold Coast Hospital and Health Service. For the complete analysis and all implementation lessons, read our full article.

Continue the conversation: We welcome feedback from clinicians, researchers, and healthcare leaders. Contact our team at clinical@lyrebirdhealth.com

Read the full study: Memon S, Brand A, Taylor B, Michael A, Smithson R. Performance, acceptability, and impact of ambient listening scribe technology in an outpatient context: a mixed methods trial evaluation. BMC Health Serv Res (2025).