Verification Drift: The Hidden Risk of Trusting AI in Legal Research
An empirical study at the University of Wollongong has uncovered a troubling trend in using generative AI (GenAI) in legal research and writing: verification drift describing how initial caution around AI use often fades, leading users to place misplaced trust in AI-generated content, even when it contains subtle inaccuracies or fabricated material.
Despite growing awareness of “AI hallucinations”—where AI tools like ChatGPT produce convincing but false information—legal professionals and students continue to fall into the same trap. The study in a law elective on emerging technologies encouraged responsible AI use alongside traditional research methods. While many students used GenAI effectively—for brainstorming and drafting—others failed to verify claims properly despite prior training. In some cases, legitimate sources supported AI-generated statements but distorted the original meaning.
One key reason for this behaviour is the seductive polish of GenAI output. Its authoritative tone and stylistic fluency make even incorrect statements seem trustworthy. Over time, this erodes users’ scepticism—hence the concept of verification drift. Compounding the problem is the sheer effort required to verify AI output. Instructors found that cross-checking every claim against its source was more mentally taxing than traditional legal research.
The lesson for legal professionals is clear: AI literacy must go beyond guidelines. Training should include practical exercises, case studies of courtroom misuse, and frank discussions about GenAI’s limitations. Like Odysseus resisting the Sirens, lawyers must be prepared in advance—not just informed—to resist the allure of AI’s apparent efficiency.
Key Takeaways:
- Verification drift explains why AI misuse persists even among trained users.
- GenAI’s convincing tone can mask factual errors or misrepresentations.
- Verifying AI output is labour-intensive and can offset efficiency gains.
- Legal education must go beyond written guidelines to develop hands-on AI literacy.
- Scepticism about AI’s performance in legal tasks is warranted until independent benchmarks are established.
Source: LSJ.com.au
Leave a Reply