Cases keep cropping up where counsel has used AI to create a court submission containing made-up cases. The common response on the part of courts and the profession has been: ‘prompt, but verify.’ It’s okay to use AI, just make sure it’s accurate.
I think this response misses the mark. But consider first how fixated we’ve become over the issue of verification — implying that this is all we need to be concerned about in deciding whether counsel should be using AI to write court submissions.
As Judge Moore in a Federal Court case wrote earlier this year:
The use of generative artificial intelligence is increasingly common and a perfectly valid tool for counsel to use; however, in this Court, its use must be declared and as a matter of both practice, good sense and professionalism, its output must be verified by a human.
In that case, counsel’s material had cited two fake cases, one that didn’t apply, and “hallucinated the proper test for the admission on judicial review.” Disclosure and accuracy were what mattered here. Not the quality of the submission or whether counsel met the duty of competence in using AI to create it.
This past fall, the spotlight shifted to a superior court case in Ontario where counsel had created a factum for a motion using ChatGPT. It contained hallucinated case citations and mis-cited the rules of court. Back in May of 2025, counsel initially pointed the finger at staff for using AI. The court accepted her apology and waived off a criminal contempt citation, due in part to its finding that counsel was unaware that AI hallucinates. (In September, however, counsel admitted that she, not staff, had used AI to make the factum — and Crown has launched a new contempt proceeding in response.)
Part of the court’s rationale in deciding against a first contempt citation (in May) was that counsel “undertook to complete no fewer than six hours of Continuing Professional Development training in legal ethics and technology, including addressing specifically the professional use and risks of AI tools in legal practice.”
I suspect that CPD training would echo what lawyers on social media are saying in response to this case. One writes on LinkedIn: “…relying on LLMs (and the products built on them) without rigorous verification is a liability… If your AI cannot provide a non-generative, hallucination-proof citation for every claim it makes, it does not belong in a courtroom.” But does this mean that if my AI submission meets the standard of ‘rigorous verification,’ it does belong in a courtroom?
Law societies across the country are also fixated on the verification issue. The Law Society of BC’s ‘Guidance on Professional Responsibility and Generative AI’ points out that these tools can “create work product that appears very polished,” but warns counsel to be “careful to not lose sight of your responsibility to review the content carefully and ensure its accuracy.”
A white paper from the LSO in mid 2024 on using generative AI tried to be more specific, but is still a bit unclear: “Generative AI can be used to prepare first drafts of certain documents including memoranda, letters and even opening statements or examination questions.” Does ‘memoranda’ here mean memoranda of argument, as in a factum or written submissions on closing? What should or shouldn’t we be using AI to do when it comes to preparing for court?
I believe the answer to this question doesn’t turn on the accuracy of the AI tool you choose. It turns on the question of competence.
The real problem in the hallucination cases, it seems to me, is not a failure to verify. It’s whether using AI to create a court submission breaches one of the fundamental duties of all lawyers in Canada: the duty of competence.
“A lawyer shall perform any legal services undertaken on a client’s behalf to the standard of a competent lawyer.” (LSO, Rules of Professional Conduct 3.1-2)
Three reasons why not to use AI to draft court submissions
I appreciate that lawyers are in a rush and may want to use AI to generate a draft submission they can edit. But I put forth for consideration the following proposition. If you use AI to produce a substantial draft of a court submission (i.e., anything more than a mere outline of an argument) — even if you verify all of your citations — you run the risk of breaching your duty of competence, in spirit at least, if not overtly. AI is good at many things, but it cannot draft a submission nearly as well as you can, if you put in the effort.
Three reasons why:
1. Verification won’t rescue an AI submission.
It makes no sense to have AI produce a draft factum citing fake cases and think that you could merely swap out the fake ones for real ones. The whole submission would be broken from the outset. Every argument, if not every sentence, would no longer correspond to the content of the real cases.
2. Legal AI is good but not that good
Platforms like Protégé and CoCounsel are better than free AI like ChatGPT in that any citations it does include in a memo it produces will link to real cases. But I’ve experimented with these tools a fair bit. I find that when they cite a set of cases to support an argument, the various steps in the argument tend to be strewn with hallucinations — claims about what cited cases stand for but don’t. Legal AI is good for research: finding cases, providing snapshots of law on point or certain issues. It’s not consistently reliable for putting together a comprehensive legal argument tailored to your facts.
3. AI can’t handle the myriad subtleties at play
More to the point, in writing a competent submission, there are simply too many variables in your case that need to be considered and combined in a legal argument for AI to be as good a tool for this as all the tiny cells in your brain working together (fueled, of course, by your favourite latté).
In order to be sensitive to all the nuances and facts in your case, your prompt would have to be as long and thoughtful as the argument that AI would generate. Put otherwise, no tool will do as good a job as you in thinking through the argument at each stage and deciding how to frame the various authorities, facts, and issues. Even the most advanced AI remains too blunt an instrument. I say this as a huge advocate and heavy user of AI.
I welcome reader feedback on this, but I’m inclined to think that aside from the most mundane, routine sorts of document — a Notice of Application containing only basic details — using AI for court submissions runs a real risk of presenting something that would involve less than competent representation.
Substantive legal writing may not be poetry or belles-lettres. But if you ask anyone who does a lot of it, they will be quick to agree that it’s a lot closer to finely crafted forms of writing than routine form filling. Just as we wouldn’t want to plonk our phone on counsel table and have AI deliver our submissions, we shouldn’t assume it’s appropriate to have AI write them.
Justice Masuhara of the BC Supreme Court, who decided one of the first hallucination cases in Canada, put it best: “generative AI is still no substitute for the professional expertise that the justice system requires of lawyers… The integrity of the justice system requires no less.”
The post The Real Problem in Hallucination Cases Is Not the Failure to Verify appeared first on Slaw.