Prof Sean Rehaag recently published, “Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal”. This research entered my feed as it pertains to immigration and refugee law. Indeed, the research demonstrates interesting trends related to Federal Court decisions and Stay Motions. For example, Winnipeg has the lowest grant rates across Canada at only 16.2%. For immigration practitioners, I will briefly discuss the conclusions of this paper and my own analysis. Prof Rehaag focused this paper on statistics and his methodology. The paper offers scant analysis of the underlying numbers. The paper is invaluable for any legal researchers who are thinking of using ChatGPT or any other AI tool in their work. From my perspective, this paper is a must-read.
Disclaimer: my spouse is an academic and she has become mildly obsessed with ChatGPT since we first heard about it in Dec 2022. Rarely a day goes by when we are not talking about its uses, both good and bad. She has already had students submit papers that were written with the use of AI. I have not (yet) found a use for ChatGPT in my practice; however, I would be surprised if that day is not in the near future.
Methodology of Legal Research
Prof Rehaag devotes a significant portion of his paper on methodology, going into significant detail on exactly how the AI was used. I would encourage you to read the paper itself. His writing is both clear and concise. For example:
The specific methodology used in this study involved several steps. First, data was collected from all Federal Court online dockets from the past ten years. Next, machine learning language models were created and applied to classify and extract information from docket entries. Additional logic was then applied to infer case-level data using the classifications and extracted information. Data verification was undertaken to ensure the accuracy of the resulting dataset. Finally, statistical analysis was undertaken on the dataset.
Prof Rehaag is famous (this is not an overstatement) among immigration lawyers for his statistics and mathematical comparisons of Federal Court judges. His research has been used as the basis for many “inherent bias” arguments against judges. Personally, I have successfully used his research to have a judge recuse himself.
Getting to the meat, Prof Rehaag explains how GPTs function for the user:
GPTs are machine learning models using neural networks – specifically transformers – that are pre-trained on large quantities of text from the Internet. The initial training is unsupervised, meaning that the system does not use data labelled by human beings and then tries to match that labelling. Instead, the task that model is trained on is to predict (or calculate the probability) of the next word or sequence of text after any given sequence of text in the massive dataset of text it uses. This form of training makes GPTs particularly well-suited to generating predicted sequences of words based on an inputted prompt.
As stated above, GPTs are not useful for all types of legal research. In the context of comparing large datasets of Federal Court decisions, however, Prof Rehaag and his team at the Refugee Law Lab have explained exactly how a GPT may be used to “fine-tune” the system. Here is an example from his article:
{“prompt”: “Order rendered by The Honourable Mr. Justice John Norris at Toronto on 27‐AUG‐ 2019 dismissing the stay of execution doc.3”, “completion”: “Norris”
“prompt”: “Ordonnance rendu(e) par Monsieur le juge Scott à Montréal le 28-MAI-2012 accordant la demande de sursis d’exécution Décision déposée le 28-MAI-2012”, “completion”: “Scott”
“prompt”: “Copy of Direction of the Court (Grammond, J.) dated 17-SEP-2019 ‘These proceedings are held in abeyance until a case management conference is held in these matters.’”, “completion”: “Grammond”
“prompt”: “Ottawa 28-JAN-2022 BEFORE The Honourable Madam Justice Roussel Language: E Before the Court: Motion Doc. No. 3 on behalf of Applicant Result of Hearing: Matter reserved held by way of Conference Call Duration per day: 28-JAN-2022 from 09:03 to 10:08 Courtroom : Ottawa (Zoom) Court Registrar: Beatriz Winter Total Duration: 1h 5min Appearances: Dotun Davies representing Applicant Rachel Beaupre representing Respondent Minutes of Hearing entered in Vol. 399 page(s) 25 – 27 Abstract of Hearing placed on file”, “completion”: “Roussel”
“prompt”: “Oral directions of the presiding judge dated 13-JUL-2020 directing ‘The Minister’s motion for a stay of release will be heard on Friday, July 17, 2020 at 10:00am Eastern Time.’ received on 13-JUL-2020”, “completion”: “none”
“prompt”: “Ordonnance rendu(e) par Madame la juge Elizabeth Walker à Ottawa le 28-JAN-2019 rejetant la requête demandant le sursis interlocutoire de la Décision Décision déposée le 28-JAN- 2019 Pris en considération par la Cour avec comparution en personne inscrit(e) dans le livre J. & O., volume 811 page(s) 495 – 498 Copie de l’ordonnance envoyé(e) à toutes les parties Lettres placées au dossier.”, “completion”: “Walker”}
Prof Rehaag has been extremely generous by sharing exactly how he used the GPT to generate the dataset and how he “trained” the system. Perhaps in the future, when law students are learning legal research, they will need to be taught how to train an AI.
AI Authored Law Journal Articles
One of Prof Rehaag’s citations that really stood out was his reference to the 2021 article, “Will Machines Replace Us? Machine-Authored Texts and the Future of Scholarship”. This was co-written by one of my professors at Queen’s Law, Prof Arthur Cockfield. He was a gifted teacher and a fixture at Queen’s. In 2022, Prof Cockfield passed away and I know many of my friends from Queen’s have sent their condolences. He will be dearly missed.
Within an immigration context, we already know IRCC has been using AI to assist Officers make their decisions. We are also extremely focused on whether the use of AI is a breach of principles in Administrative Law. If the Court renders such a decision, I will be sure to report it in this space.
In my Predictions for 2023, I wrote at the end of 2022 that I expect increasing use of AI in decision-making. I would extend this prediction beyond only IRCC. I wrote that even before I knew about ChatGPT. I now expect (along with others) that 2023 is going to be the year that GPTs creep into many facets of decision-making, including decisions within the Government of Canada.
Stay Motion Grant Rates
As noted above, Federal Court decisions (2012 to 2022) rendered out of Winnipeg have the lowest grant rates in Canada at only 16.2. To compare, decisions rendered out of Toronto have a grant rate of 38.9% – a significant difference.
To a certain extent, I am in a unique position to comment on the huge discrepancy between grant rates in Winnipeg and Toronto as I have practiced in both cities and handled Federal Court matters in both jurisdictions. Prof Rehaag does not offer any explanation for the discrepancies and simply notes they are “particularly interesting”.
To be frank, the discrepancy in grant rates does not surprise me. Mainly because many (if not most) Stay Motions are started after a Deferral Request refusal or once the options dealing with the Inland Enforcement Officer at CBSA have been exhausted. In my experience, CBSA Officers in Winnipeg are much easier to deal with and more sympathetic compared with their counterparts in Toronto. For example, a Deferral Request was granted by CBSA Winnipeg to a client with criminality issues recently. I cannot imagine that CBSA in the GTA would have granted that same request. So, in that case, we did not need to use Federal Court litigation to fight for our client in Winnipeg. It was not necessary. Hence, Stay Motions in Winnipeg are only required in the most difficult cases where CBSA has decided to remove the individual. Logically, those cases are, in general, not very sympathetic and, therefore, most Stay Motions in Winnipeg would be denied. In contrast, CBSA Officers in Toronto are routinely making decisions that would violate principles of procedural fairness and, therefore, Federal Court judges would often reverse those decisions.
Postscript
If you have any experience using WordPress, you may know that each post gets a “Readability” score. For the above, WordPress is telling me that it “needs improvement”. I was thinking that I would cut and paste the above into ChatGPT and give it instructions to improve the readability of this post; however, ChatGPT is currently overwhelmed with too many users. My guess is, in the near future, WordPress may have an AI add-on, akin to Grammerly, that will make suggestions on how I should improve my writing, etc. It seems to be a question of When, not If.
The post Using AI for Legal Research appeared first on Slaw.