How Generative Artificial Intelligence Made “Hallucinate” Cambridge Dictionary’s 2023 Word of the Year (Or How You Will Begin to Question Whether This Article Was AI-Generated)

This article originally appeared in The Bar Examiner print edition, Spring 2024 (Vol. 93, No. 1), pp. 49–57.

By Samuel ChangWhen asked what artificial intelligence (AI) looks like, examples from popular culture include HAL 9000 in Stanley Kubrick’s 2001: A Space Odyssey, SkyNet in the Terminator films, TARS in Christopher Nolan’s Interstellar, or J.A.R.V.I.S from Marvel’s Iron Man. But current, actual artificial intelligence is, for better or worse, less terrifying and daunting and instead more accessible and prevalent than these films depict.

Development of generative artificial intelligence (GenAI) systems has led to the rise of OpenAI’s ChatGPT, a free AI chatbot through which a person can type in a question or a prompt and receive a near-­instantaneous answer on nearly any topic—whether in the form of a brief, cover letter, or school essay. As of November 2023, about 100 million people, including employees at 92% of Fortune 500 companies, use ChatGPT on a weekly basis.1 Artificial intelligence has now become accessible to the masses.

But ChatGPT is not the only GenAI system out there. GenAI systems are now widespread in today’s society: Microsoft’s Copilot, Google’s Gemini (formerly known as Bard), and Meta’s (formerly Facebook) Llama 2. Goldman Sachs analysts predict that if trends hold, investment in GenAI will approach about 1% of US gross domestic product (GDP) by 2030, which would be nearly equivalent in percent GDP to the output of all farms in the United States.2

GenAI Enters the Legal Industry

The legal industry has begun to significantly employ GenAI. According to a Bloomberg Law survey, “[s]eventeen of the biggest US law firms now allow their lawyers to use ChatGPT within certain limitations.”3 Westlaw’s parent company, Thomson Reuters, reportedly has an $8 billion “war chest to spend on acquisitions and investments in artificial intelligence,”4 using such funds to acquire Casetext and its AI legal assistant, CoCounsel; Microsoft, Ford Motor, and BigLaw firms such as DLA Piper, Kirkland & Ellis, Skadden, and Orrick, Herrington & Sutcliffe all reportedly use this AI tool.5 Other companies have chosen to institute their own AI products. LexisNexis launched Lexis+ AI, which law firms such as Baker McKenzie, Reed Smith, and Foley & Lardner purportedly employ.6 Similarly, a competing legal-specific AI system, Harvey AI, is now valued at $715 million and in use at Pricewaterhouse­Coopers and the law firm Allen & Overy, which has over 2,900 attorneys, for “legal research, drafting documents and contract analysis.”7 Finally, a startup company called Latch utilizes Chat GPT-4 to “simplify the contract review and redlining process for lawyers” and already has a “wait list of more than 80 companies, including law firms and in-house counsel.”8

To illustrate the importance of this topic to the legal profession, US Supreme Court Chief Justice John Roberts devoted the entire opening letter of his 2023 year-end report on the judiciary to the potential disruption GenAI poses. In the letter, Chief Justice Roberts details the history of the legal profession’s use of technology, including courts’ reluctance to adapt to, much less pursue, technological advancement, and considers how AI will affect the legal profession, predicting that “judicial work—particularly at the trial level—will be significantly affected by AI … [though AI] risks invading privacy interests and dehumanizing the law.”9 As one state court noted,

For the legal profession, Generative AI technology offers the promise of increased efficiency through the performance of time-consuming tasks using just a few keystrokes. For example, Generative AI can draft simple legal documents such as contracts, motions, and e-mails in a matter of seconds; it can provide feedback on already drafted documents; it can check citations to authority; it can respond to complex legal research questions; it can analyze thousands of pages of documents to identify trends, calculate estimated settlement amounts, and even determine the likelihood of success at trial.… Given its myriad of potential uses, Generative AI technology seems like a superhuman legal support tool.10

Indeed, GenAI has serious potential to further disrupt the legal profession. One Goldman Sachs’ analysis determined that about 44% of legal services could be subject to automation by GenAI systems while the remaining amount can be supplemented by GenAI. This figure is only second to the figure for office and administrative support (46% of this work may be replaced by AI) and significantly higher than the figure for all industries (25%).11

In addition, out of all the industries that may be 100% complementary with GenAI (e.g., computer/mathematical work, educational instruction, business/financial operations), only legal services had a component of its tasks that could be replaced rather than simply complemented by GenAI systems.12 Another analysis agreed that the “top industries exposed to advances in” GenAI would be legal services.13 In particular, out of 774 occupations reviewed in that analysis, occupations particularly affected by the rise of GenAI, ranked in terms of “most exposed,” were expected to be

• law professors (ranked 5th),
• arbitrators and mediators (ranked 16th),
• judges and magistrate judges (ranked 17th),
• administrative law judges and hearing officers (ranked 28th),
• judicial law clerks (ranked 35th),
• lawyers (ranked 50th), and
• paralegals and legal assistants (ranked 85th).14

GenAI is already making significant inroads in the legal profession. This article serves as a primer on how GenAI works, its problems, and the current and long-term impacts on the legal system.

The Workings of GenAI’s “Black Box”

What makes GenAI so different from existing computer programs and algorithms? For one, some lawyers have unfortunately learned through the threat of court sanctions that GenAI is not a “supercharged search engine.”15 Unlike previous iterations of AI, which could “learn to make a prediction based on data,” GenAI “is trained to create new data, rather than making a prediction about a specific dataset”16 and no longer “merely describe[s] or interpret[s] existing information”17 but mimics.18 In short, GenAI collects data from around the internet and then extrapolates—or generates—its own or unique answer, image, voice, or opening brief (!) that is often “similar, but not identical, to the original data.”19

To better understand the limitations (and dangers), a deeper discussion on the concept of GenAI is instructive. GenAI was first proposed to improve understanding of human language. In 2017, Google Research published a blog post discussing a novel theoretical artificial intelligence model to speed up and improve translation of different languages.20 At the time, existing translation models would process language sequentially (essentially one step at a time) by reading words one at a time from either left to right or right to left.21 

The 2017 Google blog post proposed instead a model (called Transformer) that would simultaneously and in parallel determine “relationships between all words in a sentence, regardless of their respective position.”22 By doing so, the proposed model captured each word’s relationships with all other words simultaneously, regardless of distance or where that other word was, which provided a richer and more nuanced context. As a result, “[i]t learns the patterns of these blocks of text and uses this knowledge to propose what might come next.”23 The model can then “generate realistic responses by hazarding guesses about which fragments of text should follow other sequences”24 via “statistically probable outputs”25 and thereby act “like a powerful version of an autocomplete tool”26 that can now “generat[e] entire new passages of text”27 and “new information that is indistinguishable from human data”28—including fake or altered case law. As one may infer, GenAI does not “have perfect recall in the way a search engine or encyclopedia might.”29 Rather, similar to a junior law firm associate, GenAI can “identify and categorize documents … sometimes within seconds”30 and provide a head start in “locating hard-to-find case law, completing analyses and answering questions clearly and succinctly.”31

To train this kind of model is a herculean task. The model, also known as the Large Language Model, is limited in usefulness until the training data crosses “a certain, very large, threshold, in their size” which requires analyzing “gobsmacking quantities of data” with significant computational power that previously did not exist.32 “Gobsmacking” is likely an understatement. ChatGPT’s GPT (Generative Pretrained Transformer)-3.5 model is based on about 570 gigabytes of data, which is derived from books, articles, and other websites openly available on the internet. What does that 570 gigabytes include? About 300 billion words. ChatGPT then uses this to generate responses based on 175 billion parameters.33 By another count, “[t]he latest AI models from Google and Meta … are likely trained on over 1trn [1 trillion] words. By comparison, the sum total of English words on Wikipedia, an online encyclopedia, is about 4bn [4 billion].”34 (Similar concepts exist for images, videos, and voice, though this article does not cover that.) One note of concern: the companies working on GenAI do not exactly know what kind of data AI is analyzing, as

it is common in the AI industry to build data sets for AI models by scraping the web indiscriminately…. These methods, and the sheer size of the data set, mean tech companies tend to have a very limited understanding of what has gone into training their models.… Tech companies don’t document how they collect or annotate AI training data and don’t even tend to know what’s in the data set…. 35

Nevertheless, without this data, GenAI has limited applications. As an example of interest to readers of this magazine, ChatGPT-3.5 “flunked” the Uniform Bar Exam (UBE).36 However, its developers claim that ChatGPT-4 achieved a UBE score in the 90th percentile after increasing the amount of data it analyzed and being able to process a novella’s worth of data, instead of only a news magazine article’s worth, at one time. And while this assertion is disputed through subsequent research, it immediately caught the attention of the general public—after all, what high-stakes test has a greater reputation for complexity and rigor than the vaunted bar exam?37 

But how AI exactly trains for and accomplishes such tasks itself is unknown. This so-called black box problem means AI’s internal workings are “invisible” and “the logic that produced the output” cannot be known38 such that “even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does.”39 This is more pronounced in GenAI, precisely because it can perform tasks “it hasn’t been explicitly trained to do.”40 In fact—and perhaps alarmingly—“[b]ecause these systems learn from more data than humans could ever analyze, even A.I. experts cannot understand why they generate a particular sequence of text at a given moment.”41 It’s even possible that GenAI can generate different responses to the same prompt because GenAI operates using a probability model and combines (or rolls the dice on) “billions of patterns in unexpected ways” such that “even if they learned solely from text that is accurate, they may still generate something that is not.”42 As noted in the Economist, while “[h]umans err, too … [t]he difference is that generative-AI tools, for now, neither explain their thinking nor confess their level of confidence. That makes them hard to trust if the stakes are high.”43

GenAI Hallucination Catches Some Lawyers Off Guard

GenAI’s responses are products of its data. For example, because GenAI analyzes internet forums and social media, GenAI picks up on stereotypes and general opinions that can result in “inherit[ing] and proliferat[ing] biases that exist in training data[] or amplify[ing] hate speech and false statements.”44 In addition, “[b]ecause the internet is filled with untruthful information, the technology learns to repeat the same untruths.”45 Unfortunately, those failures may be “harder for a person reading the text to notice, because they are more obscure.”46

Therein lies the problem. As the Economist pointed out, “[t]he fundamental problem is that language models are probabilistic, while truth is not.”47 In other words, GenAI “does not decide what is true and what is not”; it simply mimics “the way humans put words together on the internet.”48 One internal Microsoft document stated that GenAI was “built to be persuasive, not truthful” such that “outputs can look very realistic but include statements that aren’t true.”49 It may then be no surprise that GenAI would “often provide credible responses to queries based on false premises, likely due to its instruction-following training.”50

The probability model GenAI uses can therefore often lead to “dangerously plausible” fabrications and false statements “unmoored from reality,” a phenomenon known as “hallucinations.”51 These hallucinations are “confident, coherent, and just plain wrong.”52 One investigation by the New York Times found that when ChatGPT cited the New York Times, “ChatGPT simply made it up. ChatGPT doesn’t just get things wrong at times, it can fabricate information. Names and dates. Medical explanations. The plots of books. Internet addresses. Even historical events that never happened.”53 In another New York Times investigation, “Microsoft’s Bing Chat provided incorrect information that was said to have come from the Times, including results for ‘the 15 most heart-healthy foods,’ 12 of which were not mentioned in an article by the paper.”54 As an experiment, a Wall Street Journal reporter asked ChatGPT to define a fake term, “argumentative diphthongization.” It subsequently responded with five paragraphs, including a citation to a fake linguist.55

In one study, “[Chat]GPT-4 … hallucinates in 3% of its summaries, Claude 2 [developed by the company Anthropic] in 8.5% and Gemini Pro in 4.8%.”56 But when focused on legal services, this error rate grows. A Stanford University study concluded that “legal hallucinations are pervasive and disturbing: hallucination rates range from 69% to 88% in response to specific legal queries for state-of-the-art language models. Moreover, these models often lack self-­awareness about their errors and tend to reinforce incorrect legal assumptions and beliefs.” In particular, “case law from lower courts, like district courts, is subject to more frequent hallucinations than case law from higher courts like the Supreme Court.… Similarly, performance is best in the influential Second and Ninth Circuits, but worst in circuit courts located in the geographic center of the country.” Even then, “hallucinations are most common among the Supreme Court’s oldest and newest cases, and least common among later 20th century cases.” The study’s authors highly encouraged that there be “careful, supervised integration” of GenAI due to its several-year lag behind current legal doctrine and GenAI’s inability to “internalize case law that is very old but still applicable and relevant law.”57 

The Economist concluded that the lesson of using GenAI is that “no matter how fluent and confident AI-generated text sounds, it still cannot be trusted.”58 This is a warning to legal practitioners, as GenAI’s “tendency to make up things confidently is alarming—and an invitation to malpractice suits—in a profession that hinges on finding and weighing facts”59 and where “the difference between success and failure can be serious, and costly.”60 As Stephen Gillers, a legal ethics professor at New York University School of Law, warned, “lawyers cannot treat A.I. as their co-counsel and just parrot what it says.”61

But that is precisely what has happened. In the now well-known Mata v. Avianca, Inc., two lawyers and their law firm were sanctioned because the United States District Court for the Southern District of New York found that they “abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.” In this case, the court found that the sanctioned lawyers had cited, but did not verify, six different cases fabricated by ChatGPT. Instead, in at least one instance, the lawyer simply asked ChatGPT whether these cases were real. ChatGPT confirmed they were and confidently continued its hallucination by telling the lawyer “that it was available on Westlaw and LexisNexis, contrary to what the Court and defendant’s counsel were saying.” Startlingly, ChatGPT seemed to have fabricated at least one whole opinion using real district and circuit courts and real judges even though the case and its contents (including several citations within the fabricated opinion) were fake. In Mata, the court observed that ChatGPT generated opinions that had “included internal citations and quotes from decisions that are themselves non-existent” but had “stylistic and reasoning flaws that do not generally appear in decisions issued by United States Courts of Appeals,” with legal analysis that was “gibberish.” The court warned that while “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance … [m]any harms flow from the submission of fake opinions,” including the “potential harm to the reputation of judges and courts whose names are falsely invoked as authors of the bogus opinions and to the reputation of a party attributed with fictional conduct.”62

Mata was unfortunately not a one-off case. For example, in Park v. Kim, the United States Court of Appeals for the Second Circuit referred an attorney to the court’s Grievance Panel because she admitted to using ChatGPT to generate a citation to a nonexistent case in her reply brief.63 Similarly, in In Re: Thomas G. Neusom, Esq. Respondent., the United States District Court for the Middle District of Florida suspended a lawyer after the court’s Grievance Committee found that the lawyer used artificial intelligence to draft a filing, which had “inaccurate citations and fabricated authorities.”64 Lastly, in Smith v. Farwell et al., a Superior Court of Massachusetts judge sanctioned an attorney after he admitted that he did not check the “bogus citations” that were generated from an “unidentified AI system” by his associate, because he was unaware that GenAI could generate false citations.65

Court sanctions were not limited to just attorneys, however. There have been also at least three cases in which pro se litigants were warned or sanctioned after citing fake cases hallucinated by GenAI.66

Courts have also disallowed the use of GenAI for novel purposes. In In re Celsius Network LLC, the United States Bankruptcy Court for Southern District of New York excluded an expert report because it was not written by the expert himself, but by ChatGPT. The court observed that “the 172-page Report, which was generated within 72 hours, was written by artificial intelligence at the instruction of” the expert but was “not based on sufficient facts or data” and “contained numerous errors, ranging from duplicated paragraphs to mistakes in its description of the trading window selected for evaluation.”67 The court quipped that “[i]n fact, it took [the expert] longer to read report [sic] than to generate it.”68 In J.G. v. New York City Dep’t of Educ., the United States District Court for the Southern District of New York rejected the feedback from ChatGPT-4 that the law firm’s requested hourly rates were proper and stated that the

Law Firm’s invocation of ChatGPT as support for its aggressive fee bid is utterly and unusually unpersuasive. As the firm should have appreciated, treating ChatGPT’s conclusions as a useful gauge of the reasonable billing rate for the work of a lawyer with a particular background carrying out a bespoke assignment for a client in a niche practice area was misbegotten at the jump … the [] Law Firm does not identify the inputs on which ChatGPT relied. It does not reveal whether any of these were similarly imaginary. It does not reveal whether ChatGPT anywhere considered a very real and relevant data point.69

Courts and State Bars Take Action

After the court’s holding in Mata v. Avianca, Inc., and in a growing number of cases in which litigants relied on GenAI, courts and state bars began to issue standing orders and guidance relating to GenAI.

According to Bloomberg Law, as of January 2024, at least 20 federal judges have issued such standing orders.70 In a significant majority of them, the judges required that any language drafted or cases identified by GenAI be verified by a human as accurate. Furthermore, in about 10 of the standing orders, the judges required that the use of GenAI be disclosed, with some going as far as to require identifying the specific GenAI-generated text. Two judges prohibited the drafting of any filing using GenAI,71 while another required certification that the use of GenAI did not inadvertently result in disclosure of any confidential or business proprietary information.72

The first judge to issue a standing order after Mata v. Avianca, Inc., US District Judge Brantley Starr of the Northern District of Texas, explained that he issued his standing order to “warn lawyers that AI tools can create fake cases and that he may sanction them if they rely on AI-generated information without verifying it themselves” and noted that “while attorneys swear an oath to uphold the law and represent their clients, the AI platforms do not.” Notably, “[t]he judge said he considered banning the use of AI in his courtroom altogether, but he decided not to do so” after consulting AI experts. He has, however, stated that his court would avoid using GenAI.73

The United States Court of Appeals for the Fifth Circuit has proposed rules substantially similar to Judge Starr’s standing order. Under the proposed rules, attorneys would either have to certify that “no generative artificial intelligence program was used in the drafting of this document” or when “a generative artificial intelligence program was used in the drafting of this document [,] all generated text, including all citations and legal analysis, has been reviewed for accuracy and approved by a human.”74 As of April 1, 2024, the comment period regarding the rules has closed and a special committee is determining what recommendation will be made to the full court.

In the meantime, at least two state bars, California’s75 and Florida’s,76 have approved guidance on the use of GenAI, while the State Bar of North Carolina77 has proposed similar guidance. The state bars’ guidance can be summed up as the following:

• Duty of Confidentiality: A lawyer cannot put confidential information regarding a client into a GenAI system without adequate confidentiality and security protections.
• Duty of Candor, Competence, and Diligence: (1) A lawyer must review and validate the accuracy and sufficiency of GenAI’s work product before submitting to the court; and (2) a lawyer’s professional judgment and responsibility cannot be delegated to GenAI.
• Prohibition on Charging for GenAI: A lawyer cannot charge time saved by using GenAI, developing competence in GenAI (only in Florida), or costs of GenAI if it is part of the law practice’s overhead (Florida/North Carolina).

The state bars differ in some respects. The State Bar of California warned that (1) the lawyer should consider disclosure to the client when using GenAI; and (2) the “lawyer should be aware of possible biases and the risks they may create when using generative AI.”78 Similarly, the State Bar of North Carolina would require (1) that “if a lawyer delegates substantive tasks in furtherance of the representation to an AI tool … the client’s advanced informed consent is required”; and (2) that the lawyer must be educated on the benefits and risks associated with the use of GenAI.79 On the other hand, the State Bar of Florida instructed that

1. lawyers cannot delegate to GenAI any task that could constitute the practice of law,
2. lawyers should be “wary” of using an “overly welcoming generative AI chatbot” that could create a lawyer-client relationship by providing legal advice, failing “to immediately identify itself as a chatbot, or fail[ing] to include clear and reasonably understandable disclaimers limiting the lawyers obligation,”
3. lawyers should “not instruct or encourage a client to rely solely on the work product”
   of GenAI,
4. lawyers must inform the client of their intent to charge the client on the use of GenAI, and
5. lawyers “cannot claim their generative AI is superior to those used by other lawyers or law firms” in advertisements.80

Notably, the State Bar of Michigan took a different angle, saying that judges must be competent in understanding GenAI and how it is used.81

Conclusion

Returning to the title of this article, you may now understand why Cambridge Dictionary named “hallucinate” its 2023 word of the year. Cambridge Dictionary explained that the word “gets to the heart of why people are talking about AI.”82 GenAI may feel near magical, but that magic can often be unwieldy and disastrous to those who are not aware of its limits. As the public and the legal profession continue to invest in and use GenAI, courts and state bars bear a responsibility to continue issuing and refining standing orders and guidance, while legal professionals bear a responsibility to conduct their legal practice in a manner that mitigates the risks GenAI poses, upholds the trust in the justice system, and protects the public.

Speaking of hallucinations, you may be wondering if this whole article was generated by, or in consultation with, GenAI. It was not, but articles like these may very well be in the future. In fact, this article may even help train how a GenAI system replies to a similar query. For the curious, I recommend you ask ChatGPT or your favorite GenAI system to write a comparable article in its full unadulterated form. A possible prompt may be: “Write a magazine article for lawyers and judges who may be unfamiliar with the concept of generative artificial intelligence—a primer on generative artificial intelligence. Include an introductory paragraph that will hook the reader with popular cultural references. Include in the primer a discussion on how generative artificial intelligence works in layman terms. Also include in the primer how generative artificial intelligence affects the legal profession and examples of uses of generative artificial intelligence in the legal profession. Include also in the primer a discussion on the benefits, concerns, and consequences of legal professionals using artificial intelligence. Include citations whenever possible.”

But for those content to end it here, consider the lesson suggested by the Economist when reporting on Mata v. Avianca, Inc.:

Lesson learned, a tech-sceptic lawyer might conclude: the old ways are the best. That is the wrong lesson. Blaming AI for Mr. Schwartz’s error-filled brief makes no more sense than blaming the printing press for mistakes in a typed one. In both cases, fault lies with the lawyer who failed to check the motion before filing it, not the tool that helped produce it. For that is what AI is: neither a fad nor an apocalypse, but a tool in its infancy—and one that could radically change how lawyers work and law firms make money.83

Contact us to request a pdf file of the original article as it appeared in the print edition.