AI’s legal limit: Why machines can’t deliver justice

While LLMs can generate outputs that mimic legal reasoning, they operate without semantic understanding. This limitation is akin to John Searle’s Chinese Room thought experiment, where a system produces coherent responses without comprehending their meaning. LLMs analyze patterns in training data to generate plausible legal advice but lack the ability to grasp the intent, purpose, or ethical considerations behind legal provisions. As such, their outputs are fundamentally syntactic rather than semantic.

The emergence of Large Language Models (LLMs) like ChatGPT has sparked considerable interest in their potential applications across various domains, including law. These AI systems, capable of processing complex language tasks, raise the question: Can LLMs apply the law? Henrique Marcos's thought-provoking paper, "Can Large Language Models Apply the Law?" published in AI & Society (2024), critically examines this possibility. The study explores two interpretations of legal application - inferential and pragmatic - and argues that LLMs, despite their linguistic capabilities, fall short of truly applying the law. By addressing the conceptual, practical, and ethical dimensions of this topic, the paper provides a nuanced understanding of AI's role in legal reasoning.

Theoretical foundations of legal application and LLM capabilities

Marcos bases his argument on D'Almeida's theory of legal application, which identifies two primary dimensions: inferential and pragmatic application. Inferential application involves mental reasoning to deduce whether a legal provision applies to a specific case. For example, interpreting whether a prohibition on vehicles in parks includes bicycles requires contextual reasoning about the law's intent and scope.

Pragmatic application, by contrast, involves external actions taken by legal authorities, such as judges, to resolve disputes authoritatively. This dimension relies on a collective understanding within a legal community and the "game of giving and asking for reasons" (GOGAR), where practitioners justify and refine legal interpretations through communal dialogue.

While LLMs can generate outputs that mimic legal reasoning, they operate without semantic understanding. This limitation is akin to John Searle's Chinese Room thought experiment, where a system produces coherent responses without comprehending their meaning. LLMs analyze patterns in training data to generate plausible legal advice but lack the ability to grasp the intent, purpose, or ethical considerations behind legal provisions. As such, their outputs are fundamentally syntactic rather than semantic.

Pragmatic application presents an even greater challenge for LLMs. Legal interpretation and application occur within a communal framework shaped by cultural, social, and linguistic norms. For example, judges engage in iterative dialogues to refine their understanding of legal principles, an inherently human process rooted in shared experiences and collective reasoning. LLMs, as non-human entities, cannot participate in these communal practices, limiting their role in pragmatic law application.

Why LLMs cannot truly apply the law

Lack of Semantic Understanding

One of the key reasons LLMs cannot apply the law is their inability to understand semantics. While they can produce text that appears coherent and contextually relevant, this capability stems from statistical correlations rather than genuine comprehension. For instance, when asked whether bicycles are included in a vehicle prohibition, an LLM might generate an accurate answer based on its training data. However, it does not understand the reasoning or implications behind the prohibition, nor can it evaluate the law's broader societal impact. This distinction between syntactic manipulation and semantic understanding highlights a fundamental limitation of LLMs in inferential legal application.

Inability to Engage in Communal Practices

Pragmatic law application requires active participation in a collective process where legal norms are interpreted, challenged, and refined. This process involves justifying actions and decisions within a framework of shared reasoning, a capability that LLMs lack. Legal practitioners, for example, engage in GOGAR to reach consensus on the interpretation of complex legal issues. LLMs, operating outside the linguistic and cultural contexts that shape these discussions, cannot contribute meaningfully to this process. Their outputs, while potentially useful, do not participate in the communal activities that define pragmatic law application.

Absence of Membership in the Linguistic Community

Legal application is deeply embedded in the linguistic and cultural practices of a community. Membership in this "linguistic community" requires more than the ability to generate text; it involves engaging with the norms, values, and shared experiences that shape collective understanding. LLMs, as tools that generate outputs based on training data, do not possess the intentionality or participatory capacity necessary to belong to this community. This limitation further underscores their inability to apply the law pragmatically.

Implications for legal systems and the role of LLMs

The findings of the study have significant implications for the integration of LLMs into legal systems:

• Enhancing Efficiency, Not Replacing Judgment: While LLMs can assist by summarizing legal documents, generating drafts, or synthesizing case law, they cannot replace human judgment. Their role should be to complement, not supplant, the nuanced reasoning of legal practitioners.

• Ethical and Accountability Concerns: The reliance on LLMs for legal tasks raises ethical questions, particularly around accountability. If an LLM generates biased or erroneous outputs, determining responsibility becomes complex. Clear guidelines and human oversight are essential to address these concerns.

• Regulatory Frameworks: As the use of AI in legal contexts expands, regulatory frameworks like the EU AI Act become increasingly important. These frameworks emphasize transparency, fairness, and accountability, ensuring that AI systems align with societal values and legal principles.

• Potential for Future Integration: While current LLMs lack the capabilities to apply the law, advancements in AI could bridge some of these gaps. Future models that incorporate semantic understanding and contextual reasoning might play a more significant role in legal processes. However, this integration must be guided by ethical considerations and a commitment to preserving human agency.

Balancing innovation and responsibility

Marcos's study concludes that LLMs cannot currently apply the law in either inferential or pragmatic terms. Their lack of semantic understanding and inability to engage in communal practices limit their role to that of auxiliary tools. However, this does not diminish their potential to enhance legal efficiency and accessibility. By addressing their limitations and aligning their use with ethical and regulatory standards, LLMs can serve as valuable assets in legal systems while maintaining the integrity of human decision-making.

The question of whether LLMs can apply the law is not merely theoretical - it has profound implications for the future of legal practice, governance, and societal trust. As AI technology continues to evolve, it is imperative to critically assess its capabilities and limitations, ensuring that its integration into sensitive domains like law is both responsible and equitable.