Distilling Jurisprudence through Argument Mining for Case Assessment

(1)

Assessment

Rachel Rietveld

[email protected] Law School University of Amsterdam Amsterdam, Netherlands

Julien Rossi

[email protected] Amsterdam Business School

Institute of Informatics University of Amsterdam Amsterdam, Netherlands

Evangelos Kanoulas

[email protected] Amsterdam Business School

Institute of Informatics University of Amsterdam Amsterdam, Netherlands

ABSTRACT

This paper describes an AI-based legal assistant that would support case assessment based on the extraction of arguments from court decisions. Open norms are often used in law as a way to set a leg- islative frame that allows for individual justice under the specific circumstances. By extracting the underlying arguments from existing corpus of annotated court decisions, a reliable legal framework can be formed in order to give more insight and clarity to both judges and parties in the case. This contributes to legal certainty and equality, without losing justice in individual cases. The system provides legal specialists a practical tool to help their clients in a legal procedure, without the necessity of going through all relevant case law themselves. The general public will also benefit from the available knowledge.

KEYWORDS

Open norms, Legal, Arguments Retrieval, Legal certainty, AI, justify

1 INTRODUCTION

It seems Artificial Intelligence and Intelligent Assistants already have their place in law firms, although much of this place is devoted to Document Management Systems and various Information Re- trieval systems. We suggest that under a stricter scrutiny, it would be clear most of these systems were pushed by the business need to reduce the cost of support systems, a vital need under the paradigm of the billable hour [10].

We observe that most of the Information Retrieval or Natural Language Processing problems are by large unsolved in the legal domain. The specifics of the language, or the manipulation of con- cepts and abstractions contribute to create this distance between legal texts and general literature, as observed in [1].

We make the hypothesis that lawyers internalise a summarised knowledge of case law, that allow them to assess individual situations with regards to norms, and that this process can be formulated as a knowledge extraction and summarising task[7] [6]. An Intel- ligent Assistant can leverage this acquired knowledge to come forward with the right pointers for the assessment of the individual situation, and put it in perspective of the legal landscape crafted by court decisions.

In: Proceedings of the First International Workshop on AI and Intelligent Assistance for Legal Professionals in the Digital Workplace (LegalAIIA 2019), held in conjunction with ICAIL 2019. June 17, 2019. Montréal, QC, Canada.

Published at http://ceur-ws.org.

2 OPEN NORMS IN LAW

Open norms are general, flexible terms, used to provide justice in individual cases and are therefore not restricted to specific situations that need to be written in advance. In this way, yet unknown cases can always fit in the norm.[4] Most common examples are terms like such as reasonable, fairness or equity. Some open norms are referred to as semi-open since they are stricter and clearer with less room for interpretation. The more open the norm, the more room there is for subjective interpretation and discussion.

By applying open norms, judges can deliver tailor-made solutions that suit the material status quo of the specific case. This is a rather difficult task. A judge is supposed to carefully collect the evidence, retrieve all arguments that can be of any value, without benefiting one of the parties over the other. At the same time, he needs to take into account the inequality of parties and therefore protect the weaker one. As a result, the judge may ask questions to get a better idea of the underlying case, but too much translating facts to legal grounds is not allowed. The broader scope of an open norm gives judges a desired flexibility. In the ideal situation, the judge will base his conclusion on all the relevant factors and still in the framework of the law, of the purpose of the legislator and reflect the social view of that time.

Although justice might be reached, one of the downsides of open norms is the high level of legal uncertainty, which leads to unpre- dictability and a lack of transparency.[5] Parties for example, do not know what the outcome of legal reasoning will be. A settlement is difficult to reach if a clear framework is missing. This may also lead to more legal procedures and therefore higher costs, since a judicial verdict is the only way to determine who is correct. In addition, the judge, who of course is in favour of maintaining his independence and stature, needs some sort of framework to come to a justified ruling. If such a scope is lacking, it is too difficult to come to an independent justified outcome, because then a judge’s subjective opinion can be the only guide.

In the common law system very few laws are written. The legal scope is set by a constitution, filled with open norms and some prohibiting laws. What is not written is allowed, unless there is case law on the certain topic or situation. The value of jurisprudence is significant, since it sets the scope of rules and (new) legal outcomes.

Although the civil law system is more about written laws and gives the assumption to be covering all possible situations beforehand, jurisprudence is still of great value. It may be impractical or even impossible to foresee, and at the same time describe, every set of characteristics of a case in all possible combinations. This is why also in the civil law systems open norms are used. The result in

(2)

Figure 1: From unstructured documents to data

Figure 2: Training and Evaluation

both systems is a lack of certainty how judges will rule. It is also quite demanding for judges themselves to deal with a broad and unclear range of possibilities. Would it not be better to provide all (legal) actors with a clear scope of arguments that influence the final outcome? If not defined in advance, it seems better to establish the ruling aspects, apparently found to be of value by other judges and parties. If possible, giving such insight will lead to more legal equality and certainty to all involved.

3 SYSTEM DESCRIPTION 3.1 Argument Extraction

We formulate the Argument Extraction problem as an Information Extraction task that has to be repeated once per jurisdiction and/or language. We consider that an information system capable of link- ing an argument of a case to an article of the common law, can only be valid within the boundaries of the jurisdiction where the article applies. Further usage of the system is bounded by the accessible underlying data.

Figure 3: Modelling Legal Qualification

Figure 4: System Usage and Workflow

To this end, we design an annotation process, pictured in Fig. 1, driven by data needs for the downstream tasks, as assessed by a collaborative team of AI and Law practitioners, providing the necessary expertise to shape both content and form, and align the outputs of the system with the needs of the practitioners, in a perspective that makes sense for the business at hand. This business perspective will as well guide the choice of relevant metrics to optimise, as there is no one-size-fits-all answer, but only tailored answers to specific needs.

Given the designed task and tools, annotators proceed with iden- tifying which fragments of the text qualify as Arguments, either as facts or as statute law references, and which pairs are tied together.

The pairing associates one fact to one statute law, while each fact can be paired to multiple statute law references, and each law article can be paired to multiple fact.

The training and evaluation process, pictured in Fig. 2, follows the classical work flow, under the supervision of an AI expert that provides the modelling setup and the evaluation tools. The selected metrics will provide the drive for system improvement.

(3)

We formulate the Precision-Recall balancing problem with regards to the usage of the system and the type of downstream errors we aim at minimising. It seems natural to prioritise Precision over Recall, for a knowledge that is correct but maybe not exhaustive.

We observe recent similar researches in the field of Legal Infor- mation Retrieval, using either traditional techniques: Ontologies and Combinatory Categorial Grammar in [3], [2] introduces Active Learning with non-neural Classifiers; or deep learning techniques:

Hierarchical RNNs in [11], Dense Word Embeddings and Topic Mod- eling in [9]. We also refer to recent implementation architectures, such as [8].

3.2 Modelling Legal Qualification

We consider legal qualification as the association of 3 information:

a fact or a collection of facts, a statute law article, and a judgement whether the facts constitute a breach of that article or not. Having annotated those links in the existing corpus of decisions, we formulate a system that models the legal qualification of facts. It is pictured in Fig. 3

As an AI system is also an attempt to reverse-engineer a decision scheme, it enables both prediction based on known inputs, and description of which inputs are the most likely to make a given decision.

The review and the evaluation of that model will be driven by legal expertise. The descriptive power of the model will be evaluated with regards to its capacity to summarise which clusters of facts explain a legal qualification, to show which factors can explain variation in the certainty of a qualification.

4 LEGAL SCOPE

In this paper we want to focus on open norms in labour law, more specifically, on dismissal law. One of the open norms in several legal systems is about culpable acts. If for instance, an employee violates the law or breaks the rules within the company, it is up to the judge to decide whether, in his perspective, the act is a reason for dismissal or if there are reasons to rule in favour of the employee.

Acts under the scope of culpable behaviour are diverse. It covers for example theft, being late for work, sexual harassment and breaching a non-compete clause. This wide range of specific situations, makes the outcome even more uncertain and unclear for both parties and even for the judge. First of all, it is not decided beforehand if a situation will fall under the scope of possible culpable behaviour and secondly, if it does, it is not definite the situation will be qualified as a culpable act. The outcome relies on a variety of details, from the view of both the employer and employee. What leads to fair dismissal within one company may not be severe enough within another, whilst the act can be exactly the same.

If machine learning makes it possible to come up with an insight into arguments that are of value for legal decision making, it will be a great gain. This way of extracting relevant factors from previous judge-made law, can contribute to these main principles in law, achieving greater legal equality, certainty and even justice.

5 PRACTICAL IMPACT

AI is influencing the work of legal practitioners in several ways. In addition, AI is used nowadays to give insights on how a specific

judge is most likely to rule in a case. By developing these kinds of AI aids, legal research can become less time-consuming and more effective. They can even contribute to more equal outcomes, thus legal certainty and equality. This however, is disputable, since the predictions are still mainly black boxes. Therefore, parties cannot define how their case might differ from the data the prediction is based on. Judges furthermore, miss out on the substantiation and therefore, will not be able to have the complete legal or practical framework in order for them to implement the prediction.

In our practical research, we aim for the public community to benefit from the result and the usage of legal professionals to differ from existing use seen in the reduction of time and costs, as well as in the creation of clarity in the legal practice, specifically, in the use of open norms. Parties can either solve their own conflicts once they know what is relevant to a case and what ought to be required behaviour, and thus prevent a conflict about the expected behaviour.

Consequently and in the most optimal result, this would lead to fewer justice seekers, which can be a relaxation of the judicial system. If parties still want a judge deciding on their dispute, our research will be an aid to come up with the valuable circumstances, since some details brought up by parties can be irrelevant and may lead to piles of legal documents for a judge to read through. On the other hand, parties may be incomplete and consequentially harm their case. In the Netherlands for instance, a judge can only base his ruling on facts presented by parties. He is allowed to ask questions and to order a hearing, but should always keep equality of parties in mind and is, thus, limited. Once less necessary effort is needed, this will as well lead to a relief of burden of the judicial system.

6 RESEARCH GOALS AND PRACTICAL RESULT

The goals of our research and development are multiple and depend on the user type. We aim to reach the following:

• Lawyers and legal support:Through an easily accessible desktop tool they can assess a case that is based on available facts extracted from previous case law. The goal is not to simply provide them the chance of a possible breach of the open norm, but to give them an overview of relevant practical arguments. These arguments can be used to build up their case and that will colour in the open norm with actual and practical circumstances. These users would only need to define and ultimately substantiate and/or prove which aspects are applicable. It saves them time and they do not have to compile extensive pleadings and other procedural documents that might contain irrelevant arguments that only distract from the valuable ones.

• Judiciary:For judges the system will also be available through a desktop. Similar to lawyers, they have the ability to assess jurisprudence. They can quickly see what aspect(s) is or was relevant in other comparable cases where there was a breach of the invoked open norm. Higher courts will prevail over lower judiciaries. As a result, judges have a practical tool that gives them an idea of what the common relevant and justified aspects are that colour in the open norm, since the

(4)

outcome is based on a large number of previous cases, without losing their own power to decide on the case. The higher goal is to create more legal equality and certainty.

•Legal sciences:Most research studies demand case studies.

Questions that arise, such as what the trend in jurisprudence is or whether there has been a change in ruling opinions, are mostly about open norms and can only be answered by going through a reasonable number of cases. It takes time and requires skills as constructing databases to get a sufficient insight. With the targeted practical tool research can be done in less time and might be of higher quality since the amount of investigated cases should increase.

•Public community/Parties:Law should be accessible to everyone. This includes not only access to court, but also having knowledge about what rules are applicable and what the practical results of these rights or prohibitions are. By setting a clear scope of practical behaviour that colours in an open norm, we can increase the desired accessibility. Parties would have access to an online tool and could perform a self-assessment. This could form the basis of negotiations between parties to either prevent a conflict or to solve one.

Summarised, the research goal and practical result are aimed to allow a new value proposition for different users. Instead of predicting outcomes, we are focusing on how to get to a justified outcome that is in line with common ideas. Furthermore, we will try to relief the judicial system by preventing conflicts, since potential parties should get a clearer insight of how they should behave towards each other, in order for them not to breach the applicable open norm.

7 RESEARCH PLAN

We will be applying this research on an existing open norm, culpable act, in order for us to have enough relevant case law. The legal research on this topic has been mostly completed humanly, so we tend to know what relevant practical arguments judges use to support their ruling in case of the underlying question whether certain behaviour is culpable. In this way we can train the system, and see whether the generated outcomes are correct.

8 METHODOLOGY 8.1 Data Sources

We consider using Court Records as a primary source of data. A properly formed court ruling will motivate the final decision by elaborating on how the given facts were legally qualified. The prolif- eration of "Open Data" policies increase the amount of information made public each year. Most of this data comes unstructured, and most of the structured data provide mainly a structure for document meta-data, such as bibliographical details, time stamps and document origin. We have to consider that our main source of data will consist of raw text files.

8.2 Multidisciplinary Teamwork

The annotation project receives a lot of attention with regards to aligning its design with the needs of the downstream tasks. We form an open collaboration team able to tackle both the content

(legal experts, legal operations experts) and the form (AI experts) in a holistic manner. Cross-education of team members on key topics is essential to lower the walls and break free from a top-down waterfall approach.

8.3 Research Questions

RQ1. How to arrange annotations of Court Decisions? How to use unstructured legacy annotations?

RQ2. How to extract arguments from court decisions? How to optimise on a business relevant metric? How to assemble a knowledge base? How to evaluate accuracy and complete- ness?

RQ3. How to model a decision boundary based on facts associa- tions? How to model facts importance? How to model facts composition?

9 CONCLUSION

In this paper, we introduced a novelty Intelligent Assistant based on Artificial Intelligence techniques, that would leverage the knowledge contained in court decisions in order to support stakeholders of the judicial system at large in assessing the effectiveness of policies, the actual usage of legislation and norms, the merits of new cases with regards to jurisprudence.

We associate the tool to multiple stakeholders of the judicial system with different use cases where the additional information benefits all parties.

We recognise that the challenge lies not only on the technology itself, but on the capacity to produce a fruitful collaboration between the world of legal work and the world of Artificial Intelligence, and the co-engineering of a solution that is driven by business or societal needs.

REFERENCES

[1] Piyush Arora, Murhaf Hossari, Alfredo Maldonado, Clare Conran, Gareth JF Jones, Alexander Paulus, Johannes Klostermann, and Christian Dirschl. 2018.

Challenges in the development of effective systems for Professional Legal Search.

(2018).

[2] Cristian Cardellino, Serena Villata, Laura Alonso Alemany, and Elena Cabrio.

2015. Information Extraction with Active Learning: A Case Study in Legal Text.

InComputational Linguistics and Intelligent Text Processing, Alexander Gelbukh (Ed.). Springer International Publishing, Cham, 483–494.

[3] Mauro Dragoni, Serena Villata, Williams Rizzi, and Guido Governatori. 2016.

Combining NLP Approaches for Rule Extraction from Legal Documents. In 1st Workshop on MIning and REasoning with Legal texts (MIREL 2016). Sophia Antipolis, France. https://hal.archives-ouvertes.fr/hal-01572443

[4] H.L.A. Hart. 1994.The Concept of Law. Oxford: Oxford University Press.

[5] Atilla Kun. 2018. How to Operationalize Open Norms in Hard and Soft Laws:

Reflections Based on Two Distinct Regulatory Examples.International Journal of Comparative Labour Law and Industrial Relations 34, no. 1(2018).

[6] Gloria T. Lau, Kincho H. Law, and Gio Wiederhold. 2005. Legal Information Retrieval and Application to e-Rulemaking. InProceedings of the 10th International Conference on Artificial Intelligence and Law (ICAIL ’05). ACM, New York, NY, USA, 146–154. https://doi.org/10.1145/1165485.1165508

[7] K Tamsin Maxwell and Burkhard Schafer. 2008. Concept and Context in Legal Information Retrieval.. InJURIX. 63–72.

[8] Marc Opijnen, Nico Verwer, and Jan Meijer. 2015. Beyond the experiment: the eXtendable legal link eXtractor. InWorkshop on Automated Detection, Extraction and Analysis of Semantic Information in Legal Texts, held in conjunction with the 2015 International Conference on Artificial Intelligence and Law (ICAIL).

[9] Srishty Saha and Karuna P Joshi. 2019. Cognitively Rich Framework to Automate Extraction and Representation of Legal Knowledge.UMBC Faculty Collection (2019).

[10] JD Supra. 2018. When Will the Billable Hour Die? Conversation with Law Firm Change Agent John Chisholm (Part Two). https://www.jdsupra.com/legalnews/

when-will-the-billable-hour-die-95132/

(5)

[11] Rao Xi and Ke Zhenxing. 2018. Hierarchical RNN for Information Extraction from Lawsuit Documents. InProceedings of the International MultiConference of

Engineers and Computer Scientists, Vol. 1.