AUTOMATED INTELLIGENT CONTRACT REVIEW

Abstract

Contract review by lawyers is an important activity, as contracts grant rights to and impose obligations on parties involved. Contracts are drafted in natural language that is not easy to read, yet all clauses of a contract need accurate scrutiny. This research investigates the use of Natural Language Processing (NLP) to implement automated contract review. Such sys-tems are already advertised by commercial vendors, but design and implementation details are not disclosed. Also, these systems are not yet widely used in legal practice. This raises two questions: to what extent is it possible to design and implement automated contract review, and what added value does such a system have for lawyers in practice?
Related work does not directly address these questions. It includes researching optimal legal text classification models, defining theoretical norm identification approaches and performing legal text extraction. No research was found that applies these topics to actual law practice, a gap this research aims to fill based on detailed research questions.
As part of this research a system (web application) is designed and implemented that per-forms contract review. It does so by first classifying the clauses in a contract with one of 10 selected clause types and subsequently looking up a policy rule based on that clause type. The policy rule is then executed, leading to approval or rejection of the clause. This result is presented as an advice to a lawyer using the system. The lawyer may correct the classification and/or overrule the result if desired.
The system uses a text classification model trained on a limited collection of contracts. From these contracts, individual clauses are extracted and a selection of clauses is anno-tated by the author based on his domain knowledge as a lawyer. This process is accommo-dated in the same web application allowing use in annotation and in review mode.
How this system performs firstly depends on the performance of the text classification model. Performance metrics are described and analyzed. The obtained dataset is relatively small, therefore domain specific word embeddings trained on a large number of legal texts are used in an alternative model. This alternative model however performs worse.
The metrics described do not fully determine the value this system may have for a profes-sional lawyer. A panel of professional lawyers used the system to review a contract. An-swers from a questionnaire subsequently filled out by the panel members indicate that de-spite classification errors and the system’s limited capability to interpret clause text, it does have added value given the lawyer’s approach to contract review. Lawyers also see potential value given perceived possibilities to improve capabilities.
The extent to which a system can review contract clauses depends on the interpretability of the clause text. Methods of improving that interpretability in interaction between system and lawyers is recommended as future research work. Recent developments around Chat-GPT seem to be an interesting area of further research. ChatGPT however still recommends seeking advise from a qualified legal professional.

Date of Award	31 Mar 2023
Original language	English
Supervisor	Martijn van Otterlo (Examiner) & Deniz Iren (Co-assessor)