The open-access journal PeerJ Computer Science has published a serious analysis of the use of natural language processing to predict outcomes in cases before the European Court of Human Rights. The mainstream media has predictably oversimplified the interesting study. There is a useful Guardian piece here which is broadly accurate.
In a real boost for worldwide awareness of SCL, BBC World interviewed SCL’s President Richard Susskind OBE FRSE when discussing the implications of the application of AI to the practice of law in the context of the study’s findings. You can download the five-minute joint interview with Orlando Conetta from Pinsent Masons here (it is a wetransfer.com url for an MP4 file). Richard and Orlando strive to put AI in a real-world context. In particular, their interview goes into the implications for the profession of the implementation of AI. For even greater context, you might like to listen to Richard’s SCL lecture in this podcast.
The published study itself is worthy of serious attention. You can access it here. Here’s the abstract:
Recent advances in Natural Language Processing and Machine Learning provide us with the tools to build predictive models that can be used to unveil patterns driving judicial decisions. This can be useful, for both lawyers and judges, as an assisting tool to rapidly identify cases and extract patterns which lead to certain decisions. This paper presents the first systematic study on predicting the outcome of cases tried by the European Court of Human Rights based solely on textual content. We formulate a binary classification task where the input of our classifiers is the textual content extracted from a case and the target output is the actual judgment as to whether there has been a violation of an article of the convention of human rights. Textual information is represented using contiguous word sequences, i.e., N-grams, and topics. Our models can predict the court’s decisions with a strong accuracy (79% on average). Our empirical analysis indicates that the formal facts of a case are the most important predictive factor. This is consistent with the theory of legal realism suggesting that judicial decision-making is significantly affected by the stimulus of the facts. We also observe that the topical content of a case is another important feature in this classification task and explore this relationship further by conducting a qualitative analysis.
There have been varying reactions to the study among legal practitioners. The 79% success rate has certainly evinced both positive and negative reactions, with five-star cynics pointing out that coin-tossing gets you to 50%. It is fairerr to point out that the writers claim only that ‘a text-based predictive system of judicial decisions can offer lawyers and judges a useful assisting tool’.
