Evaluating LLMs for named entity recognition: a cost-benefit analysis in the legal domain

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Within the legal domain, named entity recognition (NER) is an essential pre-processing tool for important tasks such as information extraction and redaction. Research to date has focused on rule-based approaches or machine learning strategies that require significant time and effort to train through labelled corpora. Large, multi-purpose generative large language models (LLMs) have been successfully employed on natural language processing (NLP) tasks with no prior supervised training and may represent a way forward in addressing these existing bottlenecks. Despite promising results in certain NLP use cases, the Few-Shot LLM performance on unfamiliar, domain-specific NER tasks has, to date, remained uncertain. In this legal clause NER task, we evaluate a Few-Shot, Chain-of-Thought (CoT) prompting strategy for OpenAI’s GPT-4 LLM on a new CUAD-NER dataset that we have repurposed for the task of Person and Organisation entity identification. In a comparative study, we demonstrate that the GPT4 + Few-Shot CoT prompt strategy outperforms existing state-of-the-art models, achieving a 5% percentage point improvement in Micro F1 score over the best performing BERT base (cased) competitor. This study also takes the novel approach of assessing the business feasibility of deploying each of these machine learning models through a detailed cost–benefit analysis that measures the trade-off between performance metrics and low and high usage running costs.