Comparative Analysis of Rule-Based, Deep Learning, and LLM Approaches for Causality Extraction: A Unified Benchmark

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Causality extraction—the identification of cause–effect relations in natural language text—has been approached through three successive methodological paradigms: rule-based systems relying on lexical and syntactic cues, deep learning models exploiting neural architectures and pre-trained encoders, and, most recently, large language models (LLMs) applied through prompting or fine-tuning. Each paradigm has been evaluated on different datasets, under different annotation schemes, and with different metrics, making it difficult to determine how far apparent performance differences reflect genuine methodological advantages rather than benchmark artifacts. This study addresses that problem by constructing a unified benchmark for comparative evaluation of rule-based, deep learning, and LLM approaches to causality extraction. We consolidate multiple existing corpora—covering news, biomedical, and safety-incident text—into a harmonized annotation format, re-annotate a stratified subset to ensure consistent treatment of implicit and explicit causal relations, and evaluate representative methods from each paradigm under identical preprocessing, train–test splits, and evaluation protocols. We report span-level precision, recall, and F1, document-level F1, calibration error, inference cost, and annotation efficiency, and we conduct cross-domain transfer experiments to assess generalization. Results are analyzed to disentangle the contributions of architecture, training data, and task formulation, with particular attention to the conditions under which each paradigm retains a comparative advantage. We further examine error profiles across paradigms, identifying whether failures are concentrated in explicit versus implicit relations, local versus distributed constructions, and domain-specific versus general-language contexts.