How accurate is accurate enough? A systematic review and meta-analysis of data-extraction errors by large language models and human reviewers, with a simulation of their consequences for Cochrane meta-analyses

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Large language models (LLMs) are increasingly used to extract data for systematic reviews. Published evaluations mostly report task-level accuracy, which does not tell users how likely extraction errors are to change the conclusions of a meta-analysis. We will (1) systematically review and meta-analyse evaluations of LLM data extraction from research reports, estimating the distribution and type spectrum of extraction errors; (2) synthesise studies of human extraction error as a comparator; and (3) propagate the estimated error distributions through real Cochrane meta-analyses by simulation, translating task-level accuracy into the probability that meta-analytic conclusions change.