Leakage and the reproducibility crisis in machine-learning-based science

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Machine-learning (ML) methods have gained prominence in the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. We systematically investigate reproducibility issues in ML-based science. Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions. Based on our survey, we introduce a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems. We propose that researchers test for each type of leakage by filling out model info sheets, which we introduce. Finally, we conduct a reproducibility study of civil war prediction, where complex ML models are believed to vastly outperform traditional statistical models such as logistic regression (LR). When the errors are corrected, complex ML models do not perform substantively better than decades-old LR models.

Last synced 2026-08-22 14:19 UTC

• Decoupled Feature Propagation and Structural Boosting: A Robust Framework for Node Classification on Heterophilous Graphs

  Engineering Technology & Applied Science Research 2026 · 0 citations

• Benchmark-path fidelity in parallel robot-policy evaluation: a CALVIN case study

  The Journal of Supercomputing 2026 · 0 citations