From M Passes to One: A MapReduce-Style Bootstrap for Pointwise Off-Policy Evaluation in Production Recommenders

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

We present Bootstrap IIPS@K, a pointwise off-policy estimator for production recommender rankers built around a MapReduce-style bootstrap factorization. The factorization constructs all M bootstrap pseudo-samples through per-tile binary masks in a single read of the logged data, collapsing M-pass variance estimation to one pass at billion-row scale. Bootstrap IIPS@K runs directly on production logs and removes the dedicated random-session logging branch that prior industrial estimators required. The factorization composes with any bounded per-session statistic, which lets structural sequence-dependence assumptions be compared empirically from the same single read; on production logs the pointwise prior stays bounded through slate depth K = 60 while the cascade prior diverges at K = 10. A non-parametric quantile calibration of model scores keeps Bootstrap IIPS@K stable across multi-objective re-weights between ranker generations. Deployed on two LinkedIn Feed surfaces, Bootstrap IIPS@K reaches 70% offline–online parity (16/23 promotions) and Kendall’s τ = +0.29 (p = 0.04) on the broader 26-pair rank-correlation pool; the Li et al. (WSDM 2011) baseline on the same workload shows near-random rank correlation with online lifts (τ = −0.13).