Monitoring Explanation Drift in Longitudinal NLP Pipelines: A Cross-Domain Study under Temporal Distribution Shift

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

This paper studies explanation drift : the temporal instability of ranked human-readable terms produced by fixed explanation mechanisms in longitudinal NLP pipelines. We compare corpus-level TF–IDF explanations with model-level explanations derived from logistic-regression coefficients in arXiv and PubMed abstracts from 2014 to 2024. The vocabulary is learned only from the 2014–2016 reference window and then frozen, approximating a deployed monitoring scenario. arXiv is used as the primary supervised setting, while PubMed provides a cross-domain robustness check with query-defined weak labels. Results show that model-level explanations drift more than corpus-level TF–IDF summaries. On arXiv, mean DriftScore is 0.50 for model-level explanations and 0.15 for TF–IDF, a 3.33-fold increase with strong statistical support (padj = 0.0040). On PubMed, the same direction appears with smaller magnitude, 0.20 versus 0.13, but is not significant after Bonferroni correction (padj = 0.0938). The proposed audit rule produces no automatic alerts under the strict configuration because observed drift does not strictly exceed the null 95th percentile, illustrating the distinction between measurable explanation turnover and drift that is sufficiently unusual to trigger operational review.