Old Stories, New Readers: Evaluating Retrieval and QA over Historical Newspapers under Diachronic Vocabulary Change

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Question answering (QA) over digital library collections allows users to obtain facts directly from digitized historical sources such as newspaper archives. Systems for this task typically follow the retrieve-then-read paradigm, in which a retriever first locates candidate passages in the collection and a reader then extracts or generates the answer from them. An answer that is never retrieved can never be read, so retrieval bounds end-to-end performance. How retrieval behaves depends on the retriever family. Lexical retrievers match the query’s surface forms, while dense retrievers match semantic representations. In the historical setting, both must bridge what we term the diachronic vocabulary gap, the mismatch that arises because users formulate queries in contemporary language whereas the target documents preserve the vocabulary of their period. This gap remains understudied. Existing historical QA benchmarks generate their questions from the source passages, so the questions inherit period wording and grant lexical matching an overlap that authentic user queries would not provide, and benchmark results may consequently overestimate real-world effectiveness. We examine this issue on ChroniclingAmericaQA, a large-scale QA benchmark over historical American newspapers. We construct a modern-query counterpart of the benchmark by re-expressing each question in present-day language under automatic validation, with answers and passages held fixed. A human preference study supports the construction. On both question sets we evaluate lexical, dense, and domain-adapted retrievers together with extractive and generative readers. We find that lexical retrieval is accurate but degrades sharply under modern phrasing, that dense retrieval is robust but less accurate, and that the degradation propagates to answer quality. Since the two retriever families fail on complementary queries, we combine them and empirically characterize the lexical–semantic trade-off, selecting the interpolation weight on held-out data. The resulting hybrid achieves the highest retrieval accuracy on both question sets. As digital libraries move from page delivery to question answering, the language their users bring belongs in the evaluation itself, and our benchmark and empirical analysis are a step in that direction.