Yingcheng Xu, Rongtian Fu, Jinliang Shi, Tong Wu, Zhi Ma, Shigang Li · · 2026
DOI: 10.1145/3828523.3857039
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Retrieval-augmented generation (RAG) improves the factuality of large language models (LLMs) by grounding responses in external knowledge, but introduces substantial pre-generation latency. We observe that both database retrieval and neural reranking can dominate time-to-first-token (TTFT) under different serving configurations, while related queries exhibit cross-query locality in retrieved chunks and high-ranked evidence after reranking. However, existing optimizations typically target retrieval or reranking in isolation, leaving this locality underexploited across the two stages. To this end, we present TwinCache, a retrieval- and reranking-aware caching system that exploits cross-query locality to jointly accelerate both stages of RAG serving. TwinCache organizes historical queries and retrieved chunks into locality-aware cache regions and constructs a compact, high-utility candidate set for each incoming query using nearest-query evidence, historical reranking feedback, and current-query similarity. This avoids expensive database retrieval on validated cache hits while reducing the number of chunks submitted to the reranker. To further reduce pre-generation latency, TwinCache introduces priority-guided reranking, which processes high-priority chunks first, overlaps subsequent reranking with LLM chunked prefill, and stops once the leading evidence chunks stabilize. TwinCache also mitigates potential false cache hits through online validation and refreshes cache regions during low-load periods using online feedback. Across diverse serving loads, TwinCache achieves up to 2.44 × the serving throughput of a vLLM-based RAG system while maintaining response quality comparable to standard RAG.
No comments yet — start the discussion below.