Retrieval Granularity as Evidence Design in Small-Model RAG Question Answering: A Diagnostic HotpotQA Study

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Retrieval-Augmented Generation (RAG) has become a practical approach for question answering over external corpora, particularly when answers should be grounded in source documents rather than generated only from model parameters. While recent large language models can process increasingly long contexts, they do not remove the need for selecting, organizing, and auditing evidence, especially when systems rely on smaller local models for privacy, cost, or deployment constraints. In this paper, we frame retrieval granularity as an evidence-design variable for answer grounding in small-model RAG question answering. After a brief exploratory NewsQA phase that motivates the error categories, the main study uses the HotpotQA distractor validation split with 7405 hard multi-hop questions and sentence-level supporting-fact annotations. With Qwen3-8B as the fixed generator, we compare closed-book, fixed-budget whole-context, retrieved-context, gold-document, and gold-supporting-fact conditions while varying retrieval granularity, retriever type, and context budget. Retrieved context substantially outperforms closed-book answering and the 1024-token fixed-budget whole-context condition but remains below gold-document and gold-supporting-fact upper bounds, indicating that retrieval, generation, and evaluation limitations should be analyzed separately. Sentence-level retrieval under-recovers multi-hop evidence, especially for questions with three or more supporting facts, while paragraph-level and moderate token-level chunks recover substantially more complete evidence. In the full condition matrix, hybrid retrieval with 256-token chunks and no overlap achieves an F1 of 0.6816 with a supporting-fact recall of 0.9609, compared with an F1 of 0.6166 and supporting-fact recall of 0.7801 for BM25 sentence retrieval. Additional ablations show that fixed-budget whole-context performance is strongly affected by truncation, that overlap has little practical effect under the tested 1024-token budget, and that a stronger BGE dense retriever improves the best retrieved-context F1 to 0.7027. These results align with a diagnostic perspective on chunking: using evidence at a task-appropriate level of granularity can improve grounding, auditability, and answer quality, but the observed patterns should be interpreted within the HotpotQA distractor setting, fixed generator, and tested context budgets.