RAGBench-X: A Reproducible Framework for Controlled Evaluation of Retrieval-Augmented Question Answering

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

RAGBench-X is a reproducible framework for controlled evaluation of retrieval-augmented question answering (RAG) systems. It is designed to make retrieval and augmentation choices explicit, configurable, traceable, and repeatable, enabling systematic investigation of how different retrieval components affect question-answering performance. The framework supports a retrieval-free language-model baseline, dense retrieval using Sentence Transformers, sparse BM25 retrieval, reciprocal-rank fusion (RRF), cross-encoder reranking, query reformulation, multi-query retrieval, context compression, source attribution, evidence verification, and structured experiment logging. It provides a FastAPI service and Streamlit research interface, together with persistent experiment storage and raw per-question artifacts. The current evaluation corpus is a pinned subset of FastAPI documentation containing 60 source-linked candidate questions. These questions are currently marked for human review and are not yet a human-verified evaluation benchmark. Engineering validation included 36 automated tests, seven Streamlit dashboard views, and an eight-question retrieval-only evaluation using real Sentence Transformers embeddings, Qdrant, BM25, reciprocal-rank fusion, and cross-encoder reranking. The retrieval check recorded Recall@5 = 1.00, MRR@5 = 0.8125, and mean retrieval latency of approximately 0.442 seconds. These measurements are reported as preliminary engineering validation and should not be interpreted as generalizable research findings. RAGBench-X provides the experimental infrastructure for a subsequent controlled empirical study comparing retrieval and augmentation strategies under a fixed corpus, question set, generation model, and evaluation protocol. The framework is intended to support reproducible research in retrieval-augmented generation, information retrieval, and domain-specific question answering.