Hila Reicher, Stav Koren, Slava Novgorodov, Tova Milo · ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 2026
DOI: 10.1145/3799682.3840817
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Evaluating Retrieval-Augmented Generation (RAG) systems in closed-domain settings --- where answers must be grounded in a fixed, domain-specific corpus --- remains an open problem. Existing approaches rely on synthetic question generation, manual annotation, or general-purpose benchmarks, each with structural limitations. We propose repurposing expert-written multiple-choice questions (MCQs) as a reusable evaluation signal for closed-domain RAG. MCQs are human-authored, curriculum-aligned assessment artifacts: they test concepts domain experts consider important, are written independently of the indexed corpus, and include expert-designed distractors that distinguish correct understanding from plausible misconceptions. Our methodology decomposes each correct answer into atomic factual claims, rewrites each option as a standalone declarative probe, and evaluates the RAG system through binary per-option judgments. We instantiate the methodology on a production university tutoring system and release a publicly reproducible benchmark. The per-option evaluation exposes differences in generation accuracy, abstention behavior, and distractor credulity that are not captured by retrieval-side evaluation alone. These results show that expert-written MCQs provide a practical and diagnostically rich evaluation resource for closed-domain RAG systems.
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
No comments yet — start the discussion below.