Akihiro Katsuno, Yuichiro Okugawa, Toshimitsu Tanaka, Tomomi Nagao, Hiroaki Kawata · ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 2026
DOI: 10.1145/3799682.3839921
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Pointwise reranking with large language models (LLMs) evaluates query-document pairs independently and allows parallel processing, but it often struggles with absolute relevance estimation. In this work, we investigate whether non-relevance prompting provides a signal that differs from ordinary relevance prompting in LLM-based pointwise reranking. Specifically, we compute both relevance and non-relevance scores for each query-passage pair and examine two simple ways to combine them: (1) a simple product and (2) a probability-weighted RMSE to the ideal label targets. We evaluate these scores on 10 datasets from TREC and BEIR using nDCG@10 with GPT-4o-mini and Llama-3.1-8B-Instruct. With GPT-4o-mini, bidirectional scoring significantly improves over relevance-only scoring and achieves performance comparable to RankGPT, although gains over a two-prompt relevance-only control are not statistically significant. Further analysis on highly relevant DL19 pairs shows that flipped non-relevance scores often raise the scores of documents underestimated by the relevance prompt. In contrast, the effectiveness of using non-relevance scores for Llama varies across datasets, suggesting that the utility of non-relevance prompting is model-dependent.
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
No comments yet — start the discussion below.