Zhaohui Wang · ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 2026
DOI: 10.1145/3799682.3841132
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Some LLM-based recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is present in the scored set, either by injecting it into the candidate list or by scoring it against sampled negatives. We show that this protocol overestimates realistic performance by 92-95% in NDCG@10 across three primary Amazon datasets, so oracle results do not transfer to deployment. The root cause is a recall ceiling: realistic retrieval covers only 2-19% of relevant items at K=100 across eight datasets in three domains (Amazon products, MovieLens movies, MIND news), placing a deterministic upper bound on any closed-candidate reranker's top-k NDCG (Theorem 1, E[NDCG@k] <= Recall@|W_pi| under LOO evaluation, where W_pi is the reranker's own candidate window). Under realistic retrieval, every optimisation strategy we tested fails to produce a statistically significant improvement over the CF baseline on our primary Amazon datasets: prompt engineering, model scaling across a 168x parameter range, sequential models, three supervised neural rerankers (LambdaMART, RankNet, MLP) under a disjoint train/eval user split, a LoRA-fine-tuned LLaMA-3.2-3B reranker, and a hybrid CF+BM25+Dense retrieval with reciprocal-rank fusion. Adding text-aware retrieval raises recall on Beauty but does not lift end-to-end NDCG. Giving the LLM the upstream CF ranks and scores directly--the fairest setting we can construct--narrows but does not close the gap to CF, and does so by making the LLM adhere more closely to the CF order rather than by adding information; an LLM+CF fusion tuned on the evaluation users themselves recovers CF's performance and no more. We propose the Recall-Aware Evaluation Protocol (RAEP) as a diagnostic: classify the regime by retrieval recall, then evaluate reranking only where the ceiling permits differentiation. In the retrieval regimes we can measure (Recall@100 of 2-19%), the path to effective LLM recommendation runs through retrieval improvement rather than reranker sophistication; we do not claim this ordering holds at the higher recall, richer feature sets, and online feedback available to production systems.
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
arXiv (Cornell University) 2026 · 0 citations
Zenodo (CERN European Organization for Nuclear Research) 2026 · 0 citations
Last synced
This work has 1 recorded citation, but citing papers have not been linked locally yet.
No comments yet — start the discussion below.