Maddalena Amendola, Giacomo Medda, Alessandro Soccol, Ludovico Boratto, Mirko Marras, Raffaele Perego · ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 2026
DOI: 10.1145/3799682.3839966
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
LLM-as-a-judge has emerged as a promising strategy for evaluating recommender systems. Yet, treating simulated ratings as real feedback can make unreliable evaluation appear reliable. This risk is especially consequential in sensitive domains such as nutrition and health. In this paper, we treat the LLM-as-a-judge paradigm as a calibrated relevance simulation problem: an LLM-based simulator predicts human preferences from user and item representations, and conformal prediction converts them into uncertainty intervals. To account for the nature of recommendation, we introduce stratified conformal calibration by user rating variance. Experiments under three LLMs and seven conformal prediction methods show that point estimates of preferences hide failures on low-rating items, while conformal interval width is driven by user-level heterogeneity. Source code and data: https://github.com/maddalena-amendola/conformal_prediction_recsys.
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
ACM International Conference on Information and Knowledge Management (CIKM) 2026 · 0 citations
No comments yet — start the discussion below.