
Ming Li, Ying Luo, Haoting Ye · Journal of the Association for Information Systems 2026 · 2026
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
E-commerce shopping advisors operate in product search settings where user needs are often expressed through short queries, incomplete descriptions, and specific attribute constraints. Large language models (LLMs) are increasingly used to address these challenges, yet their role in the shopping pipeline remains under-specified, especially with respect to whether they add more value in retrieval or in reranking. This paper addresses that question through a stage-based system comparison rather than a model-development study. We construct an aligned benchmark by joining Amazon Shopping Queries relevance judgments with Amazon Reviews 2023 Electronics metadata and compare BM25, dense retrieval, LLM reranking, and LLM retrieval followed by reranking, with evidence augmentation examined in a separate extension study. Across 10,961 benchmark queries, dense retrieval provides a modest improvement over BM25. In the matched-query comparison, LLM reranking preserves the same Hit@5 and Recall@5 as dense retrieval but increases NDCG@5 from 0.922099 to 0.960640. Extending the LLM from reranking to retrieval yields only a marginal additional gain, with Recall@5 increasing from 0.696785 to 0.700000 and NDCG@5 from 0.960640 to 0.961210. These results indicate that the main value of the LLM in this pipeline lies in improving the order of retrieved candidates rather than in substantially expanding retrieval coverage. Case analysis further shows that the LLM is particularly useful for product-type disambiguation and fine-grained attribute matching, while failure cases arise when the initial candidate set is weak. The extension study further shows that review-augmented reranking does not outperform metadata-only reranking and slightly reduces NDCG@5. The findings suggest that LLMs are most valuable as fine-grained rerankers in the shopping advisor pipeline, and that evidence augmentation must be selective and query-aligned.
arXiv (Cornell University) 2026 · 0 citations
Preprints.org 2026 · 0 citations
No comments yet — start the discussion below.