
Mohammad Naser Sadrmanesh · Figshare 2026 · 2026
DOI: 10.6084/m9.figshare.35004088
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
OverviewThis technical report presents the architecture, methodology, statistical evaluation procedures, and empirical benchmark results of llm-search-eval v0.5.0, an open-source Python library for evaluating e-commerce product search systems.The framework combines classical Information Retrieval (IR) metrics with multi-judge Large Language Model (LLM) relevance assessments. It is designed to address practical challenges in offline search evaluation, including sparse human relevance judgments, label depletion, evaluator variability, statistical uncertainty, and evaluation workflow integration.System ArchitectureThe report describes a four-stage label resolution hierarchy:Human Gold Labels: Human-provided query relevance judgments take precedence over automated assessments.Content-Addressed Judgment Cache: Previously computed judgments are reused through content-addressed caching.LLM Judge Panels: Unlabeled and uncached query-product pairs can be evaluated by configured LLM judges.Fallback and Coverage Tracking: Unresolved items receive a computational fallback score while coverage gaps are explicitly recorded. This fallback does not represent an explicit judgment of irrelevance.The report also documents model-version fingerprinting, evaluator drift monitoring, inter-rater agreement, and pairwise preference evaluation.Evaluation MethodologyThe framework implements standard rank-aware Information Retrieval metrics, including DCG, nDCG, Precision, Recall, Mean Reciprocal Rank (MRR), and Mean Average Precision (MAP).Statistical procedures include paired sign-flip permutation testing and query-level bootstrap confidence intervals. The report documents a two-sided permutation test using 2,000 random permutations with seed 13, and 95% percentile bootstrap confidence intervals based on 1,000 query-level resamples with replacement.Empirical BenchmarkThe report presents a bilingual electronics product-search benchmark comprising:10 queries: 5 English and 5 Persian22 products30 human gold relevance labels49 unique judged query-product pairs19 additional pairs assessed using a deterministic three-judge mock panelThree retrieval approaches are compared: BM25, Semantic retrieval, and Hybrid retrieval.The reported nDCG@10 scores are:BM25: 0.5305Semantic: 0.9865Hybrid: 0.9775The report also presents confidence intervals, paired comparisons against the BM25 baseline, and language-specific benchmark results.LimitationsThe benchmark is limited in scale and uses synthetic mock judgments for filling evaluation gaps. It does not establish general performance superiority across broader e-commerce search workloads. The report also identifies benchmark author bias, the absence of online behavioral validation, and the lack of latency and memory benchmarks as limitations.Software and ReproducibilityThe report documents llm-search-eval v0.5.0 and provides links to its source repository, documentation, package distribution, benchmark dataset, and software archive.Software DOI: 10.5281/zenodo.23944666Report DOI: 10.6084/m9.figshare.35004088Peer-review status: Technical report, not peer-reviewed.
AI Magazine 2026 · 0 citations
No comments yet — start the discussion below.