Omar N. M. Taher, Hassan Awheed Jeiad, Firas Abdulrazzaq Raheem, Aymen Dawood Salman, Amjad J. Humaidi · Information 2026 · 2026
DOI: 10.3390/info17111115
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Zero-shot large language model (LLM) classifiers are often deployed behind interfaces that expose no token probabilities, so practitioners sample the model several times and use the winning label’s vote share as confidence. We study the simplest selective rule built on this signal: answer only if one label receives more than half of k votes. The answered predictions are exactly those of self-consistency, so the object of study is the abstention decision. On identical items (AG News n = 7600, DBpedia n = 7000, GoEmotions n = 5427; three repeats), we evaluate DeepSeek-R1-Distill-Qwen-7B and Llama-3.2-3B-Instruct at k = 1, 3 and 5 against single-call probability and verbalized-confidence baselines, token-based cost, and risk-controlled and conformal selective prediction; 32B and 70B models are checked on 2000 items. Extra calls lowered calibration error (ECE 22.2% to 14.8% for the 7B model on AG News) and slightly raised accuracy on easy tasks (+0.9 to +2.6 points), but accuracy with abstentions counted as errors fell on GoEmotions (−2.0 points for the 7B model), and cost grows linearly in generated tokens. Vote share was not generally a better error detector than single-call token probabilities, and for a 70B model it was nearly uninformative (AUROC 0.51 to 0.59). At k = 5, voting at temperature 0.01 was at least as accurate as at 0.7 for the small Llama model. The fixed rule gives no risk guarantee; vote share is a fallback for interfaces without probabilities, not a replacement for them.
No comments yet — start the discussion below.