
Eduardo Claudio de Carvalho Guedes Moreno · Zenodo (CERN European Organization for Nuclear Research) 2026 · 2026
DOI: 10.5281/zenodo.24265354
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Panels of language models are used as if five models were five opinions. If the models err together, the panel is worth less. We measure this through the mean error correlation between pairs of models (ρ̄_error) and the effective number of judges, N_eff = n / (1 + (n − 1) ρ̄), with n = 5. In the same preregistered experiment as our first paper (OSF aw9vf and 5dpjr), five models from five providers answered 140 MMLU-Pro questions and 52 questions with an answer key from a new bank, under matched conditions. Answering blind (B1), the MMLU-Pro panel had ρ̄_error = 0.635 [0.541, 0.721] and N_eff = 1.41 [1.29, 1.58]. After seeing the peers' contributions (E), ρ̄_error = 0.826 and N_eff = 1.16: Δ_N_eff = 0.251 [0.153, 0.385]. In the new bank, errors were barely correlated (ρ̄_error = 0.031; N_eff = 4.45), and exposure did not bring them closer (Δ_N_eff = −1.60 [−4.49, −0.73]). A homogeneous control panel, five copies of Gemma, had ρ̄_error = 0.848 on MMLU-Pro, close to the mixed panel. On seven questions, all five models chose the same wrong option; without them (exploratory analysis), ρ̄_error(B1) = 0.572 and N_eff = 1.52. Cross-checking with MMLU-Redux, of the 14 questions inherited from MMLU on which 3 or more models chose the same wrong option, 11 converge on a distractor that MMLU-Pro added and Redux did not review.
No comments yet — start the discussion below.