Evaluating Jev for Cost-Effective Candidate Verification: An Exploratory Study

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Extended reasoning can increase response time and computational cost without improving an answer. Models may revisit unproductive steps or build further reasoning on an earlier mistake. This study evaluates whether Jev, a probabilistic decision maker, can provide an inexpensive error signal to help decide when additional reasoning is needed. We compare Jev with GPT-5.6 Luna on generated ordering and logic-grid puzzles and on the official released ZebraLogic test set. The generated questions are four-choice items with one correct answer among four. The official test set is used open-ended: Luna produces the full grid, Jev checks it without the solution key, and Luna may revise once when Jev does not judge the answer consistent. On the generated multiple-choice problems, Jev is 5 to 60 percentage points more accurate than Luna with reasoning disabled, at lower measured API cost, and every observed top-choice disagreement identifies a Luna error. On a stratified sample of 200 official ZebraLogic puzzles, exact-grid accuracy rises from 41.0% with reasoning disabled to 90.5% with high-effort reasoning. A revision pass adds two points in both settings, but a revision-only control shows the gain comes from the extra attempt, not from Jev's information: on the answers Jev flags, a plain second look is at least as accurate as the Jev-informed revision (16 versus 13 correct without reasoning, and 70 versus 70 at high effort). Jev detects 93% of the weak reasoner's errors with 92% precision, but only 10% of its high-effort flags identify errors. These results motivate using Jev or similar decision models in a model harness: software that controls when to continue, reconsider, retrieve evidence, or allocate more reasoning. An early check could help avoid spending further computation on a mistaken line of reasoning. The experiments measure checking of final answers, however, and do not measure reductions in hallucination or repetitive reasoning inside a harness. On the official benchmark, extra reasoning accounts for most of the accuracy, and the checker's information does not improve on a second attempt. Larger studies with balanced answer positions, selective triggers, and step-level feedback are needed to test whether checking improves accuracy or reduces total cost.