Unapplied Invariants in Evaluation Instruments: A Source Audit and a Corrected Detection Null

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Evaluation instruments are becoming the evidence base for deploying agentic systems and human–agent teams; this paper is about a way those instruments fail silently. We name a defect class in which the harness computes the correct quantity and the decision does not use it: the guard is stated — in a preregistration, a docstring, or the harness's own printed output — and is not applied where the verdict, score or exit code is produced. In the sharpest case, a metric's docstring and its runtime warning both state a return value of 1.0 while the code returns 0.0: the log shows a number the metric never returned, so that warning alone misrepresents its behavior. We report eight observed instances across five instruments of a single research programme, each sized and each with a repair, plus one unrun protocol diagnostic, and seven external unknown-handling sites across four widely used open-source evaluation frameworks, including documented/implemented mismatches (lighteval, ragas, HELM, HuggingFace evaluate), each read from source at a pinned commit. A control case that states its unknown-handling policy and applies it shows the class is avoidable, and grounds our one recommendation: decide what an unknown means and record the decision where the value is produced. The class is not confined to code: an inter-rater agreement gate that cannot represent indeterminacy erases disagreement itself. Finally, a corrected detection null: five static rules — four of them scoring 8/8 on same-file pre/post-repair pairs — raise five candidates across 2,834 gating files; none survives adjudication. Direct checks detect none of the seven selected source-verified sites; a follow-up misses two third-party-reported sites in the scanned corpus. Zero confirmed rule candidates does not mean zero instances present.