Claim–Evidence Audit Protocol (CEAP) A Contrastive Reporting Layer for Black-Box AI Evaluation

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

The Claim–Evidence Audit Protocol (CEAP) is a methodological perspective and critical synthesis for black-box AI evaluation. It addresses a recurring inferential problem: observable AI behavior is often used to support stronger claims about capabilities, internal states, mechanisms, robustness, or latent properties without making explicit which competing explanations the evaluation actually distinguishes. CEAP introduces a reusable contrastive reporting layer organized around a claim–interface–alternative triple ((C,T,\mathcal H)), where (C) is the target claim, (T) is the observational or interventional interface, and (\mathcal H) is the declared space of claim-relevant alternatives. The protocol requires evaluators to specify the target claim, identify claim-opposing alternatives, state the discriminatory role of each central test, audit dependence among evidence sources, report residual ambiguity, define a claim ceiling, and identify the next discriminating experimental need. The framework includes a six-level diagnostic hierarchy from observed outputs to latent or constitutive interpretations, a nine-step audit procedure, and the diagnostic categories D0 (non-diagnostic), D1 (partially discriminating), and D2 (claim-identifying within a justified declared scope), together with a scope-conditional warning. CEAP is not proposed as a new theory of validity, identifiability, robustness, red teaming, causal inference, or scientific evidence. It is positioned as complementary to validity-centered AI evaluation, strong inference, severe testing, construct validity, partial identification, measurement invariance, metamorphic testing, assurance cases, red teaming, and broader evaluation-science approaches. Two analytic illustrations demonstrate the framework: a finite-state example showing why persistence under pressure does not by itself identify persistent internal state, and a synthetic refusal-robustness example showing how successful adversarial evaluation can support bounded robustness claims without identifying a unique internal safety mechanism. The paper also applies CEAP retrospectively to Directional Black-Box Tomography, the Declaration–Persistence Gap, and the Relational Objectivity Principle, preserving earlier methodological contributions while narrowing claims that exceeded what the available interfaces could discriminate. The central recommendation is simple: scientific evaluations should report not only what survived a test, but which claim-opposing alternatives the test could discriminate, which remain unresolved, and what experiment would be informative next.