Who Assures the Assurer? a Position on Meta-assurance for Ai-mediated Evaluation

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Artificial intelligence is now used to evaluate other systems. Large language models (LLMs) generate assurance profiles, control questions and test cases, judge benchmark outputs, and support automated red teams. At the same time, the evaluated systems are often LLM-based agents themselves. In this position paper, we argue that this situation creates a new type of assurance problem. Three basic assumptions of classical evaluation are broken: (1) the evaluator and the target can fail in the same way (correlated blind spots); (2) the evaluation criteria are written by the same type of model that is evaluated (circular criteria); and (3) the target can influence the inputs of the evaluator (injectable evaluation). We connect this problem to known results in computer science and propose seven principles for what we call metaassurance. Finally, we present four testable propositions for future research.