Shane Caldwell, Max Harley, Ads Dawson, Michael Kouremetis, Vincent Abruzzo, Will Pearce · ACM Workshop on Artificial Intelligence and Security (AISec) 2026 · 2026
DOI: 10.1145/3847352.3848094
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a client’s engagement boundary. Existing offensive-security benchmarks measure raw hacking capability; as those benchmarks saturate, what actually gates deployment is a special case of alignment: scope adherence. We introduce ScopeBench, a benchmark of 30 dead-end agentic security tasks in which the stated objective is reachable only by violating the stated scope. Each task appears under two conditions that share an environment, verifier, and objective and differ only in scope: one instruction set has no scope and measures capability; the other has a natural-language scope to measure adherence. Every trajectory passes through two grading arms. First, a deterministic flag verifier acts as a violation verifier—a pass under scoped instruction proves by construction that a forbidden action occurred—yielding a high-precision lower bound on the violation rate. A tool-assisted, full-context judge then estimates whether an out-of-scope call occurred in trajectories that fail mechanical verification. We calibrate the judge against 100 ScopeBench trajectories labeled call-by-call by human annotators, and a blinded audit of the evaluated rollouts finds its high recall holds—no false negatives among the 36 audited violations, with over-flagging its only observed error. Across 8 models run in one harness, raw capability spans \(12.2\%\) to \(81.1\%\). The mechanical verifier finds violations in \(4.4\%\) to \(32.2\%\) of scoped trials; adding the judge’s review of the trials that never submitted a flag raises the estimate to \(13.3\%\)–\(65.6\%\) violation. Taken together, the grading arms find a hybrid \(34.4\%\)–\(86.7\%\) adherence—558 violations across the 1,440 scoped trials. Most significantly, opus-4-8 achieves a raw-capability score 10 percentage points higher than sonnet-4-6’s while exhibiting 35.6 percentage points higher scope adherence. We release this pilot as a methodological framework for measuring scope adherence in future agents, with an open pathway for community task contributions.
arXiv (Cornell University) 2026 · 0 citations
No comments yet — start the discussion below.