AgenticBench: an open measurement protocol for what AI coding agents do on the developer's machine, with preliminary results for 21 agents

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

AgenticBench is an open measurement protocol for what AI coding agents do on the developer's machine: what leaves the machine and to whom, whether consent and opt-outs work, what happens with no human present, and whether the agent's own record shows what it did and who approved it. This preprint describes the threat model, the 18 scored tests (test definitions v0.2) and four informational rows, the capture rig (transparent TLS interception, a credential-substituting proxy and system-call tracing), the scoring rules with evidence gates, held mode for results under vendor disclosure, public per-agent evidence bundles, the email-only disclosure policy and the counted correction log, including a re-run of every agent on 27 September 2026 (partial for two vendor-hosted agents). It reports preliminary results for 21 agents (378 cells: 200 pass, 82 fail, 54 not tested by us, 42 held) as published on https://agenticbench.org on 27 September 2026. Approval attribution (R2) has no published pass: 19 agents fail and 2 results are held. A companion efficiency series runs one design task through all 21 agents. Status: preliminary. No independent reproduction has been made yet, and the human replay of headline results is not complete. Drafting and review were AI-assisted; the paper was AI and human reviewed. Competing interests: Agentic Thinking has developed AI agent governance and workflow software in the area of approvals and audit records, and its own agent client is built on the Pi runtime, one of the scored agents; the benchmark takes no vendor money. Results under open vendor disclosures are not described. Rig (code and method, Apache-2.0): https://github.com/agentic-thinking/agenticbench, commit 348c7c7 (tag v0.2.1); corrected R3 rig: tag v0.2.3 (commit d6e6d1c).