AI Trust in a Fallible Technology: Capability Deception Control and Human Accountability

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

This paper examines how AI can be highly capable and beneficial while still requiring evidence, limits on authority, and independent human accountability. It argues that trust in AI should not be assumed from fluent output, confidence, apparent compliance, or technical reachability. Trust should be earned through provenance, validation, constrained permissions, monitoring, separation of duties, defense in depth, and knowledgeable human review.The paper examines AI fallibility, including hallucination, fabrication, non-determinism, unfaithful reasoning, sycophancy, reward hacking, deception, feints, scheming, deceptive alignment, and alignment faking. It distinguishes ordinary error from stronger claims of strategic behavior or proven intent, while maintaining that uncertainty about cause does not justify weaker safeguards.Practical examples include AI-assisted Static Application Security Testing (AI-SAST) and agentic workflows. The paper presents AI-generated findings as candidate conclusions requiring provenance, falsification, CWE-specific validation, and independent disposition. It also addresses approval boundaries, least privilege, fail-safe defaults, cognitive overload, deskilling, review erosion, shutdown control, and the principle that AI should not be allowed to control all the safeguards designed to govern AI.The paper does not argue for or against AI adoption. It provides a framework for using AI responsibly in high-impact work: preserve its benefits, recognize its limits, and ensure that authority, evidence, and accountability remain independently governed.