When explanations mislead: Explainability, erroneous compliance, and human oversight in judicial artificial intelligence

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

As courts increasingly adopt artificial intelligence and regulators place growing emphasis on explainability in high-stakes systems, explanations are often assumed to mitigate black-box risks and strengthen human oversight. This study tests that assumption in a simulated adjudicatory experiment involving legally trained participants. We conducted a 3 × 2 experiment with 150 law students who had passed China’s national legal professional qualification examination. Participants decided simulated cases after receiving an algorithmic recommendation under one of three conditions: no explanation, case-based explanation, or feature-based explanation. The experiment also varied case complexity through simple and complex trap cases containing clear algorithmic errors. The results show that explanations did not improve participants’ ability to detect or correct erroneous algorithmic recommendations. On the contrary, the feature-based explanation condition produced the highest level of erroneous compliance and a significantly higher compliance rate than the no-explanation condition. Although compliance differed between simple and complex trap cases, case complexity did not moderate the relationship between explanation type and compliance. Subjective difficulty likewise did not mediate that relationship. Within this experimental setting, these findings suggest a transparency paradox in judicial artificial intelligence: explanations did not necessarily promote scrutiny and may instead increase unwarranted reliance by making flawed outputs appear legally intelligible and trustworthy. For the governance and design of judicial artificial intelligence, the findings suggest that explanation-centered safeguards should be complemented by an adversarial design model that supports active questioning and contestation of algorithmic outputs.