Benchmarking Performance Judgment in Open-Weight LLM Controller Tuning: Control Knowledge Does Not Ensure Reliable Gain Ranking

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Large language models (LLMs) are increasingly embedded in control-design workflows, yet their ability to compare candidate controllers remains uncertain. A simulator-grounded performance-judgment benchmark evaluates four open-weight checkpoints across a quadruple-tank process, a recycle reactor, and a synthetic 3-by-3 cyclic plant. Declarative control-theory accuracy is nearly saturated (98–100% pooled by plant), whereas qualitative-description accuracy for ranking PI gain sets is 46.8%, 31.8%, and 53.2%, respectively. Outputs are associated with lower gain magnitudes, but the association varies with scaling, plant, and controller sampling. Across 271 replay-stable pairs, IAE rankings agree with ISE rankings on 95.2%; under a combined simulator perturbation, 88.7% of seed–item labels are preserved. On fixed gain pairs, equations and 14-point response trajectories raise pooled accuracy only from 46.8% to 51.2% and 51.0%. All 12 prespecified trajectory-versus-qualitative intervals include zero, and none survives Holm correction. Swapping displayed trajectory values changes 9.5% of parsed choices. By contrast, a numerical scaffold derived from the same samples raises evidence-consistent accuracy from 51.1% to 88.1%, with 11 of 12 contrasts surviving Holm correction. The main bottleneck under these prompts is extracting and aggregating raw numerical evidence, not the final comparison alone. This result does not identify a unique mechanism or establish a general absence of dynamic reasoning. Simulator-based validation remains necessary before LLM judgments are used for controller tuning.