Beyond Proportional Allocation: Optimal Circuit-Coupled Rank Assignment in Multi-Head Attention via Marginal Distortion Reduction and Gauge Concavity

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Existing low-rank compression methods for Multi-Head Attention (MHA), such as LoRA and SVD pruning, evaluate weight matrices in isolation. We demonstrate that this isolated evaluation creates a single-matrix blind spot, yielding a negative correlation (\(r = -0.1582\) on OpenAI GPT-2 Layer #0) between isolated truncation energy and actual MHA output distortion. To address this limitation, we establish a provable Composite MHA Distortion Bound combining row-wise Softmax Lipschitz contraction with input covariance-weighted SVD trace bounds. This bound strongly correlates (\(r = +0.9680\)) with forward output distortion, accurately identifying critical heads misclassified by isolated metrics. Furthermore, we prove that conventional proportional allocation (\(s_m \propto C_m\)) is sub-optimal due to heterogeneous spectral decay. Formulating a separable surrogate distortion objective, we prove the Discrete Marginal Allocation Theorem for Separable Surrogate Objectives: under non-increasing marginal reductions, a greedy Lagrangian water-filling algorithm achieves an exact integer global optimum. Additionally, we prove the Gauge-Concavity Identity: strict concavity of quotient manifold dimension \(p_{\text{eff}}(s) = s(d_{\text{out}} + d_{\text{in}} - s)\) implies that non-uniform allocation reduces geometric degrees of freedom (DoF) by \(\sum (s_m - \bar{s})^2\), enabling DoF reinvestment regimes (Modes A, B1, B2). Empirical evaluation on GPT-2 (124M) under a 73.44% quotient manifold dimension reduction (corresponding to 72.92% physical storage parameter reduction) shows that discrete marginal optimization reduces Layer #0 output Mean Squared Error (MSE) by 96.16% in-sample and 81.03% (Mode A/B2) to 83.65% (Mode B1) on held-out test sequences compared to uniform allocation.