Why GLUs Work?

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

The Gated Linear Unit (GLU) and its variants, particularly SwiGLU, have become the de facto standard for feed-forward networks in modern Large Language Models (LLMs). Despite their widespread adoption, the fundamental mechanisms underlying their superior performance over plain MLP activations—and their occasional instability under depth scaling—remain poorly understood. Through extensive controlled experimentation across 350+ models, 7 depth scales, and 2 datasets, we demonstrate that the success of GLUs lies in their implicit quadratic self-gating behavior, where the Hadamard product induces an effective x² growth factor. We validate this hypothesis by using single-projection MLP variants—MLP-S10, MLP-Quadratic, and MLP-Squared-ReLU—which replicate the performance of full GLUs, confirming that quadratic amplification is the dominant driver of performance. Furthermore, we identify a critical structural flaw in existing gated designs: the down-projection weight w is left mathematically unguarded in the upstream gradients, leading to uncontrolled growth and divergence in deep networks. We derive this flaw formally and introduce Wal10 and Wal10-SiLU, GLU variants where the activation cap is applied after the down-projection. This simple architectural change introduces a shared sech²(·) factor in all gradients, providing an automatic, self-regulating brake that prevents runaway growth for all three matrices. Experiments across scales show that under normal training, Wal10 variants match the performance of SwiGLU. However, when subjected to extreme conditions in a 100L Transformer, Wal10 caps activation values and suppresses dangerous gradients, while SwiGLU and SiTU-GLU allow them to grow for all three matrices; thus, even with soft capping, SiTU-GLU fails to prevent activation blow-up. Our results suggest that Wal10 offers a reliable balance of expressivity and stability for deep neural networks.