The Carried Reference Tunes a Switch–Multiplier Continuum in Gated (GLU) MLPs

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

A prior paper showed that a trained transformer implements its MLP gate thresholds as a couplingw · bL to a carried reference direction bL, the corpus-mean layer-normalised gate input. We use thatreference to give a functional reading of the gate in a gated (GLU) MLP. Each gate unit’s coupling tobL places it on a continuous switch–multiplier spectrum: a unit strongly (negatively) coupled to bLrests closed and behaves as a thresholded switch, while a unit orthogonal to bL rests at the activation’sknee and, to first order, computes a bilinear product of its two branch reads. The per-unit cosine tobL is the coordinate along that spectrum, and trained SwiGLU allocates units continuously acrossit, shifting toward the multiplier end with depth. Supporting this, the coupling localises to the gatebranch (across four SwiGLU models the gate_proj rows align with bL at 4–33× the up_projrows) and lives in the reference’s high-variance subspace; a bit-identical-init training ladder shows theelementwise nonlinearity is what creates the switch end, and that among four feed-forward variantsonly the gated one spans both ends with independent key and value reads. We suggest this coverage,rather than any single mode, as a reason the GLU family is an effective default, and note the readingextends to the experts of a mixture-of-experts. The account is geometric and makes no quality claim