
Zhongren Wang · Zenodo (CERN European Organization for Nuclear Research) 2026 · 2026
DOI: 10.5281/zenodo.24213751
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
This study tests whether information from attention heads within a Transformer layer helps a router decide how many additional heads to activate at a matched average head budget. The four-layer causal Transformer uses two always-on heads. The router can use their outputs and attention-weighted summaries of pairwise relations carried across layers. On the four-clue parity task, switching from fixed four-head routing to variable head counts raises mean test accuracy from 82.83% to 91.85%, with 3.988 active heads per query position on average. Removing the relation summary lowers accuracy in all three paired training seeds. On the key-value retrieval task, all three configurations use variable head counts and average approximately four active heads per query position. Across eight paired seeds, mean four-key exact-match accuracy is 89.02% with hidden states alone, 76.32% after adding the always-on head outputs, and 89.00% after also adding the relation summary. The additional routing inputs do not yield a consistent improvement over hidden-state routing on this task. Both tasks are synthetic and use a 128-token context. The implementation computes all candidate heads before applying hard masks, so the budget measures the number of active heads. Runtime and memory gains remain unmeasured.
No comments yet — start the discussion below.