Two Measurement Failures in Small-Scale Mixture-of-Experts Sparsity Studies

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Small-scale experiments are attractive for studying Mixture-of-Experts sparsity: a full sweep over expert counts with several seeds fits on one consumer GPU, which frontier-scale studies rarely afford. We report two measurement failures that make such studies unreliable, each of which produced a conclusion we later had to retract. First, the usual figure of merit - the best validation loss recorded during training - is a minimum over several dozen subsampled estimates and is optimistically biased, by 0.035-0.060 nats at the three checkpoints we measured. An equal-mean Gaussian model puts the expected bias at sigma_batch / sqrt(eval_iters) times the expected maximum of K standard normals, and sigma_batch moves by under 3% across a 32x range of expert counts, so the bias can be estimated before a sweep is run. Second, capacity-limited dispatch makes a token's fate depend on which other tokens share its batch: holding the evaluated tokens fixed and varying only how they are grouped moves the measured loss by 0.075 nats - more than the entire sparsity effect - and flips the sign of the dense-versus-sparse comparison. Removing the capacity limit at evaluation closes that spread to 3e-5 nats across every batch size and grouping we tried; merely raising it, as is common, leaves 0.028 nats undetermined. A three-seed control detects no training-time counterpart and cannot bound one tightly. The capacity error grows in absolute terms with more experts and with narrower models, the two directions a single-GPU study moves in, and the selection bias is largest wherever evaluation is cheapest; whether either decides a comparison depends in addition on how much signal that comparison has. Corrected, the sparsity gain scales by about 1.7x per width step across three widths and the curve is flat by N=16 at every width. Code, training logs, both evaluation protocols and the checkpoints for all 84 runs are released at https://github.com/taeyoung1/moe-sparsity-scaling.