The "Lossless" Illusion: How Speculative Decoding Silently Changes What LLMs Say

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Speculative decoding is supposed to give the same output distribution as normal decoding, just faster, and we wanted to see if vLLM and SGLang actually do that. With a real neural network it is hard to know the exact probability of a multi-token output, so we loaded small lookup-table models in Llama format into both engines and worked out the target distribution (μ) and the bug distributions (μD) beforehand. We ended up finding seven bugs across the two engines where the server runs normally but outputs the wrong distribution. In SGLang, we found that unfilled branches in the NGRAM draft tree get padded with token 0 ('!'), so the verifier emits '!' about eleven times as often as it should (0.538 vs. 0.049). In vLLM, we found verifiers that slice target logits to a narrower draft vocabulary, read narrow draft probability buffers using the target's wider row stride, mask or miss stop conditions (min_tokens and budgets) because logit processors only see already-committed tokens, accept -1 padding placeholders on CPU, and reuse the same uniform random numbers across rounds after a rejection. We reported all seven bugs with minimal reproduction scripts; within 24 hours, fix pull requests were open for all seven, five of them written by community contributors. 1. Introduction Production inference engines use speculative decoding to speed up long reasoning runs, multi-step tool calls, and reinforcement learning (RL) rollouts. A small draft model guesses a few tokens ahead, and the main model checks them in one pass. The accepted tokens are supposed to follow the same output distribution (μ) as running the main model alone. When a serving engine breaks that rule, the distribution shift silently propagates to downstream jobs. In RL post-training, a mismatch between the rollout sampler and the learner distribution can destabilize gradient updates. In reasoning or structured generation, missed stop conditions or skipped grammar masks leave extra tags or invalid syntax in the output. Checking this on a full language model is hard: the exact multi-token distribution μ is unknown, and GPU floating-point noise complicates two-sample tests, making it hard to distinguish sampling bugs from benign rounding differences. Small probability shifts do not crash the server, and large neural networks give no exact baseline against which to check token counts. To test the engine directly, we replaced the transformer weights with small lookup-table models registered inside vLLM and SGLang. We zeroed the attention output and MLP down-projection weights in a minimal Llama checkpoint so next-token logits come directly from fixed table numbers while the engine continues running the scheduler, logit processors, C++ code, and CUDA/Triton kernels. Before running the server, we compute the exact target distribution μ over short outputs and, from source inspection, compute the exact bug distribution μD that a suspected bug produces. For speculative decoding to match μ in production, the engine must satisfy four checks: When a draft token is rejected, resample the replacement from the normalized positive residual (p − q)+ rather than sampling from the raw target distribution p or reusing rejected draft tokens. At verification position i, expose both the committed prefix and the i uncommitted draft tokens ahead of it to logit processors (stop tokens, min_tokens, grammar masks, and thinking budgets). Pass the verifier the exact draft distribution q from which the drafter sampled, with matching vocabulary width and memory row stride, without padding unused tree branches with valid token IDs. Draw an independent uniform random number for every acceptance check, including after a rejection rewinds the sequence position to an earlier slot. Leviathan et al. showed that these four checks are sufficient to preserve μ. Our seven bugs (N1–N7) show that each is necessary: breaking any one of them shifts engine outputs from μ to the predicted bug distribution μD. The Seven Defects and Upstream Reports IDEngine & PathWhat Goes WrongEffect SizeUpstream Issue & Fix PR N1vLLM V2, block verifyuniforms reused across roundsP(x3=a) = 5/12 vs. 1/2vllm#61114 → #61153, #61161 (merged) N2vLLM V1, min_tokensEOS masked by committed length5/200 fixed, 13/184 adaptive (Qwen3)vllm#61115 → #61137 N3vLLM V1, thinking budgetnatural unseen8/200 fixed, 92/131 adaptive (Qwen3)vllm#61116 → #61120 N4SGLang NGRAMtree padded with token 0P(tok 0) = 0.538 vs. 0.049sglang#43725 → #43742 N5vLLM V2 samplertarget cut to draft widthTV 0.59, token never emittedvllm#61112 → #61152 N6vLLM CPU kernel-1 placeholder accepted44–52 per 600vllm#61117 → #61127 N7vLLM V1 samplernarrow q read at wide stride393/2000 stray IDs; GPU faultvllm#61113 → #61152 Code and Reproducibility Reference All code, lookup-table model builders, exact law calculators, Colab notebooks, preregistrations, and raw sequence counts are self-contained in spectxncheck-v1.0.zip in this Zenodo record (doi:10.5281/zenodo.23431956). Running bash reproduce.sh inside the unpacked archive verifies SHA256SUMS, re-scores all verdicts from the saved sequence counts in under one minute using only the Python standard library, and regenerates all figures and tables. Files spectxncheck.pdf: the paper. fig_oracle.png, fig_n1_block_rng.png, fig_n5_vocab_truncation.png, fig_real_model_rates.png: Figures 1–3. spectxncheck-v1.0.zip: the complete reproducibility archive containing the paper (PDF and LaTeX source), figures and figure script, frozen preregistrations, lookup-table oracle and engine runners (oracle/, colab/, n6_debug/, survey_sims/), committed result files (results/), reproduce.sh, README.md, licences, and SHA256SUMS. Licence: CC BY 4.0; code scripts are additionally offered under the MIT License. The paper is AI-assisted; its section "Disclosure" states how. Not yet peer reviewed.