Dongxu Liu, Bing Zhu, Xinyu Zhu, Long Zheng, Xiaofei Liao, Hai Jin, Jingling Xue · · 2026
DOI: 10.1145/3828523.3857099
Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).
Speculative decoding accelerates autoregressive generation by drafting and verifying multiple tokens in bulk. However, in many existing systems, verification remains monolithic: the candidate tree is processed through all transformer layers and projected over the full vocabulary before acceptance is resolved. This all-or-nothing design wastes verifier work on rejected branches, materializes transient KV state for discarded candidates, and creates pipeline imbalance. We present BMV, a serving system that breaks monolithic verification into staged verifier admission and exact final verification. The key insight is that intermediate target-model states provide evidence for pruning low-confidence branches before full-depth execution. BMV inserts a lightweight pruning stage between drafting and final verification: it performs prefix-guided pruning, preserves reusable KV state for surviving candidates through token-level prefix sharing, and coordinates the stages through a feedback-controlled asynchronous runtime. Final acceptance remains deferred to the exact verifier; pruning affects only speculation opportunities, not committed-token semantics. Across three LLMs (8B–70B) and diverse workloads, BMV achieves up to 5.80 × speedup over autoregressive decoding and 1.31 × over EAGLE-3 on a single H200 GPU, and improves pipeline throughput by 1.93 × on 5 × V100. A cost breakdown shows that auxiliary pruning logic adds less than 1% overhead, while verifier-side KV footprint is reduced by 30.9%. With a false negative rate of at most 2.61%, BMV demonstrates that staged verification is practical for scaling speculative LLM serving. The source code of BMV will be released at https://github.com/CGCL-codes/BMV .
No comments yet — start the discussion below.