Throughput Optimization for Multi-Level Speculative Decoding

Speculative decoding is a widely used technique for accelerating the inference step of large language models (LLMs) by generating draft tokens with smaller models and verifying them with a larger target model. While multi-level speculative decoding frameworks have been proposed, the optimal choice of parameters to maximize throughput remains poorly understood.

We present a probabilistic analysis of the multi-level speculative decoding and derive expressions to compute the optimal draft lengths via dynamic programming. We also introduce a novel adaptive drafting technique based on token confidence, and show how to compute optimal thresholds using discretized integrals to estimate the expected throughput for two-model generations.

We also provide simulations of multi-level decoding through empirical measurements of open-source LLMs, which confirm that we can derive near-optimal parameters for token generation, and that the new adaptive drafting schemes allow for increased throughput.