Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes
Lossy verification can speed decoding by changing the model’s output distribution, but the paper warns that the quality cost can be severe.
The authors analyze how these speculative decoding variants alter generation behavior once strict matching to the target model is relaxed. They group the methods into truncation-based verification and collaborative verification. In truncation-based methods, they find degradation can exceed the true truncation sampling baseline because of distributional distortion. In collaborative verification, they argue that draft probabilities must not overshoot target probabilities too far, or output quality can collapse. HF Daily Papers' note
The authors analyze how these speculative decoding variants alter generation behavior once strict matching to the target model is relaxed. They group the methods into truncation-based verification and collaborative verification. In truncation-based methods, they find degradation can exceed the true truncation sampling baseline because of distributional distortion. In collaborative verification, they argue that draft probabilities must not overshoot target probabilities too far, or output quality can collapse. HF Daily Papers' note
score 5