How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Orthrus matched the reference model exactly only when the authors reran the trajectory test in FP32.
In BF16 inference, exact sequence matching appeared in 45% of prompts for the authors’ checkpoint and 43% for an independently trained reproduction. The test covered 1,190 prompts across 12 domains, and matching was tied to the reference model’s response-conditional perplexity. The divergence did not translate into systematic drops on downstream `lm-eval-harness` benchmarks. The paper’s point is narrower: Orthrus’ “lossless” claim depends on numerical precision, so trajectory equivalence has to be measured separately from task scores. HF Daily Papers' note
In BF16 inference, exact sequence matching appeared in 45% of prompts for the authors’ checkpoint and 43% for an independently trained reproduction. The test covered 1,190 prompts across 12 domains, and matching was tied to the reference model’s response-conditional perplexity. The divergence did not translate into systematic drops on downstream `lm-eval-harness` benchmarks. The paper’s point is narrower: Orthrus’ “lossless” claim depends on numerical precision, so trajectory equivalence has to be measured separately from task scores. HF Daily Papers' note
score 4