Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
BPM preserves teacher probability mass while translating it across mismatched tokenizers.
The paper proposes Byte-Prefix Marginalization for on-policy distillation between open-weight language models that do not share a tokenizer. It maps teacher next-token probabilities into the student vocabulary through shared byte prefixes, with unmatched mass kept in an explicit residual bucket. The authors say this gives a byte-aligned, vocabulary-complete target and recovers the teacher-induced marginal at more than 99% of training positions. Tested with Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 teachers, BPM beat prior cross-tokenizer methods on six math and coding benchmarks by 3.7 to 6.6 avg@8 points.
Source: ArXiv · AI/CL/LG's note
The paper proposes Byte-Prefix Marginalization for on-policy distillation between open-weight language models that do not share a tokenizer. It maps teacher next-token probabilities into the student vocabulary through shared byte prefixes, with unmatched mass kept in an explicit residual bucket. The authors say this gives a byte-aligned, vocabulary-complete target and recovers the teacher-induced marginal at more than 99% of training positions. Tested with Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 teachers, BPM beat prior cross-tokenizer methods on six math and coding benchmarks by 3.7 to 6.6 avg@8 points.
Source: ArXiv · AI/CL/LG's note
score 5