A Unified Per-Token Gating Family for On-Policy Distillation: FKL/RKL Mixing with Multi-Channel and Bias Coefficients
The paper’s main claim is that a broader per-token KL-gating setup outperformed narrower entropy-only or gap-only versions in most tested comparisons, but the authors frame the evidence as exploratory.
Wu, Lin, Yuan, and Jiang define a four-coefficient gate that can reproduce direction-aligned versions of EOPD and ToDi while adding multi-channel and bias terms. On TweetEval emotion and hate with Qwen3-32B as teacher and Qwen3-4B as student, the full family beat matched single-channel restrictions in 33 of 36 comparable cells. A separate mean-match experiment favored dynamic gating over static baselines in 19 of 26 cells. Three-seed follow-ups, including offensive classification, stayed directionally consistent but were smaller and not significant at n=3. ArXiv · AI/CL/LG's note
Wu, Lin, Yuan, and Jiang define a four-coefficient gate that can reproduce direction-aligned versions of EOPD and ToDi while adding multi-channel and bias terms. On TweetEval emotion and hate with Qwen3-32B as teacher and Qwen3-4B as student, the full family beat matched single-channel restrictions in 33 of 36 comparable cells. A separate mean-match experiment favored dynamic gating over static baselines in 19 of 26 cells. Three-seed follow-ups, including offensive classification, stayed directionally consistent but were smaller and not significant at n=3. ArXiv · AI/CL/LG's note
score 4