EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
EPIC outperformed standard and sequence-level clipping baselines in off-policy mathematical-reasoning tests with a sparse mixture-of-experts model.
The authors identify a gap in token-level clipping: it checks the current token’s probability while overlooking changes in the policy-generated prefix. Their derivation makes the cumulative likelihood ratio of that prefix the exact importance weight for each step’s policy-improvement term. EPIC uses an exact reverse-KL formulation to constrain prefix-level drift without a local quadratic approximation or expert-routing replay. In the reported experiments, it also remained competitive with routing-replay-based alternatives.
ArXiv · AI/CL/LG's note
The authors identify a gap in token-level clipping: it checks the current token’s probability while overlooking changes in the policy-generated prefix. Their derivation makes the cumulative likelihood ratio of that prefix the exact importance weight for each step’s policy-improvement term. EPIC uses an exact reverse-KL formulation to constrain prefix-level drift without a local quadratic approximation or expert-routing replay. In the reported experiments, it also remained competitive with routing-replay-based alternatives.
ArXiv · AI/CL/LG's note
score 5