Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
The paper argues ES reaches more diverse correct reasoning paths than GRPO while still improving first-answer accuracy.
The authors say ES avoids the entropy collapse seen in GRPO and posts stronger Pass@K results. They propose a sequential GRPO-ES setup to pair GRPO’s Pass@1 strength with ES’s broader coverage. They also report that ES gains come from a sparse subset of larger updates, despite broad parameter drift, and do not necessarily cause catastrophic forgetting in held-out tests. ArXiv · AI/CL/LG's note
The authors say ES avoids the entropy collapse seen in GRPO and posts stronger Pass@K results. They propose a sequential GRPO-ES setup to pair GRPO’s Pass@1 strength with ES’s broader coverage. They also report that ES gains come from a sparse subset of larger updates, despite broad parameter drift, and do not necessarily cause catastrophic forgetting in held-out tests. ArXiv · AI/CL/LG's note
score 6