Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Evolution Strategies beat GRPO on wider reasoning coverage, especially at Pass@K.
The paper argues that ES keeps more verifier-relevant diversity across model outputs, while GRPO tends toward entropy collapse. In the authors’ experiments, ES improves Pass@1 and reaches higher Pass@K than GRPO, suggesting it can draw out more of a pretrained model’s reasoning range. They also propose running GRPO first, then ES, to combine stronger Pass@1 with ES’s Pass@K gains. The authors report that ES’s useful gains come from a sparse subset of larger updates, without clear evidence of catastrophic forgetting in their held-out tests. HF Daily Papers' note
The paper argues that ES keeps more verifier-relevant diversity across model outputs, while GRPO tends toward entropy collapse. In the authors’ experiments, ES improves Pass@1 and reaches higher Pass@K than GRPO, suggesting it can draw out more of a pretrained model’s reasoning range. They also propose running GRPO first, then ES, to combine stronger Pass@1 with ES’s Pass@K gains. The authors report that ES’s useful gains come from a sparse subset of larger updates, without clear evidence of catastrophic forgetting in their held-out tests. HF Daily Papers' note
score 5