Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B
Across 36 matched comparisons, none of the reflection-style methods beat repeated sampling at the same token cost.
The paper reruns seven prompting methods on open 1.5B, 3B, and 7B models across two math benchmarks, counting every generated token. Ten comparisons were significantly worse than repeated sampling, all from methods where the model inspected its own answer. Best-of-N selection stopped clearly hurting at 7B, but Self-Refine and forced Reflexion still trailed the baseline there. Reflexion also failed to trigger retries on the smallest model, reducing itself to a single chain of thought. ArXiv · AI/CL/LG's note
The paper reruns seven prompting methods on open 1.5B, 3B, and 7B models across two math benchmarks, counting every generated token. Ten comparisons were significantly worse than repeated sampling, all from methods where the model inspected its own answer. Best-of-N selection stopped clearly hurting at 7B, but Self-Refine and forced Reflexion still trailed the baseline there. Reflexion also failed to trigger retries on the smallest model, reducing itself to a single chain of thought. ArXiv · AI/CL/LG's note
score 5