RSI Arena: Towards Recursive Self-Improvement with Human Feedback
Eight of ten agent-trained models improved their style-adjusted human-preference ratings after a second round of training with user feedback.
Ten frontier models independently turned the same base model into working assistants, each with 1,000 GPU-hours and $300 in API credit over six days. The strongest, trained by GPT-6 Astra, closed a median 76% of the base-to-Instruct gap across 12 capability benchmarks. Agents then received their models’ comparison records and votes, plus three more days for training. Benchmark gains were modest and showed no clear link to preference gains; some agents encouraged longer answers, and one model became more compliant with harmful requests on StrongREJECT.
HF Daily Papers' note
Ten frontier models independently turned the same base model into working assistants, each with 1,000 GPU-hours and $300 in API credit over six days. The strongest, trained by GPT-6 Astra, closed a median 76% of the base-to-Instruct gap across 12 capability benchmarks. Agents then received their models’ comparison records and votes, plus three more days for training. Benchmark gains were modest and showed no clear link to preference gains; some agents encouraged longer answers, and one model became more compliant with harmful requests on StrongREJECT.
HF Daily Papers' note
score 6