Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
The benchmark is built to test whether agents learn rules through play, not prior knowledge.
Learn2Play Bench uses newly designed text games with novel or counterintuitive rules, repeated attempts, automatic scoring, and reproducible feedback. The paper reports that keeping full action-and-feedback histories helped agents more than compressing experience into summaries. Humans still reached higher peak scores, explored more varied strategies, and repeated actions less. It also finds that changing the agent harness can improve results and lower estimated inference cost with the same backbone model. HF Daily Papers' note
Learn2Play Bench uses newly designed text games with novel or counterintuitive rules, repeated attempts, automatic scoring, and reproducible feedback. The paper reports that keeping full action-and-feedback histories helped agents more than compressing experience into summaries. Humans still reached higher peak scores, explored more varied strategies, and repeated actions less. It also finds that changing the agent harness can improve results and lower estimated inference cost with the same backbone model. HF Daily Papers' note
score 4