RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States
RoMeRL keeps agent memory learning bounded by mapping growing experience histories into fixed per-task utility states.
The paper targets two failure modes in self-evolving LLM agents: feedback spreading thin as histories grow, and irrelevant retrieved memories getting rewarded by association. RoMeRL factorizes memory state by task, outcome polarity, and memory dynamics, then updates or replaces semantic coordinates instead of expanding the utility space indefinitely. In ALFWorld and LifelongAgentBench, the authors report better task performance, an 80.0% lower Cold-Q ratio, about 6.0x higher feedback density, 84.4% smaller maintained memory, and 21.1% fewer LLM calls. HF Daily Papers' note
The paper targets two failure modes in self-evolving LLM agents: feedback spreading thin as histories grow, and irrelevant retrieved memories getting rewarded by association. RoMeRL factorizes memory state by task, outcome polarity, and memory dynamics, then updates or replaces semantic coordinates instead of expanding the utility space indefinitely. In ALFWorld and LifelongAgentBench, the authors report better task performance, an 80.0% lower Cold-Q ratio, about 6.0x higher feedback density, 84.4% smaller maintained memory, and 21.1% fewer LLM calls. HF Daily Papers' note
score 4