FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
None of 150 sessions passed FINCOSMOS’s checks—even those the simulated user approved.
The evaluation requires agents to uncover a hidden investment strategy by asking questions, obtain Web data and build a backtest under a data cutoff. Acceptance requires user satisfaction and passing all 15 veto checks, with programs checking timing and recomputing every reported metric. Across four coding agents and one model without tools, agents asked about only 54–64% of unspecified details. With evaluation feedback and persistent memory, one agent subsequently learned all ten tasks, the authors report.
HF Daily Papers' note
The evaluation requires agents to uncover a hidden investment strategy by asking questions, obtain Web data and build a backtest under a data cutoff. Acceptance requires user satisfaction and passing all 15 veto checks, with programs checking timing and recomputing every reported metric. Across four coding agents and one model without tools, agents asked about only 54–64% of unspecified details. With evaluation feedback and persistent memory, one agent subsequently learned all ten tasks, the authors report.
HF Daily Papers' note
score 5