Agent Behavior Maps: Locating Agents by What They Would Do, Without Running Them
Agents with similar success rates can still take different paths—and the map captures similarities that scores miss.
The method scores how likely an LLM-and-harness combination is to take actions in shared reference trajectories, without requiring fresh environment runs from that agent. Across eight cross-scored models, map distances tracked the rank order of directly measured trajectory divergence. Tests covered 100 LLMs and four environments; in ALFWorld, behavioral similarities persisted after controlling for success rates and model size, including across model families. Changes to harness components also approximately composed, though accuracy varied by component and environment.
ArXiv · AI/CL/LG's note
The method scores how likely an LLM-and-harness combination is to take actions in shared reference trajectories, without requiring fresh environment runs from that agent. Across eight cross-scored models, map distances tracked the rank order of directly measured trajectory divergence. Tests covered 100 LLMs and four environments; in ALFWorld, behavioral similarities persisted after controlling for success rates and model size, including across model families. Changes to harness components also approximately composed, though accuracy varied by component and environment.
ArXiv · AI/CL/LG's note
score 5