CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
CordisBench tests whether models can track how plugin changes ripple through agent runtimes.
The paper introduces a 1,200-question benchmark for lifecycle reasoning in dynamic agent harnesses. It asks models to identify affected components, predict state after teardown orders, reason across possible orders, and choose reconfigurations that actually run. The evaluated efficiency-oriented models do well on smaller systems but degrade as more interactions matter, especially on final-state and teardown-order questions. The authors also report that extra reasoning effort can help, but at meaningful token cost. ArXiv · AI/CL/LG's note
The paper introduces a 1,200-question benchmark for lifecycle reasoning in dynamic agent harnesses. It asks models to identify affected components, predict state after teardown orders, reason across possible orders, and choose reconfigurations that actually run. The evaluated efficiency-oriented models do well on smaller systems but degrade as more interactions matter, especially on final-state and teardown-order questions. The authors also report that extra reasoning effort can help, but at meaningful token cost. ArXiv · AI/CL/LG's note
score 4