Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
Chronicle turns a recorded agent failure into a CI regression test by replaying some nondeterministic boundaries and running the rest live against new code.
The paper records agent runs as immutable envelopes at model, tool, and other nondeterministic crossings. Its “cut-point replay” lets developers choose which crossings come from the record and which execute with changed code. In six recorded failure cases with simulated model boundaries, full replay made zero model calls and stayed bit-stable across 20 repeats. The cut-point tests failed on faulty code and passed on guarded or benign changes in all six incidents. ArXiv · AI/CL/LG's note
The paper records agent runs as immutable envelopes at model, tool, and other nondeterministic crossings. Its “cut-point replay” lets developers choose which crossings come from the record and which execute with changed code. In six recorded failure cases with simulated model boundaries, full replay made zero model calls and stayed bit-stable across 20 repeats. The cut-point tests failed on faulty code and passed on guarded or benign changes in all six incidents. ArXiv · AI/CL/LG's note
score 5