Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
SWE-Flux tests repository-level execution reasoning with answers taken from instrumented Python test runs, and the best evaluated model reached 37% accuracy.
The benchmark contains 480 execution-grounded instances across 12 real Python repositories.
It asks single-test and multi-test questions about control flow, loops, state, dataflow, exceptions, and invariants.
Models did better on localized behavior, but struggled with dataflow, inter-procedural execution, precise state, and suite-level aggregation.
The authors also report that input perturbation produced valid fresh variants for nearly 90% of selected instances, making the task harder for the tested models.
Source: ArXiv · AI/CL/LG's note
The benchmark contains 480 execution-grounded instances across 12 real Python repositories.
It asks single-test and multi-test questions about control flow, loops, state, dataflow, exceptions, and invariants.
Models did better on localized behavior, but struggled with dataflow, inter-procedural execution, precise state, and suite-level aggregation.
The authors also report that input perturbation produced valid fresh variants for nearly 90% of selected instances, making the task harder for the tested models.
Source: ArXiv · AI/CL/LG's note
score 6