Beyond Prompts: Runtime Contracts for Coding Agents
Checks enforced during execution yielded no detected rule violations in 104 paired runs.
The paper compares executable contracts with prompted rules, retrieved guidance and checks applied after generation. In a confirmatory study targeting known failure situations, enforcement reduced violating runs by 31 percentage points versus no rules, but its advantage over prompted rules was not statistically significant. Prompted rules missed some recursive directory deletes; checks after generation could repair file edits but could not undo commands already executed. Neither confirmatory study found a loss of functional correctness, though the experiments used low-cost models and one small fixture.
ArXiv · AI/CL/LG's note
The paper compares executable contracts with prompted rules, retrieved guidance and checks applied after generation. In a confirmatory study targeting known failure situations, enforcement reduced violating runs by 31 percentage points versus no rules, but its advantage over prompted rules was not statistically significant. Prompted rules missed some recursive directory deletes; checks after generation could repair file edits but could not undo commands already executed. Neither confirmatory study found a loss of functional correctness, though the experiments used low-cost models and one small fixture.
ArXiv · AI/CL/LG's note
score 5