Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models
Direct whole-file generation beat iterative diffs across every measured test in this Flutter/Dart study.
Andrejev trained two code models in both regimes on the same dataset, then evaluated each on about 1,790 held-out tasks. Whole-file output had higher compile/static-analysis pass rates, better bits-per-byte and character similarity, and stronger blinded judge scores for goal fulfillment, correctness, and code quality. Diff-based generation only held up on short, localized changes, with its wins concentrated in refactoring and error-handling or edge-case fixes. HF Daily Papers' note
Andrejev trained two code models in both regimes on the same dataset, then evaluated each on about 1,790 held-out tasks. Whole-file output had higher compile/static-analysis pass rates, better bits-per-byte and character similarity, and stronger blinded judge scores for goal fulfillment, correctness, and code quality. Diff-based generation only held up on short, localized changes, with its wins concentrated in refactoring and error-handling or edge-case fixes. HF Daily Papers' note
score 4