TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimization
The paper frames prompt bloat as an overfitting problem and proposes regularization directly in text-space optimization.
The authors argue that iterative prompt optimizers can add long, sample-specific rules that hurt generalization outside the training distribution. They define “representational inefficiency” as a mix of capacity cost and narrow scope, then use it to explain how optimized prompts drift into overfit behavior. TextReg adds regularized textual gradients through evidence filtering, semantic edit constraints, and guided prompt updates. Across reasoning benchmarks, the paper reports out-of-distribution accuracy gains of up to 11.8% over TextGrad and 16.5% over REVOLVE. HF Daily Papers' note
The authors argue that iterative prompt optimizers can add long, sample-specific rules that hurt generalization outside the training distribution. They define “representational inefficiency” as a mix of capacity cost and narrow scope, then use it to explain how optimized prompts drift into overfit behavior. TextReg adds regularized textual gradients through evidence filtering, semantic edit constraints, and guided prompt updates. Across reasoning benchmarks, the paper reports out-of-distribution accuracy gains of up to 11.8% over TextGrad and 16.5% over REVOLVE. HF Daily Papers' note
score 4