User Feedback Provides a Unique Signal that LLMs Can not Detect
Feedback-informed revisions fixed targeted problems that model-only evaluation often missed.
The paper argues that user interaction feedback is not just noise, but an actionable signal for improving LLM outputs. The authors test revisions with and without feedback on synthetic data with ground truth, then check the pattern on more naturalistic data. In both settings, feedback helped resolve the specific issues at higher rates than baseline revisions. The reported failure is in the judges: LLM evaluators often preferred weaker baseline outputs when the real fix depended on feedback. ArXiv · AI/CL/LG's note
The paper argues that user interaction feedback is not just noise, but an actionable signal for improving LLM outputs. The authors test revisions with and without feedback on synthetic data with ground truth, then check the pattern on more naturalistic data. In both settings, feedback helped resolve the specific issues at higher rates than baseline revisions. The reported failure is in the judges: LLM evaluators often preferred weaker baseline outputs when the real fix depended on feedback. ArXiv · AI/CL/LG's note
score 4