SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
SQLMorph stress-tests text-to-SQL systems by mutating queries and scoring partial execution quality.
The paper introduces Join Query Expansion to add valid joins and Textual Query Augmentation to vary the natural-language prompt. In experiments, more joins exposed accuracy drops, while heavy abbreviation reduced accuracy by up to 17%. The authors also propose Execution Precision, Execution Recall, and F1-style scoring to show over- and under-prediction that binary execution accuracy can hide. ArXiv · AI/CL/LG's note
The paper introduces Join Query Expansion to add valid joins and Textual Query Augmentation to vary the natural-language prompt. In experiments, more joins exposed accuracy drops, while heavy abbreviation reduced accuracy by up to 17%. The authors also propose Execution Precision, Execution Recall, and F1-style scoring to show over- and under-prediction that binary execution accuracy can hide. ArXiv · AI/CL/LG's note
score 5