ARCS: Towards Precise Text-to-SQL via Structured Disambiguation
ARCS tests text-to-SQL systems on ambiguity that standard benchmarks mostly smooth over.
The paper introduces structured disambiguation: resolving unclear user questions through explicit, constrained choices rather than open-ended clarification chat. Its ARCS benchmark covers naturally occurring ambiguities in real-world databases, with annotations for ambiguity points, valid interpretations, and SQL queries. The authors report that performance drops sharply under this setup: gpt-6-sol reaches 51% end-to-end execution accuracy, while no open-source model exceeds 27%. HF Daily Papers' note
The paper introduces structured disambiguation: resolving unclear user questions through explicit, constrained choices rather than open-ended clarification chat. Its ARCS benchmark covers naturally occurring ambiguities in real-world databases, with annotations for ambiguity points, valid interpretations, and SQL queries. The authors report that performance drops sharply under this setup: gpt-6-sol reaches 51% end-to-end execution accuracy, while no open-source model exceeds 27%. HF Daily Papers' note
score 5