CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines
Three of ten models cleared 90% on core requirements; none exceeded 45% on higher-quality shopping advice.
CARTBench tests shopping assistants on complex, ambiguous requests with changing requirements over time. It pairs timestamped conversations across 17 retail domains with 1,603 product records, using 1,328 expert-annotated criteria covering Core, Excellence and Safety. The authors identify an understanding-to-execution gap, with shared failures in product evidence and unresolved customer needs.
HF Daily Papers' note
CARTBench tests shopping assistants on complex, ambiguous requests with changing requirements over time. It pairs timestamped conversations across 17 retail domains with 1,603 product records, using 1,328 expert-annotated criteria covering Core, Excellence and Safety. The authors identify an understanding-to-execution gap, with shared failures in product evidence and unresolved customer needs.
HF Daily Papers' note
score 4