Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
Models failed to apply time-dependent safety constraints in 62.1% of cases even when they correctly calculated elapsed time.
The preprint introduces TASC, a benchmark with 3,418 inputs derived from 128 templates and a separate set of 33 scenarios. Across 13 models, assistants accounted for timing constraints in only 30.5% of cases where a proposed action violated a rule, while rarely issuing unnecessary warnings. Explicitly supplying the timing corrected only about half of the failures, exposing a gap between calculating elapsed time and using it in conversational advice.
ArXiv · AI/CL/LG's note
The preprint introduces TASC, a benchmark with 3,418 inputs derived from 128 templates and a separate set of 33 scenarios. Across 13 models, assistants accounted for timing constraints in only 30.5% of cases where a proposed action violated a rule, while rarely issuing unnecessary warnings. Explicitly supplying the timing corrected only about half of the failures, exposing a gap between calculating elapsed time and using it in conversational advice.
ArXiv · AI/CL/LG's note
score 5