Follow the Gradient: Tracing Safety Degradation Back to Training Data
Removing 20% of Tulu 3’s training examples cut attack success rates by up to 22%, while largely preserving or improving capability.
The researchers’ SimpleLoRIF method flags examples whose training gradients align with unsafe responses, even when their content appears benign. It scales to datasets of roughly a million examples by restricting backpropagation to the final four transformer blocks. The filtered dataset, SafeTulu, outperformed LLM-as-a-judge and prior gradient-based filtering against GCG attacks across Qwen, Llama and Olmo. Analysis of the removed examples pointed to tool-execution failures and narrative arithmetic as possible sources of safety degradation.
ArXiv · AI/CL/LG's note
The researchers’ SimpleLoRIF method flags examples whose training gradients align with unsafe responses, even when their content appears benign. It scales to datasets of roughly a million examples by restricting backpropagation to the final four transformer blocks. The filtered dataset, SafeTulu, outperformed LLM-as-a-judge and prior gradient-based filtering against GCG attacks across Qwen, Llama and Olmo. Analysis of the removed examples pointed to tool-execution failures and narrative arithmetic as possible sources of safety degradation.
ArXiv · AI/CL/LG's note
score 5