Follow the Gradient: Tracing Safety Degradation Back to Training Data
Removing 20% of Tulu 3’s training examples cut attack success by up to 22%, while largely preserving or improving capability.
The researchers’ SimpleLoRIF method flags examples whose training gradients align with unsafe responses, even when their content looks benign. Restricting backpropagation to the final four transformer blocks lets it process datasets of roughly a million examples. The filtered dataset, SafeTulu, outperformed both LLM-as-a-judge and prior gradient-based filtering against GCG attacks across Qwen, Llama and Olmo models. Removed examples pointed to possible sources of safety degradation, including tool-execution failures and narrative arithmetic.
HF Daily Papers' note
The researchers’ SimpleLoRIF method flags examples whose training gradients align with unsafe responses, even when their content looks benign. Restricting backpropagation to the final four transformer blocks lets it process datasets of roughly a million examples. The filtered dataset, SafeTulu, outperformed both LLM-as-a-judge and prior gradient-based filtering against GCG attacks across Qwen, Llama and Olmo models. Removed examples pointed to possible sources of safety degradation, including tool-execution failures and narrative arithmetic.
HF Daily Papers' note
score 5