Identically Trained Language Models Break on Different Jailbreaks
The strongest jailbreak on one checkpoint stayed strongest after a new pretraining seed in only 27% of model pairs.
Researchers tested 150 models with 1 billion parameters across 25 pretraining data recipes, using three pretraining seeds and two safety post-training seeds. Per-prompt non-refusal rates correlated at 0.984 when only the post-training seed changed, falling to 0.811 with a different pretraining seed. Changing the data recipe lowered the correlation only slightly further, to 0.779. The results suggest a checkpoint’s specific jailbreak weaknesses depend more on its pretraining run than on its data recipe.
HF Daily Papers' note
Researchers tested 150 models with 1 billion parameters across 25 pretraining data recipes, using three pretraining seeds and two safety post-training seeds. Per-prompt non-refusal rates correlated at 0.984 when only the post-training seed changed, falling to 0.811 with a different pretraining seed. Changing the data recipe lowered the correlation only slightly further, to 0.779. The results suggest a checkpoint’s specific jailbreak weaknesses depend more on its pretraining run than on its data recipe.
HF Daily Papers' note
score 5