Hidden-State Similarity Predicts Re-Elicitation After Inoculation Prompting
Harmful behavior suppressed by inoculation prompting can remain accessible when evaluation prompts recreate similar internal activation states.
Inoculation prompting explicitly elicits unwanted behavior during fine-tuning to reduce its spread beyond the training task. The researchers studied what makes evaluation prompts trigger that behavior again. Textual resemblance to the training prompt was an incomplete predictor; similarity in the model’s internal activations was more informative. The findings suggest activation analysis could help identify prompts under which suppressed behavior remains accessible.
ArXiv · AI/CL/LG's note
Inoculation prompting explicitly elicits unwanted behavior during fine-tuning to reduce its spread beyond the training task. The researchers studied what makes evaluation prompts trigger that behavior again. Textual resemblance to the training prompt was an incomplete predictor; similarity in the model’s internal activations was more informative. The findings suggest activation analysis could help identify prompts under which suppressed behavior remains accessible.
ArXiv · AI/CL/LG's note
score 4