Distillation Defenses Easily Break After Reinforcement Learning
The paper says defenses that look solid right after distillation can fail once the copied model gets reinforcement learning.
The authors argue that evaluations are using too narrow a threat model when they stop at the distillation step. In their experiments, reinforcement learning makes simpler API-based attacks competitive with attacks that extract full hidden reasoning traces. Their conclusion is that defenses leaking enough information to rebuild approximate reasoning traces are likely ineffective. ArXiv · AI/CL/LG's note
The authors argue that evaluations are using too narrow a threat model when they stop at the distillation step. In their experiments, reinforcement learning makes simpler API-based attacks competitive with attacks that extract full hidden reasoning traces. Their conclusion is that defenses leaking enough information to rebuild approximate reasoning traces are likely ineffective. ArXiv · AI/CL/LG's note
score 5