Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection
Explainability tools helped identify ways to make Prompt Guard 2 miss rewritten injection attempts.
The paper tests Prompt Guard 2 with attribution methods including Vanilla Gradient and SHAP. It finds the classifier depends on many token-level signals, not one obvious trigger. Guided synonym swaps and paraphrases sometimes changed the classifier’s decision while modifying only part of the prompt, and in some cases enabled a jailbreak against the underlying model. The authors warn that transparency methods can help defenders inspect guardrails while also making bypass construction cheaper. ArXiv · AI/CL/LG's note
The paper tests Prompt Guard 2 with attribution methods including Vanilla Gradient and SHAP. It finds the classifier depends on many token-level signals, not one obvious trigger. Guided synonym swaps and paraphrases sometimes changed the classifier’s decision while modifying only part of the prompt, and in some cases enabled a jailbreak against the underlying model. The authors warn that transparency methods can help defenders inspect guardrails while also making bypass construction cheaper. ArXiv · AI/CL/LG's note
score 4