Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
High reconstruction scores can miss whether an activation explanation’s individual claims are true.
The paper argues that natural-language autoencoder tests can reward explanations that preserve gist while leaving false specific claims unpenalized. On a released Qwen-2.5-7B verbalizer, only about 2% of specific claims were ones the reconstruction depended on. The proposed RECAP method trains auxiliary linear heads so designated content remains decodable and checkable by probes. In sandbox and Pythia-160M tests, RECAP improved lie detection by independent probes, including against adversarially edited explanations. HF Daily Papers' note
The paper argues that natural-language autoencoder tests can reward explanations that preserve gist while leaving false specific claims unpenalized. On a released Qwen-2.5-7B verbalizer, only about 2% of specific claims were ones the reconstruction depended on. The proposed RECAP method trains auxiliary linear heads so designated content remains decodable and checkable by probes. In sandbox and Pythia-160M tests, RECAP improved lie detection by independent probes, including against adversarially edited explanations. HF Daily Papers' note
score 4