The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models
A two-pass audit caught every injected causality leak that mask checks missed.
The paper argues that prefix invariance has to be tested beyond attention masks, because future-token leaks can come through scans, aggregations, or normalization. Its audit needs no training or gradients and reports a per-layer score showing where the break occurs. In 192 injected-fault trials across eight checkpoints, mask inspection found none; the audit localized all 192 to the exact layer. The authors also report finding the same chunked-scan axis defect in Zamba2 and Nemotron-H. HF Daily Papers' note
The paper argues that prefix invariance has to be tested beyond attention masks, because future-token leaks can come through scans, aggregations, or normalization. Its audit needs no training or gradients and reports a per-layer score showing where the break occurs. In 192 injected-fault trials across eight checkpoints, mask inspection found none; the audit localized all 192 to the exact layer. The authors also report finding the same chunked-scan axis defect in Zamba2 and Nemotron-H. HF Daily Papers' note
score 5