How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
DeepSeek-V4-Flash’s four-stream mHC pathway mostly uses selective routing, with late layers doing little useful stream mixing.
The paper finds that attention and FFN sites typically draw on about two streams, not all four, and the dominant stream shifts by layer. Early residual mixers matter: replacing them with identity raises C4 perplexity by 41%. Late mixers are close to dispensable in the reported tests, with identity replacement raising C4 perplexity by only 1.9% while preserving the six-task average. The authors conclude that this model uses only part of the flexibility mHC makes available. ArXiv · AI/CL/LG's note
The paper finds that attention and FFN sites typically draw on about two streams, not all four, and the dominant stream shifts by layer. Early residual mixers matter: replacing them with identity raises C4 perplexity by 41%. Late mixers are close to dispensable in the reported tests, with identity replacement raising C4 perplexity by only 1.9% while preserving the six-task average. The authors conclude that this model uses only part of the flexibility mHC makes available. ArXiv · AI/CL/LG's note
score 4