DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction
DepthGraft cuts the KV cache by roughly a third while reporting higher benchmark scores.
The architecture reconstructs deep-layer keys and values from pairs of shallow and middle layers. The authors argue that this sharing creates direct gradient paths that improve KV representations and help deep layers use information more effectively. Tests span dense and mixture-of-experts models from 0.5B to 16B parameters, with gains under both GQA and MLA. On the 16B MoE model, they report a 7.0 percentage-point improvement on MMLU-Pro, alongside gains on CMMLU and C3.
HF Daily Papers' note
The architecture reconstructs deep-layer keys and values from pairs of shallow and middle layers. The authors argue that this sharing creates direct gradient paths that improve KV representations and help deep layers use information more effectively. Tests span dense and mixture-of-experts models from 0.5B to 16B parameters, with gains under both GQA and MLA. On the 16B MoE model, they report a 7.0 percentage-point improvement on MMLU-Pro, alongside gains on CMMLU and C3.
HF Daily Papers' note
score 5