HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
HiKV claims a two-stage KV-cache compression design that cuts attention memory work with little accuracy loss.
The paper pairs token-level eviction with element-level loading inside retained tokens, then maps both stages onto a dedicated accelerator. Its reconfigurable importance sorter switches between the sorting paths needed by the two stages. On representative LLMs, the authors report up to 7.95x attention speedup and 90% energy reduction versus a vanilla KV cache baseline, with about 1% accuracy loss. Under iso-accuracy constraints, they report 1.82x to 4.87x fewer external memory accesses than other importance-based methods, with an 8% system-area overhead. Source: ArXiv · AI/CL/LG's note.
The paper pairs token-level eviction with element-level loading inside retained tokens, then maps both stages onto a dedicated accelerator. Its reconfigurable importance sorter switches between the sorting paths needed by the two stages. On representative LLMs, the authors report up to 7.95x attention speedup and 90% energy reduction versus a vanilla KV cache baseline, with about 1% accuracy loss. Under iso-accuracy constraints, they report 1.82x to 4.87x fewer external memory accesses than other importance-based methods, with an 8% system-area overhead. Source: ArXiv · AI/CL/LG's note.
score 5