ReplaySSM: Cache SSM Inputs, Not State
ReplaySSM cuts SSM decoding traffic by caching recent inputs and updating state checkpoints only periodically.
The paper says eager state reads and writes can bottleneck hybrid SSM models during decoding. ReplaySSM represents state as a checkpoint plus a buffer of recent SSM inputs, avoiding most per-step state writes and making speculative rollback a buffer-pointer move. In vLLM tests on Mamba-2 and Gated DeltaNet, it reports 1.20-1.47x faster standard decoding at batch size 256 versus a tuned vLLM baseline. On two 120B-class models, it reports about 1.8x the throughput of vLLM speculative decoding, while separate checkpoint precisions reduce divergence and accuracy loss. ArXiv · AI/CL/LG's note
The paper says eager state reads and writes can bottleneck hybrid SSM models during decoding. ReplaySSM represents state as a checkpoint plus a buffer of recent SSM inputs, avoiding most per-step state writes and making speculative rollback a buffer-pointer move. In vLLM tests on Mamba-2 and Gated DeltaNet, it reports 1.20-1.47x faster standard decoding at batch size 256 versus a tuned vLLM baseline. On two 120B-class models, it reports about 1.8x the throughput of vLLM speculative decoding, while separate checkpoint precisions reduce divergence and accuracy loss. ArXiv · AI/CL/LG's note
score 6