Megadose The AI curve, compressed.

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

· ArXiv · AI/CL/LG ·
Spool removes expired video frames from recurrent model state without replaying the remaining window.

The training-free framework caches a transformation for each frame and combines those transformations as the video window advances. Across five hybrid vision-language models and four benchmarks, the authors report the lowest p95 time-to-first-token in all 20 tests, with a 6.6× geometric-mean speedup over the faster of two recent vLLM baselines. Accuracy improved by 2.3 percentage points on average. Removal is exact within each recurrent layer, but expired frames can still influence cached inputs computed by earlier layers.

ArXiv · AI/CL/LG's note

score 5

Categories: Research