ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
ShallowStream uses shallow MLLM layers as a live video index, then calls deeper processing only when answering.
The paper argues that full-depth prefill on every incoming frame is the expensive part streaming systems have missed. Its method keeps a lightweight KV-cache index from shallow layers during the stream, then uses shallow-layer attention scores to pick evidence frames at query time. The authors report performance comparable to the strongest existing streaming methods, with up to 52.1x lower per-frame prefill latency and 11.9x lower 10-second end-to-end latency. ArXiv · AI/CL/LG's note
The paper argues that full-depth prefill on every incoming frame is the expensive part streaming systems have missed. Its method keeps a lightweight KV-cache index from shallow layers during the stream, then uses shallow-layer attention scores to pick evidence frames at query time. The authors report performance comparable to the strongest existing streaming methods, with up to 52.1x lower per-frame prefill latency and 11.9x lower 10-second end-to-end latency. ArXiv · AI/CL/LG's note
score 5