StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval
StreamEmbed encodes screenshots up to 3.7 times faster without sacrificing retrieval accuracy in the authors’ benchmark.
It skips repeated image blocks and reuses cached computations for unchanged regions, with a small learned adapter preserving embedding fidelity. Tests on an M2 Pro GPU used 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent. At matched latency, it beat OCR-based indexing by up to 6.3 Recall@10 points. Its memory-efficient vector index held a simulated year of screenshots in 0.13 GB of RAM.
ArXiv · AI/CL/LG's note
It skips repeated image blocks and reuses cached computations for unchanged regions, with a small learned adapter preserving embedding fidelity. Tests on an M2 Pro GPU used 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent. At matched latency, it beat OCR-based indexing by up to 6.3 Recall@10 points. Its memory-efficient vector index held a simulated year of screenshots in 0.13 GB of RAM.
ArXiv · AI/CL/LG's note
score 5