VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
VisCo compresses VLM visual tokens by making the pretrained model reuse its own encoding capacity.
The paper frames the problem as inference latency and memory overhead from large visual-token counts. VisCo uses a parameter-sharing autoencoder with a small set of memory tokens, transferring hierarchical information from encoding to decoding. The authors report stronger results than prior methods at every tested compression ratio, with the biggest gains under aggressive compression and stability even with a single token. They also say the learned memory tokens can improve the base model when used alongside the original visual tokens. HF Daily Papers' note
The paper frames the problem as inference latency and memory overhead from large visual-token counts. VisCo uses a parameter-sharing autoencoder with a small set of memory tokens, transferring hierarchical information from encoding to decoding. The authors report stronger results than prior methods at every tested compression ratio, with the biggest gains under aggressive compression and stability even with a single token. They also say the learned memory tokens can improve the base model when used alongside the original visual tokens. HF Daily Papers' note
score 4