FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
FLAT trains one shared multimodal representation to serve both retrieval and generation, instead of handing frozen embeddings to a separate decoder.
The paper maps image and text inputs into a unified 1D continuous token sequence, with flexible prefix lengths controlled by nested dropout. Its training combines contrastive alignment with text-to-image and image-to-text generation objectives. The authors report 83.1 GenEval after T2I fine-tuning, plus MS-COCO captioning and retrieval scores competitive with state-of-the-art baselines. They also show qualitative uses for interpolation, latent arithmetic, and zero-shot composed retrieval. HF Daily Papers' note
The paper maps image and text inputs into a unified 1D continuous token sequence, with flexible prefix lengths controlled by nested dropout. Its training combines contrastive alignment with text-to-image and image-to-text generation objectives. The authors report 83.1 GenEval after T2I fine-tuning, plus MS-COCO captioning and retrieval scores competitive with state-of-the-art baselines. They also show qualitative uses for interpolation, latent arithmetic, and zero-shot composed retrieval. HF Daily Papers' note
score 5