A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss
SimLoss aims to get fine-grained captions without the slowdown of multi-stage verification.
The paper proposes a reference-free embedding-space training objective that supervises a captioning model before it decodes text. Its fully differentiable version, SimLoss FFT, reports the highest precision and nearly matches the F1 of a multi-stage method while keeping single-pass inference. The authors say it runs roughly 20 times faster than the multi-stage pipeline. SimLoss GRPO, the reward-based version, posts the strongest recall. HF Daily Papers' note
The paper proposes a reference-free embedding-space training objective that supervises a captioning model before it decodes text. Its fully differentiable version, SimLoss FFT, reports the highest precision and nearly matches the F1 of a multi-stage method while keeping single-pass inference. The authors say it runs roughly 20 times faster than the multi-stage pipeline. SimLoss GRPO, the reward-based version, posts the strongest recall. HF Daily Papers' note
score 4