Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Cultivar tests whether translation models handle locale-specific content, not just language pairs.
The paper introduces a localised subset of FLORES built for source-contrastive evaluation. Paired with unlocalised examples, it is meant to expose contamination and weak localisation robustness. In tests of 32 open-weight models, MT-specialised systems were less robust, some models appeared to overfit FLORES, and US content translated better than other locales across languages. HF Daily Papers' note
The paper introduces a localised subset of FLORES built for source-contrastive evaluation. Paired with unlocalised examples, it is meant to expose contamination and weak localisation robustness. In tests of 32 open-weight models, MT-specialised systems were less robust, some models appeared to overfit FLORES, and US content translated better than other locales across languages. HF Daily Papers' note
score 5