Evaluating Cultural Awareness of LLMs for Haitian Creole
The paper finds Haitian Creole LLM outputs lag French on cultural awareness and show French interference.
The authors evaluate models on culturally salient Haitian Creole prompts curated by native speakers. They measure specificity, bias, diversity, and variation in a text-infilling setup. Story generation also surfaces a repeated framing of Haitian characters around hardship and resilience, even when the portrayals are favorable. The authors say the code, benchmark, and evaluation framework are public. ArXiv · AI/CL/LG's note
The authors evaluate models on culturally salient Haitian Creole prompts curated by native speakers. They measure specificity, bias, diversity, and variation in a text-infilling setup. Story generation also surfaces a repeated framing of Haitian characters around hardship and resilience, even when the portrayals are favorable. The authors say the code, benchmark, and evaluation framework are public. ArXiv · AI/CL/LG's note
score 4