AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures
Three models scored well on cultural-norm questions in English but performed much worse when prompted in Yorùbá.
The AfriNORM paper reports this gap for Gemini 2.0 Flash, GPT-5 and Llama 3.1 8B Instruct. Its authors attribute the weaker Yorùbá results to semantic confusion about cultural norms rather than simple bias. The benchmark uses scenario-based multiple-choice questions to test reasoning about African social norms and taboos. It also introduces separate metrics for commonsense bias and limited understanding of African cultural norms.
ArXiv · AI/CL/LG's note
The AfriNORM paper reports this gap for Gemini 2.0 Flash, GPT-5 and Llama 3.1 8B Instruct. Its authors attribute the weaker Yorùbá results to semantic confusion about cultural norms rather than simple bias. The benchmark uses scenario-based multiple-choice questions to test reasoning about African social norms and taboos. It also introduces separate metrics for commonsense bias and limited understanding of African cultural norms.
ArXiv · AI/CL/LG's note
score 4