CulturalRiddles: A Multicultural Benchmark of Riddles
Even frontier models scored no higher than 22% accuracy in some cultural communities.
CulturalRiddles contains human-created riddles spanning 62 cultures, 45 countries and 51 languages. Solving them requires reasoning alongside factual and social knowledge, language skills and culturally specific associations. Researchers tested 28 LLM configurations using automatic scoring and large-scale human evaluation. They publicly release the benchmark’s artifacts and invite contributions to expand its cultural and linguistic coverage.
ArXiv · AI/CL/LG's note
CulturalRiddles contains human-created riddles spanning 62 cultures, 45 countries and 51 languages. Solving them requires reasoning alongside factual and social knowledge, language skills and culturally specific associations. Researchers tested 28 LLM configurations using automatic scoring and large-scale human evaluation. They publicly release the benchmark’s artifacts and invite contributions to expand its cultural and linguistic coverage.
ArXiv · AI/CL/LG's note
score 4