Evaluating LLM-Generated World Models for Neuro-Symbolic Reasoning
Of 134 LLM-generated world models that passed mechanical checks, only 28 exactly matched the authors’ reference models.
The researchers tested five LLMs across 44 scenarios, analyzing 410 first-pass generations of formal models for multi-agent decision problems. In the reference-backed game tests, 134 of 135 models passed a strict mechanical audit, but average structural F1 was just 0.366. In cultural scenarios, naming the country changed most LLMs’ generated structures while automated semantic scores changed only slightly. The findings show that an executable model can still misrepresent the decision problem used for downstream reasoning and planning.
ArXiv · AI/CL/LG's note
The researchers tested five LLMs across 44 scenarios, analyzing 410 first-pass generations of formal models for multi-agent decision problems. In the reference-backed game tests, 134 of 135 models passed a strict mechanical audit, but average structural F1 was just 0.366. In cultural scenarios, naming the country changed most LLMs’ generated structures while automated semantic scores changed only slightly. The findings show that an executable model can still misrepresent the decision problem used for downstream reasoning and planning.
ArXiv · AI/CL/LG's note
score 4