RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios
The best end-to-end score among 11 frontier LLMs was 55.29% on RuleTrek.
The benchmark tests whether agents can identify and apply domain-specific rules when scenario information is spread across multiple files. It separately measures rule identification, rule deduction and end-to-end reasoning in interactive workspaces. The highest identification and deduction scores were 62.47% and 60.43%, respectively. The authors say their construction process keeps scenarios consistent and solvable, and have made the code and dataset public.
HF Daily Papers' note
The benchmark tests whether agents can identify and apply domain-specific rules when scenario information is spread across multiple files. It separately measures rule identification, rule deduction and end-to-end reasoning in interactive workspaces. The highest identification and deduction scores were 62.47% and 60.43%, respectively. The authors say their construction process keeps scenarios consistent and solvable, and have made the code and dataset public.
HF Daily Papers' note
score 4