FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
FinAutoRubric turns expert finance-evaluation standards into reusable guidance that generates and checks task-specific rubrics.
The system has experts define general evaluation rules, then uses agents and code to create, review, and validate rubrics for each query. A writer agent researches expected values, a reviewer agent verifies them, and unresolved failures go to a human. On three expert-authored finance benchmarks, the paper says its rubrics matched expert scoring closely, agreed with human grading, and were preferred by in-house analysts in blind review. The authors also release a 100-query benchmark spanning 78 tasks and eight asset classes. ArXiv · AI/CL/LG's note
The system has experts define general evaluation rules, then uses agents and code to create, review, and validate rubrics for each query. A writer agent researches expected values, a reviewer agent verifies them, and unresolved failures go to a human. On three expert-authored finance benchmarks, the paper says its rubrics matched expert scoring closely, agreed with human grading, and were preferred by in-house analysts in blind review. The authors also release a 100-query benchmark spanning 78 tasks and eight asset classes. ArXiv · AI/CL/LG's note
score 5