Codifying the Judge: Scalable Evaluation via Program Distillation
PAJAMA replaces repeated LLM judging with inspectable programs, escalating only low-confidence cases back to an LLM.
The paper argues that LLM-as-a-judge is costly, slow, and hard to inspect. Its proposed system distills judging behavior into a committee of programs that score candidates directly and combine their verdicts. Across five datasets and four model families, the authors report performance matching a 13B-size LLM judge. They also say program-derived reward labels beat a model trained on proprietary LLM labels on RewardBench at far lower API cost. HF Daily Papers' note
The paper argues that LLM-as-a-judge is costly, slow, and hard to inspect. Its proposed system distills judging behavior into a committee of programs that score candidates directly and combine their verdicts. Across five datasets and four model families, the authors report performance matching a 13B-size LLM judge. They also say program-derived reward labels beat a model trained on proprietary LLM labels on RewardBench at far lower API cost. HF Daily Papers' note
score 5