Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning
The paper finds that readable chain-of-thought steps only partly reveal which steps actually matter to the model’s answer.
The authors measure a reasoning step’s importance by its estimated effect on expected reward when included. LLM judges can identify high-importance steps better than a baseline, but remain far from the paper’s noise ceiling. Fine-tuning a step-level critic helps most on incorrect responses, while correct responses remain harder to interpret from the trace alone. ArXiv · AI/CL/LG's note
The authors measure a reasoning step’s importance by its estimated effect on expected reward when included. LLM judges can identify high-importance steps better than a baseline, but remain far from the paper’s noise ceiling. Fine-tuning a step-level critic helps most on incorrect responses, while correct responses remain harder to interpret from the trace alone. ArXiv · AI/CL/LG's note
score 5