Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems
The paper proposes testing an AI system’s rationality from its own answers, without labels or human feedback.
Isaiah Andrews argues that decision-theory representation theorems can turn synthetic choice problems into computable evaluation penalties. The checks cover probabilistic coherence, preference rationality, and subjective expected utility, with zero penalty when the model’s behavior can be rationalized under the relevant theorem. Passing such a test only clears that rationality standard for the elicited data; it does not say the model’s objective is desirable. ArXiv · AI/CL/LG's note
Isaiah Andrews argues that decision-theory representation theorems can turn synthetic choice problems into computable evaluation penalties. The checks cover probabilistic coherence, preference rationality, and subjective expected utility, with zero penalty when the model’s behavior can be rationalized under the relevant theorem. Passing such a test only clears that rationality standard for the elicited data; it does not say the model’s objective is desirable. ArXiv · AI/CL/LG's note
score 4