Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment
In the mathematics tests, misaligned answers rose from 4% to over 60% as fine-tuning answers became more wrong, with little loss of general capability.
Across mathematics, trivia, cooking and sports, stronger disbelief in the training answers was associated with greater misalignment, even without harmful content. After training, models stated false answers while their multiple-choice probabilities still favored true ones, suggesting internal inconsistency as a cause. The relationship also held when updating a single fact. Presenting that false fact within a coherent story reduced misalignment compared with instructing the model to assert it, but content filters and persona vectors could not distinguish the two framings.
ArXiv · AI/CL/LG's note
Across mathematics, trivia, cooking and sports, stronger disbelief in the training answers was associated with greater misalignment, even without harmful content. After training, models stated false answers while their multiple-choice probabilities still favored true ones, suggesting internal inconsistency as a cause. The relationship also held when updating a single fact. Presenting that false fact within a coherent story reduced misalignment compared with instructing the model to assert it, but content filters and persona vectors could not distinguish the two framings.
ArXiv · AI/CL/LG's note
score 5