PORT: Preference Optimization via Robust Token-Level Reweighting
PORT targets faulty tokens in preference data without discarding useful parts of a response.
Rejected responses can contain sound reasoning, while preferred responses can include errors. PORT reweights individual tokens using the empirical distribution of their log-probabilities, requiring no additional backward passes. The authors provide theoretical analysis showing reduced gradient bias from noisy preference labels and report better robustness than sequence-level baselines across the tested noise settings.
ArXiv · AI/CL/LG's note
Rejected responses can contain sound reasoning, while preferred responses can include errors. PORT reweights individual tokens using the empirical distribution of their log-probabilities, requiring no additional backward passes. The authors provide theoretical analysis showing reduced gradient bias from noisy preference labels and report better robustness than sequence-level baselines across the tested noise settings.
ArXiv · AI/CL/LG's note
score 4