Differentially Private Sparse Reward Estimation with Preference Feedback
Private reward estimation can avoid polynomial dependence on model size when only a small subset of features drives preferences.
The paper assumes sparse reward models and introduces algorithms for Sample-DP and Label-DP using pairwise preference feedback. It derives estimation-error bounds governed by the number of relevant features, training-set size and privacy strength. The authors show the Sample-DP rate is nearly optimal; their Label-DP guarantee cannot improve within the randomized-response framework. They also report better privacy–utility trade-offs than dense baselines on sentiment generation and safety alignment benchmarks.
ArXiv · AI/CL/LG's note
The paper assumes sparse reward models and introduces algorithms for Sample-DP and Label-DP using pairwise preference feedback. It derives estimation-error bounds governed by the number of relevant features, training-set size and privacy strength. The authors show the Sample-DP rate is nearly optimal; their Label-DP guarantee cannot improve within the randomized-response framework. They also report better privacy–utility trade-offs than dense baselines on sentiment generation and safety alignment benchmarks.
ArXiv · AI/CL/LG's note
score 4