Overcoming Prior Barriers: Supervised Fine-Tuning under Long-Tail Distribution
PASS targets the SFT examples at concepts where the pretrained model has the weakest support.
The paper defines “prior barriers” as a way to measure how much a pretrained model favors competing concepts over the target one. It argues those barriers are long-tailed: common concepts start easier, rare concepts need more supervision. PASS selects instructions by estimating which examples add distinguishing evidence, then shifts budget toward still-undercovered concepts. The authors report gains over seven instruction-selection methods across four backbone-budget settings. ArXiv · AI/CL/LG's note
The paper defines “prior barriers” as a way to measure how much a pretrained model favors competing concepts over the target one. It argues those barriers are long-tailed: common concepts start easier, rare concepts need more supervision. PASS selects instructions by estimating which examples add distinguishing evidence, then shifts budget toward still-undercovered concepts. The authors report gains over seven instruction-selection methods across four backbone-budget settings. ArXiv · AI/CL/LG's note
score 3