Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)
The paper argues that a selection network breaks under the standard meta-learning objective, so it introduces a pointwise value-matching loss instead.
Du, Yang, and Li say the usual move from per-sample weights to a transferable selection network causes unstable optimization and weak generalization. They trace the failure to weight suppression and continued dependence on easy-to-learn features. Their proposed framework, TESS, uses Pointwise Value Matching to score examples at scale. The reported experiments cover LLM safety and targeted instruction tuning, with transfer from subsets to full corpora and from smaller to larger models. ArXiv · AI/CL/LG's note
Du, Yang, and Li say the usual move from per-sample weights to a transferable selection network causes unstable optimization and weak generalization. They trace the failure to weight suppression and continued dependence on easy-to-learn features. Their proposed framework, TESS, uses Pointwise Value Matching to score examples at scale. The reported experiments cover LLM safety and targeted instruction tuning, with transfer from subsets to full corpora and from smaller to larger models. ArXiv · AI/CL/LG's note
score 5