Learning to Learn from Context: Synthetic Training from Perturbed Public Documents
A synthetic pipeline turned 3,500 perturbed public documents into context-learning training data that lifted a Qwen student sharply on CL-bench.
The authors rewrite public documents to reduce memorization risk, then generate questions, rubrics, and document-grounded answers. They filter for samples that actually require the provided context, yielding about 10,000 examples without human annotators. SFT moves Qwen3.6-35B-A3B from 13.7% to 22.8% on CL-bench; a rubric-reward RL stage reaches 24.6%. The paper says gains also transfer to long-context understanding, instruction following, and reasoning, while code generation and knowledge mostly stay flat. HF Daily Papers' note
The authors rewrite public documents to reduce memorization risk, then generate questions, rubrics, and document-grounded answers. They filter for samples that actually require the provided context, yielding about 10,000 examples without human annotators. SFT moves Qwen3.6-35B-A3B from 13.7% to 22.8% on CL-bench; a rubric-reward RL stage reaches 24.6%. The paper says gains also transfer to long-context understanding, instruction following, and reasoning, while code generation and knowledge mostly stay flat. HF Daily Papers' note
score 5