Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency
Chronological pre-training made models cheaper to update later.
Zhao and Zhu call these recent-knowledge-skewed models “specialists,” contrasting them with shuffled-order “generalists.” In their experiments, specialists needed far fewer update steps to learn new knowledge. The paper derives a law tying that update cost to optimization dynamics, then uses it to guide a data-ordering strategy meant to keep older knowledge stable while making current knowledge easier to refresh. The authors say the effect also appears in real-world pre-trained models and existing knowledge update tools. ArXiv · AI/CL/LG's note
Zhao and Zhu call these recent-knowledge-skewed models “specialists,” contrasting them with shuffled-order “generalists.” In their experiments, specialists needed far fewer update steps to learn new knowledge. The paper derives a law tying that update cost to optimization dynamics, then uses it to guide a data-ordering strategy meant to keep older knowledge stable while making current knowledge easier to refresh. The authors say the effect also appears in real-world pre-trained models and existing knowledge update tools. ArXiv · AI/CL/LG's note
score 4