Continual-ARC: Measuring Skill Acquisition over a Working Lifetime
No tested harness-based method improved on unseen tasks; weight-based learning showed substantial gains.
Continual-ARC mixes new ARC-AGI tasks with returning tasks presented through fresh instances after dormant gaps. It scores learners by how much feedback budget they consume, putting memory-based and weight-based systems on the same scale. In tests on Qwen3.5-9B and GLM-5.3 Flash, persistent harness memory generally failed to improve memorization without explicit expert guidance. Most harness-based methods learned less than a baseline that simply cached each task’s demonstrations.
HF Daily Papers' note
Continual-ARC mixes new ARC-AGI tasks with returning tasks presented through fresh instances after dormant gaps. It scores learners by how much feedback budget they consume, putting memory-based and weight-based systems on the same scale. In tests on Qwen3.5-9B and GLM-5.3 Flash, persistent harness memory generally failed to improve memorization without explicit expert guidance. Most harness-based methods learned less than a baseline that simply cached each task’s demonstrations.
HF Daily Papers' note
score 5