Megadose The AI curve, compressed.

MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition

· ArXiv · AI/CL/LG ·
MAGIC predicted data-removal effects accurately over short training windows, but faltered across a full training run.

Researchers tested the attribution method on 86M- and 152M-parameter models learning multistep arithmetic from scratch. At 152M, scores across four 100-step windows closely matched retraining results, with LDS scores of 0.86–0.99; full-run attribution at 86M encountered numerical overflow and scored just 0.17–0.22 even with clipping. Individual training-block rankings did not transfer across seeds or hyperparameters, though patterns by data format did. Worked examples and primitive facts were overrepresented among both helpful and harmful examples—a measure of local sensitivity, not proof of net benefit.

ArXiv · AI/CL/LG's note

score 5

Categories: Research