Megadose The AI curve, compressed.

Direction or Magnitude? Learning Scalars over Task-Calibrated Directions

· ArXiv · AI/CL/LG ·
SCALAR matches LoRA’s commonsense reasoning accuracy with about 4% of its trainable parameters, the authors report.

The method builds fixed input directions from task activations, pairs them with random output dictionaries, and trains only layer-specific scalar coefficients. This reduces storage for gradients and optimizer states. On Qwen2.5-7B, the authors measured 20% lower peak training memory and 6% higher throughput than LoRA. Experiments also covered mathematical reasoning, code generation, and knowledge evaluation after instruction tuning.

ArXiv · AI/CL/LG's note

score 5

Categories: Research