Direction or Magnitude? Learning Scalars over Task-Calibrated Directions
SCALAR matches LoRA’s commonsense reasoning accuracy with about 4% of its trainable parameters, the authors report.
The method builds fixed input directions from task activations, pairs them with random output dictionaries, and trains only layer-specific scalar coefficients. This reduces storage for gradients and optimizer states. On Qwen2.5-7B, the authors measured 20% lower peak training memory and 6% higher throughput than LoRA. Experiments also covered mathematical reasoning, code generation, and knowledge evaluation after instruction tuning.
ArXiv · AI/CL/LG's note
The method builds fixed input directions from task activations, pairs them with random output dictionaries, and trains only layer-specific scalar coefficients. This reduces storage for gradients and optimizer states. On Qwen2.5-7B, the authors measured 20% lower peak training memory and 6% higher throughput than LoRA. Experiments also covered mathematical reasoning, code generation, and knowledge evaluation after instruction tuning.
ArXiv · AI/CL/LG's note
score 5