Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
ZFO lets a standard optimizer pick the update direction, then spends two extra objective checks deciding the step size.
The paper frames this as a cheaper alternative to full line search for LLM fine-tuning. Its method builds a local one-dimensional model along the proposed direction and chooses a curvature-aware step inside a bounded interval. The authors report theoretical guarantees for the finite-difference curvature estimates, the step choice, and convergence near a stationary point. Across their tested models and datasets, ZFO often improved optimization and final performance versus fixed-step first-order baselines. ArXiv · AI/CL/LG's note
The paper frames this as a cheaper alternative to full line search for LLM fine-tuning. Its method builds a local one-dimensional model along the proposed direction and chooses a curvature-aware step inside a bounded interval. The authors report theoretical guarantees for the finite-difference curvature estimates, the step choice, and convergence near a stationary point. Across their tested models and datasets, ZFO often improved optimization and final performance versus fixed-step first-order baselines. ArXiv · AI/CL/LG's note
score 4