Accelerating Sharded Data Parallelism at Scale with Federated Learning
The paper claims FedAvg-style grouping can make sharded training faster and improve model quality at large GPU counts.
Mittone and Aldinucci introduce FL+FSDP and FL+HSDP, which split large sharded data-parallel jobs into smaller federation groups. The design cuts inter-group communication while keeping global batch growth tied to group size. In a Llama3.1 8B pre-training run on 512 A100 GPUs, the authors report up to 8.04x faster data processing and 4.48 lower evaluation perplexity than the baseline counterparts. ArXiv · AI/CL/LG's note
Mittone and Aldinucci introduce FL+FSDP and FL+HSDP, which split large sharded data-parallel jobs into smaller federation groups. The design cuts inter-group communication while keeping global batch growth tied to group size. In a Llama3.1 8B pre-training run on 512 A100 GPUs, the authors report up to 8.04x faster data processing and 4.48 lower evaluation perplexity than the baseline counterparts. ArXiv · AI/CL/LG's note
score 5