SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
SCOUT localizes failed or corrupted training ranks by making equivalent replicas vote on outliers.
The paper says LLM pre-training failures often look job-wide because synchronization spreads a local stall, slowdown, or numerical error. SCOUT compares replica progress, timing, numerical signatures, and collective fingerprints to find the rank that disagrees with a strict majority. It also uses an out-of-band CPU observer for hangs and live in-situ replay to test recurring stragglers or silent data corruption under the same workload conditions. The framework is described as integrating with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without changing training loops or framework source. ArXiv · AI/CL/LG's note
The paper says LLM pre-training failures often look job-wide because synchronization spreads a local stall, slowdown, or numerical error. SCOUT compares replica progress, timing, numerical signatures, and collective fingerprints to find the rank that disagrees with a strict majority. It also uses an out-of-band CPU observer for hangs and live in-situ replay to test recurring stragglers or silent data corruption under the same workload conditions. The framework is described as integrating with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without changing training loops or framework source. ArXiv · AI/CL/LG's note
score 5