DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs
A 128-GPU cluster built from second-hand V100s served LLaMA-70B competitively, but only makes economic and climate sense under cheap, clean power.
The authors built the system from used components and ran it for a year. They put its current hardware cost at about $22,000, compared with about $600,000 for an 8-GPU B200 system. Pipeline-parallel optimizations made the V100 cluster viable for 70B inference, but energy use per token was much worse than newer hardware. At grid-average carbon intensity, the paper says emissions per token can be about 4x higher for 8B models and over 40x higher for 70B models. HF Daily Papers' note
The authors built the system from used components and ran it for a year. They put its current hardware cost at about $22,000, compared with about $600,000 for an 8-GPU B200 system. Pipeline-parallel optimizations made the V100 cluster viable for 70B inference, but energy use per token was much worse than newer hardware. At grid-average carbon intensity, the paper says emissions per token can be about 4x higher for 8B models and over 40x higher for 70B models. HF Daily Papers' note
score 6