Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
The paper claims Qwen2-1.5B-level performance can be reached for about $4.4K in training cost.
The authors trained Puro-2B models from scratch on consumer RTX 5090 GPUs, using up to 1.4 trillion tokens and FP8 precision. Their best run cost under $6.9K and approached Qwen2.5-1.5B performance under their evaluation setup. They attribute the cost drop to hardware choice, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. They also release the data, code, recipe, and weights under Apache 2.0. ArXiv · AI/CL/LG's note
The authors trained Puro-2B models from scratch on consumer RTX 5090 GPUs, using up to 1.4 trillion tokens and FP8 precision. Their best run cost under $6.9K and approached Qwen2.5-1.5B performance under their evaluation setup. They attribute the cost drop to hardware choice, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. They also release the data, code, recipe, and weights under Apache 2.0. ArXiv · AI/CL/LG's note
score 6