Training a 3.8B LLM to 0.384 CORE for $998
A solo training run produced a 3.8B-parameter model scoring 0.384 on CORE after 65.3B tokens, 43 hours, and $998 of B200 rental.
The author says the result beat GPT-2’s listed CORE score and landed ahead of nanochat’s roughly $1,000 configuration. Much of the write-up is a cost-and-throughput postmortem: ClimbMix data, Muon, FP8, vocab padding, fused cross-entropy, and local shards all mattered. The 2048-token run raised CORE mainly because SQuAD and BoolQ were badly cropped at 1024, not because the model broadly improved. Several choices, including value embeddings and nanochat-inherited settings, remain only partly tested.
Source: HN · LLMs' note
The author says the result beat GPT-2’s listed CORE score and landed ahead of nanochat’s roughly $1,000 configuration. Much of the write-up is a cost-and-throughput postmortem: ClimbMix data, Muon, FP8, vocab padding, fused cross-entropy, and local shards all mattered. The 2048-token run raised CORE mainly because SQuAD and BoolQ were badly cropped at 1024, not because the model broadly improved. Several choices, including value embeddings and nanochat-inherited settings, remain only partly tested.
Source: HN · LLMs' note
score 5
Discussions
- hn · 100 points · 15 comments
- hn · 101 points · 15 comments
- hn · 102 points · 15 comments
- hn · 103 points · 15 comments
- hn · 104 points · 17 comments
- hn · 106 points · 17 comments
- hn · 106 points · 17 comments
- hn · 107 points · 17 comments
- hn · 107 points · 17 comments
- hn · 108 points · 17 comments
- hn · 110 points · 17 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 18 comments
- hn · 110 points · 19 comments
- hn · 110 points · 19 comments
- hn · 110 points · 19 comments
- hn · 110 points · 19 comments
- hn · 111 points · 19 comments
- hn · 111 points · 19 comments
- hn · 111 points · 19 comments
- hn · 111 points · 19 comments
- hn · 112 points · 19 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 112 points · 20 comments
- hn · 113 points · 20 comments
- hn · 113 points · 20 comments
- hn · 113 points · 20 comments
- hn · 113 points · 20 comments
- hn · 113 points · 20 comments
- hn · 113 points · 20 comments
- hn · 113 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 114 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 20 comments
- hn · 115 points · 21 comments
- hn · 116 points · 21 comments
- hn · 116 points · 21 comments
- hn · 116 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 117 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 118 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 119 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments
- hn · 120 points · 21 comments