Dust: Pretraining Transformers Without Backpropagation
Dust gets close to backprop by turning each token into a parallel activation-space perturbation trial.
The method avoids weight-space evolution by adding noise to activations and using token-level loss changes to estimate gradients. In the paper’s transformer pretraining runs, Dust is competitive with backprop at large populations and is far more compute-efficient than an EGGROLL-style weight perturbation baseline. The authors say larger models were not harder for the method in the expected way: a 243M-parameter model often beat a model 120 times smaller at the same population sizes. They do not claim Dust is ready to replace backprop now, noting it still needs orders-of-magnitude efficiency gains.
HN · Transformers' note
The method avoids weight-space evolution by adding noise to activations and using token-level loss changes to estimate gradients. In the paper’s transformer pretraining runs, Dust is competitive with backprop at large populations and is far more compute-efficient than an EGGROLL-style weight perturbation baseline. The authors say larger models were not harder for the method in the expected way: a 243M-parameter model often beat a model 120 times smaller at the same population sizes. They do not claim Dust is ready to replace backprop now, noting it still needs orders-of-magnitude efficiency gains.
HN · Transformers' note
score 6
Discussions
- hn · 103 points · 18 comments
- hn · 114 points · 19 comments
- hn · 118 points · 20 comments
- hn · 126 points · 22 comments
- hn · 133 points · 29 comments
- hn · 139 points · 30 comments
- hn · 144 points · 31 comments
- hn · 147 points · 33 comments
- hn · 149 points · 35 comments
- hn · 154 points · 36 comments
- hn · 163 points · 40 comments
- hn · 172 points · 42 comments
- hn · 180 points · 43 comments
- hn · 187 points · 44 comments
- hn · 197 points · 47 comments
- hn · 203 points · 49 comments
- hn · 209 points · 52 comments
- hn · 215 points · 55 comments
- hn · 225 points · 59 comments
- hn · 229 points · 61 comments
- hn · 234 points · 62 comments
- hn · 236 points · 62 comments
- hn · 239 points · 62 comments
- hn · 240 points · 64 comments
- hn · 243 points · 65 comments
- hn · 247 points · 66 comments
- hn · 249 points · 66 comments
- hn · 252 points · 68 comments
- hn · 253 points · 69 comments
- hn · 255 points · 69 comments
- hn · 256 points · 69 comments
- hn · 257 points · 73 comments
- hn · 261 points · 75 comments
- hn · 262 points · 77 comments
- hn · 264 points · 79 comments
- hn · 265 points · 79 comments
- hn · 269 points · 79 comments
- hn · 271 points · 79 comments
- hn · 272 points · 79 comments
- hn · 272 points · 79 comments
- hn · 272 points · 79 comments
- hn · 273 points · 79 comments
- hn · 273 points · 79 comments
- hn · 273 points · 79 comments
- hn · 275 points · 79 comments
- hn · 275 points · 79 comments
- hn · 275 points · 80 comments
- hn · 275 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 80 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 81 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 276 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments
- hn · 277 points · 82 comments