Megadose The AI curve, compressed.

Multi-Branch Transformer

· HF Daily Papers ·
Parallel Transformer branches improved validation loss at matched training budgets, with the final design delivering up to 51.4% higher decoding throughput on one H200 GPU.

The researchers tested mixture-of-experts models with 0.8 billion to 2.1 billion total parameters, routing inputs through shared layers, independent parallel branches and a final fusion step. Two-branch Wide-Shallow models consistently beat sequential Transformers; Balanced variants improved further. The final recipe pairs Balanced branching with Joint-KV, a shared cache that reduced full-model KV-cache storage by up to 34%. Fitted scaling curves suggest standard Transformers need up to 1.26 times the training body compute to match the final models’ validation loss.

HF Daily Papers' note

score 5

Categories: Research