UltraBERT: Beyond the Limit of BERT Pre-Training
UltraBERT claims one encoder can hit state of the art on both classification and retrieval by splitting how it trains on masking.
The paper says encoder pretraining stops paying off after a few trillion tokens, so more compute alone is not the fix. Its tests found lower masking ratios help classification, while higher masking ratios help retrieval. UltraBERT handles that split by using separate language-model heads for low and high masking ratios in the same batch, and applying distillation only to the low-ratio side. The authors report 91.0 on GLUE and 47.7 on BEIR. HF Daily Papers' note
The paper says encoder pretraining stops paying off after a few trillion tokens, so more compute alone is not the fix. Its tests found lower masking ratios help classification, while higher masking ratios help retrieval. UltraBERT handles that split by using separate language-model heads for low and high masking ratios in the same batch, and applying distillation only to the low-ratio side. The authors report 91.0 on GLUE and 47.7 on BEIR. HF Daily Papers' note
score 5