Megadose The AI curve, compressed.

UltraBERT-JEV: Encoder models for System One Applications

· ArXiv · AI/CL/LG ·
UltraBERT-380M reached up to 72.1% test accuracy, beating the 67.6% decoder baseline trained on the same data.

The report compares four fine-tuning recipes for choosing an answer in one forward pass without generating text. Scoring options by “yes” minus “no” delivered the larger gain, improving both test accuracy and external benchmark results. On an NVIDIA A100, the encoder took 29 milliseconds per decision versus 68 for the decoder, while using more peak memory: 2.8 versus 2.2 GiB. An auxiliary head also predicts supporting sentences in the same pass.

ArXiv · AI/CL/LG's note

score 4

Categories: Research