GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
GoiBERT-large beats all non-teacher models on average across eight benchmarks, with under half the parameters and twice the throughput of the strongest compact baseline.
The two Vietnamese encoders have 26 million and 49 million parameters and process raw text without external word segmentation. Training combines general representations from a Vietnamese teacher with task-level soft labels from a larger teacher. GoiBERT-base retains 99% of the strongest compact baseline’s performance at a quarter of its size. Compression is nearly lossless for sentiment and classification, but performance losses remain for natural language inference and named entity recognition.
ArXiv · AI/CL/LG's note
The two Vietnamese encoders have 26 million and 49 million parameters and process raw text without external word segmentation. Training combines general representations from a Vietnamese teacher with task-level soft labels from a larger teacher. GoiBERT-base retains 99% of the strongest compact baseline’s performance at a quarter of its size. Compression is nearly lossless for sentiment and classification, but performance losses remain for natural language inference and named entity recognition.
ArXiv · AI/CL/LG's note
score 4