NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
An 8.9B-parameter model adds a concept-level training target alongside next-token prediction and reports faster loss matching than OLMo-3-7B.
NCP-ArchPreview builds a product-quantized concept vocabulary from hidden states, predicts future concepts with a separate module, then feeds those concepts back into token generation. The team trained it on 5.73T Dolma-3 tokens and says it reached OLMo-3-7B’s final pretraining loss after using 51.3% of the total training tokens. After full pretraining, it reports a 2.45-point downstream macro-average gain over OLMo-3-7B, including 5.99 points on GSM8K. The authors also describe lightweight domain adaptation by updating only a 17M-parameter VQ module. HF Daily Papers' note
NCP-ArchPreview builds a product-quantized concept vocabulary from hidden states, predicts future concepts with a separate module, then feeds those concepts back into token generation. The team trained it on 5.73T Dolma-3 tokens and says it reached OLMo-3-7B’s final pretraining loss after using 51.3% of the total training tokens. After full pretraining, it reports a 2.45-point downstream macro-average gain over OLMo-3-7B, including 5.99 points on GSM8K. The authors also describe lightweight domain adaptation by updating only a 17M-parameter VQ module. HF Daily Papers' note
score 6