Shanghai AI Lab and SJTU Open Source 8.9B Latent-Space Foundation Model

Shanghai Artificial Intelligence Laboratory and the LUMIA Lab at Shanghai Jiao Tong University's School of AI released the NCP-ArchPreview Tech Report in September 2026, introducing a novel pre-training objective called Next Concept Prediction (NCP). The team also open-sourced the first 8.9B-parameter discrete latent-space foundation model, and the paper has topped the HuggingFace Daily Papers ranking.

Unlike the conventional Next-Token Prediction (NTP) paradigm, NCP performs explicit concept-level modeling in latent space. A token encoder compresses hidden states into discrete concept sequences, a concept module autoregressively predicts the next concept, and the predicted representations are then fed back to the decoder to guide token generation. Pre-trained on the full 5.73T-token Dolma-3 corpus, the model matched the final pre-training loss of OLMo-3-7B using only 51.3% of the token budget, achieving an equivalent 1.95x convergence speedup and a final loss 0.091 lower than the baseline.

On downstream benchmarks, NCP-ArchPreview outperforms OLMo-3-7B by 2.45 points on average across 30 benchmark categories, with a 5.99-point improvement on GSM8K and a 4.28-point gain on HumanEval. The team also showed that fine-tuning only the VQ codebook and concept prediction head—about 17 million parameters—beats a same-size LoRA on math tasks without degrading general capabilities. In addition, injecting concept representations into a speculative decoding draft model improves mean accepted length by 4.17% without adding computation to the target model.

The team has released all assets on HuggingFace, including checkpoints from dozens of training stages, evaluation code, and draft model weights.