NVIDIA and AI4Bharat Release 2.8-Trillion-Token Indic Pretraining Dataset With Open Pipeline

NVIDIA, together with AI4Bharat at the Indian Institute of Technology Madras, has unveiled Nemotron-pretraining-Indic-v1, a pretraining dataset spanning 22 Indian languages with approximately 2.8 trillion tokens. The release also includes an open, GPU-accelerated data generation pipeline that developers can reuse to create their own training data.

The dataset covers 22 languages listed in India's Constitution, representing first languages for nearly 97% of Indians according to the 2011 Census. It provides native-script data for all 22 languages and romanized text for twelve, resulting in 34 language-script configurations. The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, allowing teams to adapt it to their own source documents.

The dataset was built from 15 Nemotron-CC-v2 subsets, using NVIDIA's NeMo Data Designer to perform translation, quality assessment, and targeted repair. A Gemma-4-31B-it model served as translator and evaluator, applying an MQM-inspired rubric to retain outputs scoring at least 75 out of 100. Repair workflows recovered an additional 438.4 billion tokens. Every record retains its source identifier to support traceability and auditing.

AI4Bharat previously released Sangraha, an open pretraining corpus for Indian languages. The new joint release significantly expands the open synthetic Indic training data available to the community. NVIDIA noted that the corpus has been used for continued pretraining of Nemotron 3.5 Super VL, with further technical details in accompanying reports.