NVIDIA and AI4Bharat detail a 2.8-trillion-token training corpus for 22 Indian languages

What NVIDIA and AI4Bharat announced

NVIDIA announced the corpus in a Hugging Face blog post dated 16 October 2026, written with AI4Bharat researchers Mohammed Safi Ur Rahman Khan, Mitesh Khapra and Anoop Kunchukuttan. Called Nemotron-pretraining-Indic-v1, it is described as "approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it."

The 22 languages are those listed in India's Constitution. NVIDIA says they are the first languages of 97 percent of Indians, or nearly 1.2 billion people at the 2011 census. AI4Bharat, IIT Madras's Indian-language research lab, already publishes open resources such as Sangraha, an open pretraining corpus for the same 22 languages.

The problem the release targets is simple to state. Khapra, who heads AI4Bharat, has said Indian-language technology trails English because "we do not have enough data for Indian languages." NVIDIA's answer is to translate a large, curated body of English material, including mathematics, code and question-and-answer content, into those languages, and to publish the recipe so others can generate more.

As of 16 October the dataset itself was not yet listed on Hugging Face. What was public is a companion technical report, posted in an NVIDIA dataset repository that morning, and the open-source translation plugin used to build the corpus.

What is in the 2.8 trillion tokens

By NVIDIA's count, the corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, measured with the Gemma-4 tokenizer. It spans 34 language-script configurations, because twelve languages come in both their native script and Latin-script romanized form.

Those twelve are Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu. The other ten, Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali and Sindhi, have native-script data only. Romanized here means Indic-language text written in Latin letters, the way many people type Hindi or Tamil on a phone, not an English translation.

The authors are careful about what the numbers mean. Native-script and romanized records can express the same underlying content, document records include translations and script variants of shared sources, and token counts describe the retained corpus rather than a training budget. In other words, 2.8 trillion tokens is not 2.8 trillion tokens of unique information.

How English documents became Indic training data

The source material is documents from 15 subsets of Nemotron-CC-v2, NVIDIA's curated English corpus, covering web text, question-and-answer material, mathematics, code and other specialised content. The work runs on NeMo Data Designer, NVIDIA's open-source framework for synthetic data, and every output keeps its source identifier so it can be traced back to the original document.

For the twelve dual-script languages, documents are translated whole, so the model can use context across paragraphs, tables and equations. NVIDIA chose Gemma-4-31B-it as translator after comparing models on complete documents. The prompts separate prose to translate from content to preserve, such as code, equations, URLs and formatting.

The same model then grades each translation against its English source using a rubric inspired by Multidimensional Quality Metrics, an open framework for classifying translation errors, covering meaning, fluency, terminology, style and formatting. Only outputs scoring at least 75 out of 100 are kept, a rubric score rather than an accuracy percentage. Lower-scoring outputs are repaired using the grader's feedback and reassessed; that repair step recovered 438.4 billion tokens that are included in the total.

Romanized text is made in a second step from accepted native-script translations rather than directly from English. NVIDIA says an initial human comparison found this two-step approach produced fewer hallucinated additions and less leftover English than romanizing straight from the English source.

Ten harder languages, a different workflow

The other ten languages needed a different approach because of long-document quality and script-stability problems. For them, translation is done by Indic Translate, a document-level model for English and 22 Indian languages from Bodhan AI, a nonprofit centre for AI in education incubated at IIT Madras. Gemma-4-31B-it still grades the results against the same 75-point threshold, but there is no feedback-based rewriting and no romanized output.

Two languages got special handling. Long Santali documents are translated in overlapping windows and stitched back together. Manipuri is first translated into Bengali script and then converted to Meitei Mayek, its native script, a pivot the open plugin supports as an optional intermediate-language step.

What it did for Nemotron 3.5 Super VL

The corpus has already been used once. During continued pretraining of Nemotron 3.5 Super VL, NVIDIA allocated 10 percent of a 100-billion-token stage, or 10 billion tokens, to a mix of synthetic text from a twelve-language subset of this corpus and natural Indic text. The rest of the stage kept the earlier training mixture.

According to the Nemotron 3.5 technical report cited in the blog, scores rose on all five reported Indic knowledge and reasoning benchmarks, MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy and Indic GSM8K, with gains between 4.98 and 9.82 percentage points. NVIDIA notes that this compares checkpoints across a stage that mixed synthetic and natural data, so it does not isolate the synthetic corpus's own contribution.

Comparing the released Nemotron 3.5 Super VL with the earlier Nemotron 3 Super, NVIDIA reports higher scores on five of six Indic task benchmarks and improvements on all four dimensions of IndiVibe, Sarvam AI's open benchmark of assistant-style prompts in Indian languages. Instruction following on IFEval 1.5 dipped by 2.04 points. NVIDIA says it plans to build on the corpus in future Nemotron models.

How developers can use it

Once the dataset is published, configurations will use language, script and region tags; hi-Deva-IN is Hindi in Devanagari and hi-Latn-IN is romanized Hindi. NVIDIA's blog includes a streaming example with the Hugging Face datasets library and tells readers to review the dataset card and licence before training and to pin a dataset revision for reproducible experiments. The licence was not stated in the blog.

The pipeline is usable now. The Indic translation plugin in NVIDIA's DataDesignerPlugins repository, which is Apache 2.0, registers a document-translation generator that translates a text column, scores it with an MQM-based evaluator, optionally rewrites low-scoring outputs, and supports windowing for long documents and an intermediate-language pivot. It has a generic mode for instruction-tuned models such as Gemma and a mode tuned for Indic Translate.

NVIDIA's advice for a first study is to compare mixtures at a fixed token budget, record the shares of natural versus synthetic and native versus romanized text, and track per-language results alongside English capability retention. Because the same English source appears in several languages and scripts, held-out evaluation splits need to account for those shared origins to avoid leakage.

Limits the authors spell out

The blog is direct about what the corpus cannot do. Translation broadens access to source material "but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture." The authors recommend mixing it with appropriately sourced natural Indic text, an issue regional-language efforts such as the Falcon Emirati speech and OCR models and Jais 3 face too.

Quality control is mostly automated. Human review was sampled and its coverage varies by language. In the twelve-language workflow, the same Gemma model is both translator and judge, a setup that can miss errors the model systematically makes, and the authors say the ten-language workflow needs broader native-speaker validation. Source-ID checks remove processing duplicates but are not semantic deduplication.

Evaluation is the other half of the work. AI4Bharat, with NVIDIA's support, is building Indic-LLM-Eval-Suite, an open collection of benchmarks for Indic model capabilities and language quality. The useful next milestones are the dataset's actual release with a clear licence, and outside groups reporting what it does for models other than Nemotron.