SHUDDHI · SOVEREIGN DATA FACTORY
A model is what it eats.
AgentAnywhere Shuddhi — शुद्धि, purification — is the data factory behind our from-scratch models. It takes raw text and code through a staged cleansing pipeline — provenance gating, language identification, deduplication, quality scoring, domain balancing, contamination checks — and seals the result with a corpus build manifest. Every training run cites the manifest of exactly what it learned from: provenance receipts for training, the same discipline our serving layer applies to every call.
The cheapest useless token is the one you never train on.
Every duplicated document, boilerplate page, licence-ambiguous scrape, or contaminated benchmark answer that reaches a training run burns GPU-hours teaching the model nothing — or worse, teaching it something you cannot defend. For a resource-constrained sovereign program, corpus composition is not a preprocessing chore; it is one of the largest cost and quality levers that exists.
Most teams treat data preparation as ad-hoc scripts run once and forgotten. Nothing is measured, nothing is repeatable, and when a regulator, customer, or court asks what exactly did this model learn from? — there is no answer.
Shuddhi makes the corpus an engineered, audited product: a staged pipeline with a measured report at every stage, and a sealed manifest at the end that answers that question precisely.
Seven stages. Every one measured.
Provenance gate — every shard enters tagged with source, licence, and data class. Untagged material is refused. Customer data is excluded mechanically at this stage: the pipeline will not accept it, so no downstream mistake can train on it.
Language identification — the Indic/English mix is controlled deliberately, not inherited accidentally from whatever was scraped.
Deduplication — exact and near-duplicate removal, so the model never pays twice to read the same page.
Quality scoring — coherent, information-rich material is prioritised; boilerplate, spam, and junk are scored out.
Domain balancing — tokens are allocated across coding, reasoning, Indic languages, and regulated domains by decision, with the mix reported.
Contamination check — the corpus is screened against our evaluation batteries, so a benchmark result can never be an echo of training data. This protects the integrity of every number we publish.
The build manifest — the accepted corpus is sealed under a content hash with its full composition report: tokens by domain and language, dedup rate, quality distribution, and every exclusion counted by reason.
Provenance receipts — for training, not just serving.
On this platform, every served call carries a signed Trust Receipt. Shuddhi extends the same discipline upstream: every training run cites the corpus build it consumed — a manifest hash in the training ledger, with the composition report behind it.
That turns *what did this model learn from?* into a question with a precise, auditable answer — for a regulator applying data-protection law, for a defence customer demanding clean lineage, for a BFSI risk office proving licence compliance, and for our own engineers reproducing a run months later.
We built Shuddhi because we needed it ourselves: our from-scratch model corpora are its first customer — and the first sealed build is already done.
The build, applied: tatva-sangraha-v1.2.
Shuddhi processed our full 176.1 GB from-scratch training corpus — 33,047,370 documents across 15 languages, all CC-BY-4.0 licensed — end to end on one 2-vCPU CPU box. Zero GPU, zero data egress.
The checklist is closed: every filter now runs on the whole corpus, not a sample. Shuddhi's chain — exact and near-duplicate removal, quality scoring, perplexity screening, a toxicity pass, contamination checking, and PII redaction — is applied and receipted end to end.
The filtered build kept 32,289,800 of 33,047,370 documents (97.71%), every drop counted by reason: 127,024 exact duplicates (0.38%), 272,640 near-duplicates (0.83%), 345,505 perplexity outliers (1.05%, a p99 cutoff by design), 7,086 low-quality (0.02%), and 5,306 toxicity (0.016%) — the last a conservative lexicon tier that removes the unambiguous tail, not a comprehensive safety classifier; that tier is on the roadmap.
Contamination: zero — every one of 33,047,370 documents checked against our evaluation sets, not a sample, so the benchmark numbers we publish measure capability, not memorisation. 182,781 PII spans redacted on emission (email, phone, Aadhaar, PAN, Luhn-checked cards, IP). Tokens stay an estimate — ~45.9 billion (derived) under our own tokenizer — which is why the headline figures are byte- and document-denominated (176 GB, 33 million documents): a token count shifts with the tokenizer, the corpus does not.
The result is sealed under a filtered build hash chained to the source-corpus hash, the exact filter config, and the tokenizer — the config pins the *tools*, not just the thresholds: the near-duplicate droplist, the toxicity lexicon, and the tokenizer are all hashed into the receipt, so every training run cites corpus, filters, and tokenizer together in its ledger. Because the pipeline is partition-invariant, builds parallelise without changing the receipt.
One result is worth dwelling on. Run at full scale, the largest near-duplicate cluster was 84,275 copies of a single document — and it sat in the English shard (2.4% near-duplicates), not where an earlier sampled pass had suggested. Sampling hinted; the full-scale receipt corrected it. That is precisely why filters have to be *applied* at scale, not estimated.
Built for perimeters, like everything else here.
Shuddhi is CPU-first and runs inside your boundary — no corpus leaves your infrastructure to be cleaned. For organisations building their own sovereign models or fine-tuning corpora, that matters as much as the cleaning itself: the data factory has to be as sovereign as the model it feeds.
It slots into the platform the way the other modules do: provenance ledgers alongside TrustFabric's signed claims, exclusion policy under Governance, and the corpus manifests visible where your training and evaluation history lives.
FAQ
Frequently asked questions.
- What is AgentAnywhere Shuddhi?
- AgentAnywhere Shuddhi is a sovereign data factory — a staged corpus-cleansing engine that takes raw text and code through provenance gating, language identification, deduplication, quality scoring, domain balancing, and contamination checks, then seals the accepted corpus with a build manifest that training runs cite. It is in development now; our own from-scratch model corpora are its first customer.
- What is a corpus build manifest?
- A sealed record of exactly what a training corpus contains: a content hash over the accepted data plus a composition report — tokens by domain and language, deduplication rate, quality distribution, and every exclusion counted by reason. Training runs cite the manifest hash in their ledgers, so 'what did this model learn from?' has a precise, auditable answer.
- How does Shuddhi handle customer data?
- It refuses it, mechanically. Every shard must enter with source, licence, and data-class tags; customer data is excluded at the provenance gate — the first stage — so no downstream step can train on it by mistake. Exclusion is enforced by the pipeline, not by a policy document.
- Why does contamination checking matter?
- If benchmark questions leak into training data, benchmark results become echoes rather than measurements. Shuddhi screens the corpus against our evaluation batteries before sealing, so the numbers we publish measure capability, not memorisation. It is part of the same receipts discipline the rest of the platform runs on.
- Is Shuddhi available today?
- The v1 engine is running today on our own corpora — its first production build (tatva-sangraha-v1: 33 million documents, 15 languages) is sealed, the build hash reproduced across four independent full passes, and a filtered, PII-redacted build produced with a receipt chained to the source. As an external product it is in development: organisations building sovereign corpora can talk to us about the design-partner program. Status stated plainly, per the same rule as every model on this site.
Feed your model something you can defend.
Shuddhi is in development, with our own model corpora as its first customer. If you are building a sovereign corpus — for a from-scratch model or a regulated fine-tune — talk to us about the design-partner program.