How far can a 93M phone recognizer go? Auditing varna against its 600M teacher across 12 Indian languages

TL;DR — Our 93M-parameter phone recognizer varna reads speech in 11 Indic languages + Indian English and emits IPA phones at 19.15% phone error rate (PER). We just measured what a 600M teacher model achieves through the exact same label pipeline: 6.85%. That 12-point gap is modeling headroom, not label noise — which reorders our entire roadmap. Along the way we audited every one of the 624,652 training rows, found (and quantified) a label artifact that makes Assamese look 3× worse than it is, discovered our validation split was partially contaminated while our test split is clean, and built a 2,400-clip golden review set. If you're a native speaker of an Indian language, we'd love your help verifying 200 short clips — details at the end.

What varna is

varna is a Conformer-CTC model (93M params) that maps 16 kHz audio directly to a stream of IPA phones over a shared 110-token vocabulary covering 11 Indic languages and Indian English — 106 phones plus word boundaries and a sentence-end token. No language ID, no lexicon, no VAD: one model, one phone inventory, all languages. Labels come from text transcripts via espeak-ng grapheme-to-phoneme (G2P) conversion.

Version 2 was trained on 1,222 hours (~100 h/language, balanced) and scores 19.15% macro PER on a frozen FLEURS-based test set, improving every single language over v1.1 (29.69%).

The question that gates everything: are the labels the ceiling?

Phone labels generated by a rule-based G2P are imperfect: espeak emits citation-form ("dictionary") pronunciations, real speakers delete schwas and reduce vowels, and some espeak language rules are much more mature than others. Before investing in bigger training runs, we needed to know: how much of our 19.15% is model error, and how much is an irreducible label floor?

The classical answer needs human phonetic transcriptions — slow and expensive. We found a cheap proxy first: take IndicConformer-600M (AI4Bharat's state-of-the-art Indic ASR model, and varna's original distillation teacher), transcribe our test clips to text, then push that text through the exact same normalize → espeak → segment pipeline that produced our reference labels, and score the result as if it were a phone hypothesis.

The trick is that systematic G2P error cancels: where the teacher hears the words correctly, its phones match the reference phones even where espeak's pronunciation rules are wrong — both sides went through the same rules. What remains is genuine recognition difference. The number this produces is a reachable operating point, not a theoretical bound: a model at least as good as "teacher + same G2P" demonstrably exists.

The result: 6.85% is reachable. We're at 19.15%.

100 clips per language, frozen test set, identical scoring:

Language	Teacher	varna v2	v2 ÷ teacher	Language	Teacher	varna v2	v2 ÷ teacher
Malayalam	4.50	12.21	2.7×	Telugu	7.66	15.58	2.0×
Bengali	5.18	14.89	2.9×	Marathi	6.40	18.44	2.9×
Hindi	5.33	15.13	2.8×	Punjabi	5.75	18.66	3.2×
				Tamil	10.11	21.43	2.1×
Kannada	6.47	10.53	1.6×	Odia	8.41	23.78	2.8×
Gujarati	7.44	15.86	2.1×	Assamese	8.05	44.09	5.5×
				Macro	6.85	19.15	2.8×

Three things jump out.

First, the floor is low everywhere. No language's teacher PER exceeds 10.2%. The "espeak floor" we worried about — the doc gating question was "if the floor is ~18%, fix labels; if ~10%, train more" — is at most ~7%, and even that overstates it: the teacher's errors include its own genuine ASR mistakes and a systematic digit-formatting mismatch (it writes "నైన్టీన్ సిక్స్టీ" where the reference kept "1960" and expanded it differently). Conclusion: train more. Specifically, knowledge distillation from the teacher at full data scale — which we had only ever run at 100 total hours — is now the top-priority move.

Second, Assamese was never really a 44%-language. The teacher hits 8.05% on the same clips — including the broken labels. espeak's Assamese voice can't map several Assamese-specific characters (ৰ, য়, ্…) and emits their Unicode character names as pseudo-phones ("hɔʃonto", "bindu"). We verified this live: phonemizing a clean FLEURS sentence appends ontostedʒɔ bindu to the phone string. The teacher "gets these right" because its correct text passes through the same broken rules — but varna, which must produce phones from audio, cannot: those tokens have no acoustic correlate. Assamese doesn't need more data (a 12× data increase barely moved it); it needs its G2P fixed — our v3 plan maps Assamese codepoints to their Bengali equivalents before espeak, which handles them correctly.

Third, difficulty is not what the raw PER table suggests. The v2÷teacher ratio is a better lens on model difficulty than absolute PER:

Kannada (1.6×) is our best-modeled language — closest to its reachable floor, likely because Kathbath/FLEURS Kannada is clean read speech with a compact effective phone inventory (41 phones covering 99% of tokens).
Tamil is intrinsically hard for everyone — hardest for the teacher (10.11%) and high for varna, with the fastest phone rate in our corpus (13.9 phones/sec median vs ~9–11 elsewhere) and over a thousand training clips where the label barely fits the audio frames. Agglutinative morphology plus espeak's verbose Tamil output (geminates, cluster expansion) squeeze the CTC alignment budget.
Punjabi (3.2×) is the sleeper problem. Its espeak inventory carries no tone marks — but Punjabi is genuinely tonal (the historical voiced aspirates became tones). The label set literally cannot express contrasts the language makes, and Punjabi shows the worst model-to-teacher ratio after Assamese. Tone-aware G2P is on the v3 list.
Odia and Marathi (2.8–2.9×) have no single smoking gun — they're the honest "more modeling, more data diversity" cases where distillation should pay directly.
English is unmeasured. Our test split has no English rows at all; the validation "measurement" is 100 lecture clips scored against a model trained 80% on LibriSpeech audiobooks, with phones from espeak's American English voice — for Indian-English speech. Every one of those choices mismatches the others. English needs its own eval before any claim about it means anything.

What the full-corpus audit found

While the teacher ran, we audited all 624,652 rows of the training corpus end-to-end. Findings worth sharing:

The test set is clean; validation is not. Zero test transcripts appear in training. But FLEURS reuses sentences across splits with different speakers, so 12% of Malayalam and ~10% of Marathi validation rows have train-identical transcripts. Our released benchmark number stands; some validation-based conclusions (including two "regressions" we'd been chasing) dissolved under scrutiny — they were split-composition artifacts, not real.
Training consumption was perfectly balanced: 38,000 steps × batch 48 = 2.99 epochs, every language at 8.4–8.5% share. "Language X regressed because the mix underfed it" is ruled out, cleanly.
The sentence-end token is currently untestable: our dev/test rows carry zero sentence punctuation (Kathbath, IndicVoices and LibriSpeech transcripts have none — we measured 0 in 102k sampled), so </s> quality simply isn't evaluated by the benchmark yet.
The phone inventory is linguistically sane: the frequency-weighted "effective inventory" per language (phones covering 99% of tokens, 35–52 per language) matches PHOIBLE's documented inventories almost exactly. The handful of single-language phones are real (Tamil ʉ/ɻ, Malayalam ɨ), not artifacts.

New insights we'll reuse

Teacher-through-G2P is a $0 label-floor estimator. Any time labels come from a deterministic pipeline over text, you can bound label noise by pushing a stronger model's text output through the same pipeline. Error cancellation does the work; no human transcription needed for the bound.
Rule-based G2P artifacts are unlearnable-from-audio, and that's diagnostic. A language that refuses to improve with more data while its teacher-through-G2P number is fine = broken labels, not a hard language.
Cross-split sentence reuse in multi-speaker corpora silently contaminates text-derived-label tasks. For word-level ASR, same-text/different-speaker across splits is mild; when your labels are a deterministic function of the text, it's memorization fuel. Audit it explicitly.

Is there really no golden phoneme set for Indian languages?

Surprisingly — to the best of our knowledge, no open one exists at corpus scale. What exists today:

PHOIBLE documents phone inventories per language (which phones exist), but contains no audio and no transcribed utterances.
TTS lexicons (IndicTTS and friends) give word → canonical-pronunciation mappings for limited vocabularies — dictionaries, not transcribed speech.
Rule G2P engines (espeak-ng, Epitran, and Indic-specific efforts) generate pronunciations from text — exactly the machinery whose errors we're trying to measure, so they can't be their own gold standard.
English has TIMIT: real audio with human-verified, time-aligned phone transcriptions, and it shaped 30 years of ASR research. Nothing like TIMIT exists openly for any Indian language, let alone twelve of them under one consistent IPA inventory.

We think that's a gap worth closing, and we've built the scaffolding.

Call for volunteers: help us build the golden set

We've assembled golden_v1: 2,400 clips — 200 per language across Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu and Indian English — drawn from held-out evaluation data, never seen in training, with contaminated rows excluded. Every clip ships with its transcript, espeak's IPA, and automated quality flags; we also ran the 600M teacher over every clip and pre-flagged the ones where its transcript disagrees with the reference (~5% of clips), so the likely-broken rows sort to the top of each review sheet.

What a volunteer does (per language, ~2–4 hours total):

Open the language's review sheet (CSV) and listen to each clip.
Mark whether the transcript matches what's said (text_ok), fixing it if not.
Mark whether the IPA is a fair citation-form pronunciation of the transcript (pron_ok), correcting it if not. You're judging pronunciation, not accent.

Flagged rows first; unflagged rows are spot-checks of espeak itself. The corrected sheets directly yield the espeak-vs-human PER — the true label floor per language — and become a permanent, open evaluation asset.

Native speakers of Assamese and Punjabi: you are our highest-value volunteers. Assamese review confirms (or refutes) the artifact story above; Punjabi review tells us how much the missing tone marks actually cost.

If you can help with any language, get in touch and we'll send you a review kit (audio + sheet, ~70 MB per language).

What's next for varna

In priority order, all driven by today's numbers:

Full-scale knowledge distillation from IndicConformer-600M over all 1,222 hours — the 12-point gap says this is where the wins are.
Golden-set human review (that's you, hopefully!) — turns our ~7% bound into a real per-language floor.
An honest English track — a held-out Indian-English test set and an Indian-English-aware G2P, replacing the current unmeasurable setup.
Assamese G2P fix (codepoint mapping before espeak) — the single biggest per-language win available, ~15–20 points by our estimate.
Sentence-end supervision for Hindi/Telugu, plus punctuated eval rows so the </s> token can actually be tested.