No diacritics in the references.
None of the training corpus’s accented spellings appeared as written, while 69.4% matched once the accents were folded away. Training moved to folded text.
04 / SELECTED WORK SPEECH AI · AUG — OCT 2026
Javanese ASR.
Fine-tuning Whisper for code-switched Indonesian–Javanese speech in Mozilla’s Lost in Transcription challenge
The model was the easy part.
Knowing which numbers to trust was not.
WHAT ZERO-SHOT WHISPER WROTEJEMBER CORPUS / DEV SPLIT
| Speaker said | became | Whisper wrote | Meaning | Times |
|---|---|---|---|---|
| ikuJAVANESE | that | 121 | ||
| singJAVANESE | which, who | 62 | ||
| dadiJAVANESE | so, become | 56 | ||
| ndèkJAVANESE | at, in | 51 |
- MY ROLE
- Research and pipeline, end to end
Team Bulu Hidung (Indonesian for “nose hair”) - BUILT WITH
- Whisper large-v3-turbo · PyTorch · Transformers
Kaggle T4 · Hugging Face · 14 notebooks - CODE
- Public on GitHub · MPL 2.0
Competition data stays out, as the rules require
OFFICIAL RESULT
The result, on the board.
The official private leaderboard lists team Bulu Hidung at #18 of 128 ranked entries, with a word error rate of 0.2576. This snapshot was captured on .
View the live official leaderboard (opens in a new tab)
01 / THE PROBLEM
Whisper heard Javanese.
Then wrote Indonesian.
The challenge, run by Mozilla Data Collective with DrivenData, asked for transcripts of conversational voice notes where speakers switch between Indonesian and Javanese mid-sentence. Scoring is word error rate (WER): the share of words a system gets wrong.
Out of the box, Whisper treated Javanese as accented Indonesian. It caught the sound of a word, then wrote its Indonesian equivalent or dropped it. That is a language problem, not an acoustic one, which made fine-tuning worth trying.
- 2.4×
- Javanese words were wrong 2.4 times as often as Indonesian ones (WER 0.437 against 0.184).
- 33%
- of substitutions on Javanese words replaced them with an Indonesian word.
- 0.2695
- Zero-shot WER on the official dev set: the number to beat.
Zero-shot whisper-large-v3-turbo, measured on the official dev set (372 clips). Only aggregate scores are shown.
02 / THE CONSTRAINTS
One free GPU.
No internet at grading time.
Before writing any code, I wrote a feasibility study: what the rules allow, what the compute can carry, and whether the timeline was realistic. The answer was go. The working rule that followed: every run changes one variable, so each result can be explained.
Notebooks were written locally and run on Kaggle. Each run saved a small digest of its numbers back to the repository, and every submission went into a ledger with its ZIP hash.
- Compute1 × T4
One free Kaggle GPU for every run. The budget was zero.
- GradingOffline
The grading container has no internet, so the weights ship inside a 1.6 GB ZIP.
- Time limit2 hours
To load the model and transcribe 2,118 test clips.
- Data ruleAggregates
Competition transcripts and per-clip predictions are never published. Only aggregate scores leave the notebooks.
03 / READ THE EXAM FIRST
Study the answer sheet.
Then the material.
The training corpus and the official dev set looked alike on paper. Measured side by side, they were not, and two early decisions had to be reversed.
Clips five times longer.
Dev clips averaged 24.9 seconds against 5.4 in training, and 10.2% ran past Whisper’s 30-second window. Training segments were joined into roughly 25-second items.
0.2695 → 0.2524
On the official dev set, with a confidence interval that excludes zero. The run was also cheaper: 2.45 hours instead of 4.36. The corpus’s own dev split ranked the two models the other way round.
04 / TEN FULL RUNS
Ten test runs.
One change at a time.
The team could send three full submissions every seven days. Each one tested a single change against an earlier run. Lower is better.
PUBLIC TEST WER BY SUBMISSIONLOWER IS BETTER
- E4
First full run: Jember + 6.5 h of curated Common Voice Indonesian
24 Sep0.3005 - A
Jember only. Dev said E4 was 0.034 better; on test the gap was 0.002
24 Sep0.3027 - B
E4 with beam search (5 beams)
24 Sep0.2950 - D
Native Jember segments instead of 25-second joins
25 Sep0.2901 - F
Stage 2: three short epochs on the official dev set
28 Sep−0.0240.2660 - P
F plus 30 post-processing rewrite rules: a tie
29 Sep0.2664 - H
Stage 2 with no stage 1. Dev predicted −0.012
30 Sepdev wrong0.3088 - J
Stage 1 for two epochs instead of one: a tie
1 Oct0.2655 - K
+ 7.7 h of Indonesian podcast speech. Dev predicted −0.022
2 Octdev wrong0.2698 - L
+ 8 h of Central Javanese TTS speech in stage 1
3 Octbest0.2497
05 / THREE LESSONS
What the ten runs
actually taught.
One lesson came with a correction. Early on, I left the Javanese TTS data out on reasonable evidence. When the Indonesian podcast run failed on test, the pattern pointed back to Javanese data, and the TTS run became the best one. Writing down why each decision was made is what made it easy to reopen.
- 01
Learn the grader’s handwriting.
A real share of the remaining error was not mishearing but the references’ writing convention: capitals, punctuation, fillers, spelling variants. A short fine-tune on the official dev set taught it.
Test 0.2901 → 0.2660, the largest gain of the project - 02
Data in the right language beats more of everything.
A second epoch did nothing, and Indonesian podcast speech made test worse. Javanese speech, even synthetic, moved it.
Javanese-word WER 0.243 → 0.207 · test −0.016 - 03
A small dev set can point the wrong way.
Four speakers in two conversations could not judge stage-1 changes. Every claim got a paired confidence interval, and ties went to a test submission.
Wrong direction twice: H and K
06 / THE FINAL PIPELINE
Two stages of training.
One proof before shipping.
The final submission, arm L, fine-tunes whisper-large-v3-turbo with the Javanese language token. Every ZIP was checked against the training notebook’s own predictions before it was sent.
- 01 / STAGE 1
Learn Javanese
- Jember spontaneous speech · 8.4 h
- Curated Common Voice Indonesian · 6.5 h
- Central Javanese TTS · 8.0 h
- One epoch, final step kept
- 02 / STAGE 2
Learn the convention
- Official dev clips up to 30 s · 335 of 372
- Three epochs at LR 3e-6
- About 12 minutes on a T4
- 03 / PACKAGE
Prove it runs
- fp16 weights + main.py in one ZIP
- Beam 5, chunked decoding past 30 s
- Identical on 372 of 372 clips in a replica of the grading container
07 / DEAD ENDS
What didn’t work
is a result too.
Each idea was measured or weighed, then closed when the evidence said so. Knowing when to stop saved GPU hours for the runs that mattered.
- Whisper large-v3
Worse than turbo zero-shot (+0.014 WER, 95% CI [+0.005, +0.022]), and a full fine-tune ran out of memory on a T4 even at batch size 1.
- omniASR LLM 3B, zero-shot
0.2518 on the official dev set, against 0.2268 for our fine-tuned model.
- 30 post-processing rules
Test 0.2664 against 0.2660 without them. A tie, so they stayed out.
- XLS-R CTC + language model
Not pursued: KenLM needs packages outside the official runtime, while the Whisper path needs only PyTorch and Transformers.
THE WRITE-UP AND THE CODE
The full story,
and the code behind it.
The Medium article tells the whole story, from the first zero-shot run to the dev set that pointed the wrong way. The repository holds every notebook, the reports behind each decision, and a ledger of all ten full submissions. Model weights and competition data are not included.