Home / Blog / Code-mixed speech data: why ASR breaks when speakers switch languages
Code-mixed speech data: why ASR breaks when speakers switch languages
Speech recognition models struggle with code-mixed audio because most training data is monolingual, so models are rarely shown a sentence that changes language halfway through. Fixing it takes real recordings of people switching languages naturally, transcribed with a consistent script convention and tagged by language.
Code-mixing (or code-switching) is when a speaker moves between two languages within a conversation, often within a sentence. It is ordinary speech for a large share of the world's multilingual speakers: Hindi and English, Spanish and English, Arabic and French, Tagalog and English, Malay and English, and many more. Support calls are full of it.
A typical turn from a support call looks like this:
Why ASR models fail on it
- Training data is mostly monolingual. Models learn strong expectations about which words follow which. A switch mid-sentence breaks those expectations, so the model often "corrects" the second language into the first.
- Script choice is ambiguous. Should Hindi words be written in Devanagari or in Latin script? Real users type and expect Latin script; many models output the other. Without a convention, both the training data and the evaluation are inconsistent.
- Language ID breaks down. Pipelines that first detect the language and then pick a model can't handle a clip that is both.
- Call-centre audio is noisy. Narrowband phone audio, background noise and crosstalk compound the problem.
What good code-mixed training data looks like
| Property | Why it matters |
|---|---|
| Natural speech, not read scripts | People switch differently when speaking freely than when reading prepared sentences |
| Consistent script convention | Decide, for example, "Hindi in Latin script" and apply it everywhere, so word error rates mean something |
| Word- or span-level language tags | Lets you measure accuracy on each language separately and on the switch points |
| Speaker and channel metadata | Speaker turns, timestamps, and phone versus wideband audio |
| De-identification of the audio and text | Names, order numbers and phone numbers spoken aloud must be redacted in both |
| Recorded consent | Both the organisation and, where required, the callers must have agreed to the use |
Evaluate on the switch points
An overall word error rate can look fine while the model fails at exactly the moments a speaker switches. Report error rates separately for each language and for the words around switch points. Research benchmarks for code-switched language such as LinCE and GLUECoS, and the CALCS workshop series on computational approaches to code-switching, are good places to see how the research community measures this.
Collecting it with consent
The richest source of natural code-mixed speech is recorded customer calls, which belong to the companies that made them. Licensing them takes a written agreement with the company covering AI training, a check that callers were notified about recording, and redaction of personal details spoken in the call before the audio leaves the company.
How we do it
Datadhara licenses call-centre audio with checked, code-mixed transcripts, speaker turns, timestamps and language tags, redacted at the source. See the speech data page, or download the free synthetic call-transcript sample to see the format.
Questions
What is code-mixed speech?
Speech in which the speaker switches between two or more languages within a conversation or a single sentence, such as Hindi and English or Spanish and English.
Why do speech recognition models struggle with code-switching?
Most training data is monolingual, so models rarely see a sentence that changes language. They tend to force the second language into the first, and pipelines that detect a single language per clip fail on mixed clips.
How should code-mixed transcripts be written?
Pick one script convention per language pair and apply it everywhere, and tag each word or span with its language, so error rates are consistent and can be measured per language.
Tell us what your models are missing
A few lines is enough: the domain, the format, and roughly how much.
Request data →