Home / Blog / Code-mixed speech data: why ASR breaks when speakers switch languages

Code-mixed speech data: why ASR breaks when speakers switch languages

Speech recognition models struggle with code-mixed audio because most training data is monolingual, so models are rarely shown a sentence that changes language halfway through. Fixing it takes real recordings of people switching languages naturally, transcribed with a consistent script convention and tagged by language.

Code-mixing (or code-switching) is when a speaker moves between two languages within a conversation, often within a sentence. It is ordinary speech for a large share of the world's multilingual speakers: Hindi and English, Spanish and English, Arabic and French, Tagalog and English, Malay and English, and many more. Support calls are full of it.

A typical turn from a support call looks like this:

customer: Haan hi, mera order abhi tak deliver nahi hua, it was supposed to come yesterday.

Why ASR models fail on it

What good code-mixed training data looks like

PropertyWhy it matters
Natural speech, not read scriptsPeople switch differently when speaking freely than when reading prepared sentences
Consistent script conventionDecide, for example, "Hindi in Latin script" and apply it everywhere, so word error rates mean something
Word- or span-level language tagsLets you measure accuracy on each language separately and on the switch points
Speaker and channel metadataSpeaker turns, timestamps, and phone versus wideband audio
De-identification of the audio and textNames, order numbers and phone numbers spoken aloud must be redacted in both
Recorded consentBoth the organisation and, where required, the callers must have agreed to the use

Evaluate on the switch points

An overall word error rate can look fine while the model fails at exactly the moments a speaker switches. Report error rates separately for each language and for the words around switch points. Research benchmarks for code-switched language such as LinCE and GLUECoS, and the CALCS workshop series on computational approaches to code-switching, are good places to see how the research community measures this.

Collecting it with consent

The richest source of natural code-mixed speech is recorded customer calls, which belong to the companies that made them. Licensing them takes a written agreement with the company covering AI training, a check that callers were notified about recording, and redaction of personal details spoken in the call before the audio leaves the company.

How we do it

Datadhara licenses call-centre audio with checked, code-mixed transcripts, speaker turns, timestamps and language tags, redacted at the source. See the speech data page, or download the free synthetic call-transcript sample to see the format.

Questions

What is code-mixed speech?

Speech in which the speaker switches between two or more languages within a conversation or a single sentence, such as Hindi and English or Spanish and English.

Why do speech recognition models struggle with code-switching?

Most training data is monolingual, so models rarely see a sentence that changes language. They tend to force the second language into the first, and pipelines that detect a single language per clip fail on mixed clips.

How should code-mixed transcripts be written?

Pick one script convention per language pair and apply it everywhere, and tag each word or span with its language, so error rates are consistent and can be measured per language.

Tell us what your models are missing

A few lines is enough: the domain, the format, and roughly how much.

Request data →