I recently moved to the US. Long before I moved, I built my very first voice agent. It didn’t take long to realize a fundamental flaw in standard architectures: the underlying models were heavily biased towards standard American accents.
The moment an input featured a non-standard accent, regional dialect, or code-switched phrasing, the pipeline broke down. Misinterpretations escalated, Word Error Rates (WER) spiked, and downstream processing costs exploded as LLMs burn tokens trying to make sense of mangled text. Working in a call centre environment in the US has made the reality of this problem undeniable.
Automated systems currently fail for a different reason: downstream processes are completely audio-blind. A downstream LLM or intent-parser cannot hear the original audio. It has no access to the speaker’s tone, acoustic cadence, or native phrasing. If the ASR engine misinterprets an accented phrase, the downstream model blindly accepts that corrupted text as truth, guesses the caller’s intent, and fails.
This is why I started Accent Labs: to fix the transcription bottleneck so voice technology can accurately serve every person.
WER Lies in Dialect AI (And What I’m Building to Fix It)
When evaluating speech recognition for low-resource languages and code-switched dialects like Nigerian English or West African Pidgin, standard Word Error Rate (WER) is a not an informative metric.
During my testing, I kept running into a frustrating issue: my ASR model would transcribe speech perfectly, capturing exactly what was spoken, yet it would still receive poor scores on the WER scorecard.
The Flaw of Standard WER in Multilingual Contexts
Standard WER is an edit-distance calculation that counts insertions, deletions, and substitutions against a ground-truth text. It does not have semantic awareness.
Every error flagged by the metric was often just a matter of formatting, punctuation or orthographic shifts. The model showed no signs of hallucinating; it was disagreeing with the transcriber’s spelling choices.
Stripping hesitations (e e e) when a speaker stutters, is flagged as a deletion penalty. Converting phonetic representations like "monino" to "money no" counts as a substitution error. Capitalising "i" to "I" or inserting commas creates string-distance penalties.
The WER jumped from 0.11–0.22, up to 0.333.
This isn’t a sign of model degradation, my language model is actively cleaning up acoustic noise. This normalisation process adds latency downstream so I tried something different.
Re-evaluating the LLM Layer: Gemma 4 vs Claude Haiku
Rather than forcing a general-purpose model to act as a grammarian, I shifted my focus to Gemma 4 26B MoE (google/gemma-4-26b-a4b-it).
Because of its Mixture-of-Experts architecture, Gemma 4 only activates a fraction of its parameters during inference, requiring significantly less VRAM while allowing low latency. It’s multi-token routing is better at preserving dialect markers (waka, dey, sha) compared to SOTA models.
The Prompt Shift:
Moving away from defensive prompts and few-shot examples in favour of a simple directional rule dropped my Gemma 4 WER down to 0.333, outperforming Claude Haiku (0.344 WER) with lower operational costs by over 80%.
The Claude Haiku Dynamics:
Claude Haiku’s slight performance dip when prompt-constrained occurred because SOTA models are highly sensitive to guardrails. Over-prompting caused Haiku to over-correct valid dialect syntax into Standard English.
Closing the Loop: The Audio Label Ecosystem
Achieving near 0% WER on un-normalised ground truth is impossible through prompting alone. The long-term solution is establishing a Human-in-the-Loop (HITL) data flywheel.
To scale this, I am moving forward with Audio Label to power a crowdsourced human review ecosystem.
[ Raw Audio ] ──> [ Acoustic ASR ] ──> [ Gemma 4 Normaliser ]
│
▼ (Flagged Edge Cases)
[ Audio Label ]
│
▼ (Human-Verified Pair)
[ Fine-Tuning Dataset ]
To make human annotation as seamless as possible, I am designing the platform around a lightweight, link-based workflow:
-
Team Project Links: Any team or project can generate a unique, dedicated link for their project. Native speakers and annotators can jump straight into a review session.
-
Verification: Low-confidence transcriptions and dropped dialect markers are automatically routed to the queue, letting reviewers fix acoustic edge cases.
-
Automated Data Harvesting: Every human correction creates a verified training pair
(Raw Audio -> Verified Text)directly into my dataset pipeline.
Conclusion: Trust Your Model and Iterate
Building speech technology for underrepresented dialects is an iterative process. My fine-tuned ASR already understands the phonetic nuances of Nigerian English and Pidgin far better than a general-purpose LLM.
By combining the lightweight, open-weights efficiency of Gemma 4 with a structured human-in-the-loop review platform, teams can continuously convert edge-case failures into proprietary training data and build an end-to-end speech model.
This article was published under HackerNoon’s Business Blogging program.