Avoiding Catastrophic Forgetting in Text-Only Adaptation of LLM-based ASR via Multi-View Text Denoising
| Type of publication: | Conference paper |
| Citation: | Burdisso_INTERSPEECH_2026 |
| Publication status: | Accepted |
| Booktitle: | INTERSPEECH 2026 |
| Year: | 2026 |
| Abstract: | Adapting LLM-based automatic speech recognition (ASR) systems to new domains using text-only data is challenging. Naively fine-tuning the LLM on target text disrupts the speech–text alignment learned by the projector, causing catastrophic forgetting. We propose a lightweight text-only adaptation strategy that formulates adaptation as a denoising task and introduces a multi-view noise-driven batching scheme to preserve alignment. Each mini-batch mixes paired source audio–text examples, projector-induced noisy transcripts, synthetically corrupted source transcripts, and corrupted target transcripts. This mixing enables domain adaptation while maintaining speech–text alignment. Our method requires no architectural changes or additional parameters and achieves up to 25.4% relative WER improvement over the non-adapted base model, outperforming recent text-only adaptation methods. |
| Additional Research Programs: |
AI for Everyone |
| Keywords: | |
| Projects: |
UNIPHORE ELOQUENCE |
| Authors: | |
| Added by: | [UNK] |
| Total mark: | 0 |
|
Attachments
|
|
|
|
|
Notes
|
|
|
|
|
