logo Idiap Research Institute        
 [BibTeX] [Marc21]
End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation
Type of publication: Conference paper
Citation: Juan_EMNLP_2023
Publication status: Published
Booktitle: The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Series: 1
Volume: 1
Number: 1
Year: 2023
Month: December
Location: Singapore
URL: https://arxiv.org/abs/2311.006...
Abstract: Conventional speech-to-text translation (ST) systems are trained on single-speaker utterances, and they may not generalize to real-life scenarios where the audio contains conversations by multiple speakers. In this paper, we tackle single-channel multi-speaker conversational ST with an end-to-end and multi-task training model, named Speaker-Turn Aware Conversational Speech Translation, that combines automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format. We run experiments on the Fisher-CALLHOME corpus, which we adapted by merging the two single-speaker channels into one multi-speaker channel, thus representing the more realistic and challenging scenario with multi-speaker turns and cross-talk. Experimental results across single- and multi-speaker conditions and against conventional ST systems, show that our model outperforms the reference systems on the multi-speaker condition, while attaining comparable performance on the single-speaker condition. We release scripts for data processing and model training.
Projects Idiap
Authors Juan, Zuluaga-Gomez.
Huang, Zhaocheng
Niu, Xing
Srinavasan, Sundararajan
Mathur, Prashant
Thompson, Brian
Federico, Marcello
Added by: [UNK]
Total mark: 0
  • Juan_EMNLP_2023.pdf
       (Paper - Arxiv)