logo Idiap Research Institute        
 [BibTeX] [Marc21]
A Domain Adaptation Approach to Improve Speaker Turn Embedding Using Face Representation
Type of publication: Conference paper
Citation: Le_ICMI_2017
Publication status: Accepted
Booktitle: ACM International Conference on Multimodal Interaction
Year: 2017
Month: November
Publisher: ACM
Location: Glasgow, Scotland
Abstract: This paper proposes a novel approach to improve speaker modeling using knowledge transferred from face representation. In particular, we are interested in learning a discriminative metric which allows speaker turns to be compared directly, which is beneficial for tasks such as diarization and dialogue analysis. Our method improves the embedding space of speaker turns by applying maximum mean discrepancy loss to minimize the disparity between the distributions of facial and acoustic embedded features. This approach aims to discover the shared underlying structure of the two embedded spaces, thus enabling the transfer of knowledge from the richer face representation to the counterpart in speech. Experiments are conducted on broadcast TV news datasets, REPERE and ETAPE, to demonstrate the validity of our method. Quantitative results in verification and clustering tasks show promising improvement, especially in cases where speaker turns are short or the training data size is limited.
Keywords: domain adaptation, Metric learning, Multimodal person diarization
Projects Idiap
Authors Le, Nam
Odobez, Jean-Marc
Added by: [UNK]
Total mark: 0
  • Le_ICMI_2017.pdf