CONF Hung_CVPR2008_2008/IDIAP Associating Audio-Visual Activity Cues in a Dominance Estimation Framework Hung, Hayley Huang, Yan Yeo, Chuohao Gatica-Perez, Daniel EXTERNAL http://publications.idiap.ch/attachments/papers/2008/Hung_CVPR2008_2008.pdf PUBLIC http://publications.idiap.ch/index.php/publications/showcite/Hung_Idiap-RR-66-2008 Related documents First IEEE Workshop on CVPR for Human Communicative Behavior Analysis 2008 We address the problem of both estimating the dominant person in a meeting from a single audio source and identifying them visually in a multi-camera setting. We use a speaker diarization algorithm to perform speaker segmentation and clustering, representing when they spoke. Using a greedy ordered audio-visual association algorithm, we investigate using the speaker clusters to find the corresponding person in one of the video channels. The difficulty of the problem is that firstly the speaker diarization output is noisy (e.g. for participants who speak little) and often produces an unequal number of clusters to true participants. Secondly, personal visual activity from natural upper torso motion, which can include highly deformable pose changes and perspective distortion, is computed through computationally efficient coarse features. Our results using almost 2 hours of audio-visual data from 4-participant meetings show a strong correlation between the estimated speaker diarization and visual activity features, enabling the identification of the most dominant person as a pair of audio-visual channels. REPORT Hung_Idiap-RR-66-2008/IDIAP Associating Audio-Visual Activity Cues in a Dominance Estimation Framework Hung, Hayley Huang, Yan Yeo, Chuohao Gatica-Perez, Daniel EXTERNAL http://publications.idiap.ch/attachments/reports/2008/Hung_Idiap-RR-66-2008.pdf PUBLIC http://publications.idiap.ch/index.php/publications/showcite/Hung_CVPR2008_2008 Related documents Idiap-RR-66-2008 2008 Idiap August 2008 We address the problem of both estimating the dominant person in a meeting from a single audio source and identifying them visually in a multi-camera setting. We use a speaker diarization algorithm to perform speaker segmentation and clustering, representing when they spoke. Using a greedy ordered audio-visual association algorithm, we investigate using the speaker clusters to find the corresponding person in one of the video channels. The difficulty of the problem is that firstly the speaker diarization output is noisy (e.g. for participants who speak little) and often produces an unequal number of clusters to true participants. Secondly, personal visual activity from natural upper torso motion, which can include highly deformable pose changes and perspective distortion, is computed through computationally efficient coarse features. Our results using almost 2 hours of audio-visual data from 4-participant meetings show a strong correlation between the estimated speaker diarization and visual activity features, enabling the identification of the most dominant person as a pair of audio-visual channels.