CONF Anemueller_ICMI2008_2008/IDIAP The DIRAC AWEAR Audio-Visual Platform for Detection of Unexpected and Incongruent Events Anemueller, Joern Back, Joerg-Henrik Caputo, Barbara havlena, michal Luo, Jie Kayser, Hendrik Leibe, Bastian Motlicek, Petr Pajdla, Tomas Pavel, Misha Torii, Akihiko Gool, Luc Van Zweig, Alon Hermansky, Hynek EXTERNAL http://publications.idiap.ch/attachments/papers/2008/Anemueller_ICMI2008_2008.pdf PUBLIC http://publications.idiap.ch/index.php/publications/showcite/Anemueller_Idiap-RR-41-2010 Related documents Proceedings of the International Conference on Multimodal Interfaces 2008 It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the fieldof- view does not move her lips. In the project DIRAC (“Detection and Identification of Rare Audio-visual Cues”) we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform (“AWEAR” – audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies. REPORT Anemueller_Idiap-RR-41-2010/IDIAP The DIRAC AWEAR Audio-Visual Platform for Detection of Unexpected and Incongruent Events Anemueller, Joern Back, Joerg-Henrik Caputo, Barbara havlena, michal Luo, Jie Kayser, Hendrik Leibe, Bastian Motlicek, Petr Pajdla, Tomas Pavel, Misha Torii, Akihiko Gool, Luc Van Hermansky, Hynek Zweig, Alon EXTERNAL http://publications.idiap.ch/attachments/reports/2008/Anemueller_Idiap-RR-41-2010.pdf PUBLIC Idiap-RR-41-2010 2010 Idiap November 2010 It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the fieldof- view does not move her lips. In the project DIRAC (“Detection and Identification of Rare Audio-visual Cues”) we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform (“AWEAR” – audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.