REPORT Korchagin_Idiap-RR-10-2011/IDIAP Just-in-Time Multimodal Association and Fusion from Home Entertainment Korchagin, Danil Motlicek, Petr Duffner, Stefan Bourlard, Hervé association rules data analysis multimodal signal processing sensor fusion EXTERNAL http://publications.idiap.ch/attachments/reports/2011/Korchagin_Idiap-RR-10-2011.pdf PUBLIC Idiap-RR-10-2011 2011 Idiap Martigny, Switzerland May 2011 In this paper, we describe a real-time multimodal analysis system with just-in-time multimodal association and fusion for a living room environment, where multiple people may enter, interact and leave the observable world with no constraints. It comprises detection and tracking of up to 4 faces, detection and localisation of verbal and paralinguistic events, their association and fusion. The system is designed to be used in open, unconstrained environments like in next generation video conferencing systems that automatically “orchestrate” the transmitted video streams to improve the overall experience of interaction between spatially separated families and friends. Performance levels achieved to date on hand-labelled dataset have shown sufficient reliability at the same time as fulfilling real-time processing requirements.