CONF Muckenhirn_INTERSPEECH_2019/IDIAP Understanding and Visualizing Raw Waveform-based CNNs Muckenhirn, Hannah Abrol, Vinayak Magimai-Doss, Mathew Marcel, Sébastien CNN visualization deep learning raw waveforms EXTERNAL http://publications.idiap.ch/attachments/papers/2019/Muckenhirn_INTERSPEECH_2019.pdf PUBLIC http://publications.idiap.ch/index.php/publications/showcite/Muckenhirn_Idiap-RR-11-2018 Related documents Proceedings of Interspeech 2019 Modeling directly raw waveforms through neural networks for speech processing is gaining more and more attention. Despite its varied success, a question that remains is: what kind of information are such neural networks capturing or learning for different tasks from the speech signal? Such an insight is not only interesting for advancing those techniques but also for understanding better speech signal characteristics. This paper takes a step in that direction, where we develop a gradient based approach to estimate the relevance of each speech sample input on the output score. We show that analysis of the resulting ``relevance signal" through conventional speech signal processing techniques can reveal the information modeled by the whole network. We demonstrate the potential of the proposed approach by analyzing raw waveform CNN-based phone recognition and speaker identification systems. REPORT Muckenhirn_Idiap-RR-11-2018/IDIAP Gradient-based spectral visualization of CNNs using raw waveforms Muckenhirn, Hannah Abrol, Vinayak Magimai-Doss, Mathew Marcel, Sébastien EXTERNAL http://publications.idiap.ch/attachments/reports/2018/Muckenhirn_Idiap-RR-11-2018.pdf PUBLIC Idiap-RR-11-2018 2018 Idiap July 2018 Modeling directly raw waveform through neural networks for speech processing is gaining more and more attention. Despite its varied success, a question that remains is: what kind of information are such neural networks capturing or learning for different tasks from the speech signal? Such an insight is not only interesting for advancing those techniques but also for understanding better speech signal characteristics. This paper takes a step in that direction, where we develop a gradient based approach to estimate the relevance of each speech sample input on the output score. We show that analysis of the resulting ``relevance signal" through conventional speech signal processing techniques can reveal the information modeled by the whole network. We demonstrate the potential of the proposed approach by analyzing raw waveform CNN-based phone recognition and speaker identification systems.