CONF Le_INTERSPEECH2009_2009/IDIAP Automatic vs. human question answering over multimedia meeting recordings Le, Quoc Anh Popescu-Belis, Andrei evaluation meeting browsers meeting recording passage retrieval question answering EXTERNAL http://publications.idiap.ch/attachments/papers/2009/Le_INTERSPEECH2009_2009.pdf PUBLIC http://publications.idiap.ch/index.php/publications/showcite/Le_Idiap-RR-13-2009 Related documents 10th Annual Conference of the International Speech Communication Association 2009 Information access in meeting recordings can be assisted by meeting browsers, or can be fully automated following a question-answering (QA) approach. An information access task is defined, aiming at discriminating true vs. false parallel statements about facts in meetings. An automatic QA algorithm is applied to this task, using passage retrieval over a meeting transcript. The algorithm scores 59% accuracy for passage retrieval, while random guessing is below 1%, but only scores 60% on combined retrieval and question discrimination, for which humans reach 70%-80% and the baseline is 50%. The algorithm clearly outperforms humans for speed, at less than 1 second per question, vs. 1.5-2 minutes per question for humans. The degradation on ASR compared to manual transcripts still yields lower but acceptable scores, especially for passage identification. Automatic QA thus appears to be a promising enhancement to meeting browsers used by humans, as an assistant for relevant passage identification. REPORT Le_Idiap-RR-13-2009/IDIAP Automatic vs. human question answering over multimedia meeting recordings Le, Quoc Anh Popescu-Belis, Andrei EXTERNAL http://publications.idiap.ch/attachments/reports/2009/Le_Idiap-RR-13-2009.pdf PUBLIC Idiap-RR-13-2009 2009 Idiap June 2009 Information access in meeting recordings can be assisted by meeting browsers, or can be fully automated following a question-answering (QA) approach. An information access task is defined, aiming at discriminating true vs. false parallel statements about facts in meetings. An automatic QA algorithm is applied to this task, using passage retrieval over a meeting transcript. The algorithm scores 59% accuracy for passage retrieval, while random guessing is below 1%, but only scores 60% on combined retrieval and question discrimination, for which humans reach 70%-80% and the baseline is 50%. The algorithm clearly outperforms humans for speed, at less than 1 second per question, vs. 1.5-2 minutes per question for humans. The degradation on ASR compared to manual transcripts still yields lower but acceptable scores, especially for passage identification. Automatic QA thus appears to be a promising enhancement to meeting browsers used by humans, as an assistant for relevant passage identification.