logo Idiap Research Institute        
 [BibTeX] [Marc21]
TextGraphs 2022 Shared Task on Natural Language Premise Selection
Type of publication: Conference paper
Citation: Valentino_TEXTGRAPHS-16_2022
Publication status: Published
Booktitle: Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing
Year: 2022
URL: https://aclanthology.org/2022....
Abstract: The Shared Task on Natural Language Premise Selection (NLPS) asks participants to retrieve the set of premises that are most likely to be useful for proving a given mathematical statement from a supporting knowledge base. While previous editions of the TextGraphs shared tasks series targeted multi-hop inference for explanation regeneration in the context of science questions (Thayaparan et al., 2021; Jansen and Ustalov, 2020, 2019), NLPS aims to assess the ability of state-of-the-art approaches to operate on a mixture of natural and mathematical language and model complex multi-hop reasoning dependencies between statements. To this end, this edition of the shared task makes use of a large set of approximately 21k mathematical statements extracted from the PS-ProofWiki dataset (Ferreira and Freitas, 2020a). In this summary paper, we present the results of the 1st edition of the NLPS task, providing a description of the evaluation data, and the participating systems. Additionally, we perform a detailed analysis of the results, evaluating various aspects involved in mathematical language processing and multi-hop inference. The best-performing system achieved a MAP of 15.39, improving the performance of a TF-IDF baseline by approximately 3.0 MAP.
Projects Idiap
Authors Valentino, Marco
Mendes, Deborah
Thayaparan, Mokanarangan
Freitas, Andre
Ustalov, Dmitry
Added by: [UNK]
Total mark: 0