REPORT VILLATORO-TELLO_Idiap-RR-01-2022/IDIAP Applying Attention Based Models for Detecting Cognitive Processes and Mental Health Conditions Villatoro-Tello, Esaú Parida, Shantipriya Kumar, Sajit Motlicek, Petr BERT deep learning Natural language processing Operant Motive Test Psycholinguistics Supervised Autoencoders EXTERNAL https://publications.idiap.ch/attachments/reports/2021/VILLATORO-TELLO_Idiap-RR-01-2022.pdf PUBLIC https://publications.idiap.ch/index.php/publications/showcite/VILLATORO-TELLO_COGNITIVECOMPUTATION_2021 Related documents Idiap-RR-01-2022 2022 Idiap January 2022 Preprint Accepted in Cognitive Computation Journal, Springer (IF 4.3) Objective: According to the psychological literature, implicit motives allow for the characterization of behavior, subsequent success, and long-term development. Contrary to personality traits, implicit motives are often deemed to be rather stable personality characteristics. Normally, implicit motives are obtained by Operant Motives, unconscious intrinsic desires measured by the Operant Motive Test (OMT). The OMT test requires participants to write freely descriptions associated with a set of provided images and questions. In this work, we explore different recent machine learning techniques and various text representation techniques for facing the problem of the OMT classification task. Methods: We focused on advanced language representations (e.g, BERT, XLM, and DistilBERT) and deep Supervised Autoencoders for solving the OMT task. We performed an exhaustive analysis and compared their performance against fully connected neural networks and traditional support vector classifiers. Results: Our comparative study highlights the importance of BERT which outperforms the traditional machine learning techniques by a relative improvement of 7.9%. In addition, we performed an analysis of how the BERT attention mechanism is being modified. Our findings indicate that the writing style features acquire higher importance at the moment of accurately identifying the different OMT categories. Conclusions: This is the first time that a study to determine the performance of different transformers-based architectures in the OMT task is performed. Similarly, our work propose, for the first time, using Deep Supervised Autoencoders in the OMT classification task. Our experiments demonstrate that transformers-based methods exhibit the best empirical results, obtaining a relative improvement of 7.9% over the competitive baseline suggested as part of the GermEval 2020 challenge. Additionally, we show that features associated with the writing style are more important than content-based words. Some of these findings show strong connections to previously reported behavioral research on the implicit psychometrics theory. ARTICLE VILLATORO-TELLO_COGNITIVECOMPUTATION_2021/IDIAP Applying Attention-Based Models for Detecting Cognitive Processes and Mental Health Conditions Villatoro-Tello, Esaú Parida, Shantipriya Kumar, Sajit Motlicek, Petr BERT deep learning Natural language processing Operant Motive Test Psycholinguistics Supervised Autoencoders https://publications.idiap.ch/index.php/publications/showcite/VILLATORO-TELLO_Idiap-RR-01-2022 Related documents Cognitive Computation 18 2021 https://link.springer.com/article/10.1007/s12559-021-09901-1 URL 10.1007/s12559-021-09901-1 doi According to the psychological literature, implicit motives allow for the characterization of behavior, subsequent success, and long-term development. Contrary to personality traits, implicit motives are often deemed to be rather stable personality characteristics. Normally, implicit motives are obtained by Operant Motives, unconscious intrinsic desires measured by the Operant Motive Test (OMT). The OMT test requires participants to write freely descriptions associated with a set of provided images and questions. In this work, we explore different recent machine learning techniques and various text representation techniques for facing the problem of the OMT classification task. We focused on advanced language representations (e.g, BERT, XLM, and DistilBERT) and deep Supervised Autoencoders for solving the OMT task. We performed an exhaustive analysis and compared their performance against fully connected neural networks and traditional support vector classifiers. Our comparative study highlights the importance of BERT which outperforms the traditional machine learning techniques by a relative improvement of 7.9%. In addition, we performed an analysis of how the BERT attention mechanism is being modified. Our findings indicate that the writing style features acquire higher importance at the moment of accurately identifying the different OMT categories. This is the first time that a study to determine the performance of different transformer-based architectures in the OMT task is performed. Similarly, our work propose, for the first time, using deep supervised autoencoders in the OMT classification task. Our experiments demonstrate that transformer-based methods exhibit the best empirical results, obtaining a relative improvement of 7.9% over the competitive baseline suggested as part of the GermEval 2020 challenge. Additionally, we show that features associated with the writing style are more important than content-based words. Some of these findings show strong connections to previously reported behavioral research on the implicit psychometrics theory.