CONF Kumar_IEEEASRU2025_2025/IDIAP TokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activation Kumar, Shashi Madikeri, Srikanth Villatoro-Tello, Esaú Burdisso, Sergio Rangappa, Pradeep Carofilis, Andrés Motlicek, Petr S, Karthik Pandia D Venkatesan, Shankar Hacioğlu, Kadri Stolcke, Andreas language identification multitask training named entity recognition Speaker change detection speech recognition XLSR-Transducer EXTERNAL http://publications.idiap.ch/attachments/papers/2025/Kumar_IEEEASRU2025_2025.pdf PUBLIC 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 2025 IEEE Token-based multitasking frameworks like TokenVerse require all training utterances to have labels for all tasks, hindering their ability to leverage partially annotated datasets and scale effectively. We propose TokenVerse++, which introduces learnable vectors in the acoustic embedding space of the XLSR-Transducer ASR model for dynamic task activation. This core mechanism enables training with utterances labeled for only a subset of tasks, a key advantage over TokenVerse. We demonstrate this by successfully integrating a dataset with partial labels, specifically for ASR and an additional task, language identification, improving overall performance. TokenVerse++ achieves results on par with or exceeding TokenVerse across multiple tasks, establishing it as a more practical multitask alternative without sacrificing ASR performance.