CONF Labrak_SLT2026_2026/IDIAP How to Leverage Synthetic Speech for LLM-Based ASR Systems? Labrak, Yanis Sanchez-Cortes, Dairazalia Burdisso, Sergio Baroudi, Severin Kumar, Shashi Villatoro-Tello, Esaú Madikeri, Srikanth E, Manjunath K Plchot, Oldřich Hacioğlu, Kadri Motlicek, Petr Stolcke, Andreas EXTERNAL https://publications.idiap.ch/attachments/papers/2026/Labrak_SLT2026_2026.pdf PUBLIC IEEE Spoken Language Technology 2026 2026 https://arxiv.org/abs/2606.29031 URL In regulated domains such as banking and healthcare, where privacy makes real speech costly to collect, synthetic speech from text-to-speech is an appealing alternative for training automatic speech recognition (ASR). Yet a distributional gap limits how far synthetic data can replace real recordings. While prior work treats this gap as a black box to engineer around, we probe its origin in a SLAM-ASR architecture. We localise the signal separating real from synthetic speech to early-to-middle layers of the LLM backbone, where temporal and prosodic perturbations disrupt it most. Representation-level separability helps but does not directly predict downstream ASR gains. Convolving synthetic audio with room impulse responses (RIR) narrows the gap not by making speech more natural, but by reproducing the acoustic irregularities of real recordings. Combining a layer-selection module with RIR augmentation matches an all-real baseline using only 25% of real speech (13.6h) and surpasses it at all higher proportions.