CONF He_ICLR_2026/IDIAP Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction He, Mutian Garner, Philip N. EXTERNAL https://publications.idiap.ch/attachments/papers/2026/He_ICLR_2026.pdf PUBLIC https://publications.idiap.ch/index.php/publications/showcite/He_Idiap-RR-01-2026 Related documents ICLR 2026 Workshop on Memory for LLM-Based Agentic Systems Rio de Janeiro, Brazil 2026 https://openreview.net/forum?id=nyJy6R348F URL Linear-attention models that compress the entire input sequence into a fixed-size recurrent state offer an efficient alternative to Transformers, but their finite memory induces forgetfulness that harms retrieval-intensive tasks. To mitigate the issue, we explore a series of hybrid models that restore direct access to past tokens. We interleave token mixers with intermediate time and space complexity between linear and full attention, including sparse attention with token eviction, and the query-aware native sparse attention. Particularly, we propose a novel learnable token eviction approach. Combined with sliding-window attention, an end-to-end trainable lightweight CNN aggregates information from both past and future adjacent tokens to adaptively retain a limited set of critical KV-pairs per head, maintaining linear attention's constant time and space complexity. Efficient Triton kernels for the sparse attention mechanisms are provided. Empirical evaluations on retrieval-intensive benchmarks support the effectiveness of our approaches. REPORT He_Idiap-RR-01-2026/IDIAP Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction He, Mutian Garner, Philip N. EXTERNAL https://publications.idiap.ch/attachments/reports/2025/He_Idiap-RR-01-2026.pdf PUBLIC Idiap-RR-01-2026 2026 Idiap March 2026 Linear-attention models that compress the entire input sequence into a fixed-size recurrent state offer an efficient alternative to Transformers, but their finite memory induces forgetfulness that harms retrieval-intensive tasks. To mitigate the issue, we explore a series of hybrid models that restore direct access to past tokens. We interleave token mixers with intermediate time and space complexity between linear and full attention, including sparse attention with token eviction, and the query-aware native sparse attention. Particularly, we propose a novel learnable token eviction approach. Combined with sliding-window attention, an end-to-end trainable lightweight CNN aggregates information from both past and future adjacent tokens to adaptively retain a limited set of critical KV-pairs per head, maintaining linear attention's constant time and space complexity. Efficient Triton kernels for the sparse attention mechanisms are provided. Empirical evaluations on retrieval-intensive benchmarks support the effectiveness of our approaches. https://arxiv.org/abs/2510.20787 URL