In plain words: The model folds a running, word-by-word importance signal into attention's scoring step, so each token can both look anywhere and know what matters, all in one standard attention call. It beat a plain Transformer on word prediction and hit perfect retrieval seven times sooner.
Abstract
Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling. Transformers see everywhere but cannot prioritize; SSMs know what matters but cannot revisit. Existing hybrids -- Jamba (block level) and Hymba (head level) -- place the two in separate compartments, so neither informs the other during the attention computation itself. We propose SISA (SSM-Informed Softmax Attention), which adds an SSM-derived importance term directly inside the attention score and realizes the full operation as a single SDPA call on augmented query/key vectors -- no recurrent state, no custom kernel. At 152M / 5B tokens, SISA reaches LAMBADA-greedy 17.3% (vs. Transformer 13.9 and Mamba-3 15.5) and attains NIAH 100% from step 1K, 7x faster than Transformer's retrieval convergence; at 369M, Mamba-3 leads LAMBADA while SISA preserves perfect NIAH and stock-SDPA execution. SISA thus defines a third design axis for SSM-attention hybrids -- score-level fusion -- beyond the block-level and head-level paradigms that have dominated the field.
Suhyeong Shin, Yeongwook Yang
arXiv:2606.02332 · cs.AI, cs.CL, cs.LG · submitted Jun 1, 2026 · updated Jun 2, 2026
abstract · pdf · html · 20 pages, 6 figures, 25 tables