about
DeepSeek V4's indexer OOMs at 65K context. We got it to 1M in 6G (arxiv.org)
8 points by OsamaJaber 152 days ago | hide | past | pdf | discuss on HN

In plain words: Instead of building a score table to pick which tokens attention reads, it scores tokens in small chunks and keeps only the best ones as it goes. The usual approach runs out of memory at 65,536 tokens; this reaches 1,048,576 with 6.21 GB peak.

Abstract · StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

DeepSeek-V3.2 and V4 introduce Compressed Sparse Attention (CSA): a lightning indexer (a learned scoring projection over compressed keys) scores them, the top-k are selected per query, and a sparse attention kernel reads only those. Public CSA implementations materialize a [B, S, H_I, T] FP32 score tensor before the top-k reduction. With H_I=64 indexer heads and the V4-Flash compression ratio m=4, that intermediate is 256 GB at sequence length S=65,536, exceeding any single-GPU high-bandwidth-memory (HBM) budget. We present StreamIndex, a Triton implementation of the CSA pipeline whose central component is a chunked partition-merge top-k driver that never materializes the full intermediate. On synthetic-but-realistic V4-shaped inputs at the indexer-step (layer) level on a single NVIDIA H200, the materialize path runs out of memory (OOMs) at S=65,536 with V4-Flash dimensions; StreamIndex runs the same indexer to S=1,048,576 with 6.21 GB peak HBM, a 32x regime extension. Set-overlap recall against the materialize ground truth is bit-exact at small S where both fit; across three 5-point design-space sweeps (chunk size, key-tile size, top-k), mean recall rounds to 1.0000 with min recall at least 0.9980 in every cell. The chunked driver composes with TileLang's pipelined attention kernel: at S=262,144 with V4-Flash dimensions, the materialize indexer paired with TileLang attention OOMs while the chunked indexer paired with the same attention runs in 1.97 s at 18.56 GB peak. Our contribution targets the indexer step; we make no claim of a faster attention kernel or of real-checkpoint end-to-end behavior. Code: https://github.com/RightNow-AI/StreamIndex.

Jaber Jaber, Osama Jaber
arXiv:2605.02568 · cs.LG, cs.PF · submitted May 4, 2026
abstract · pdf · html · 11 pages, 3 figures, 7 tables, 2 algorithms, 36 references. Memory-bounded indexer kernel for DeepSeek-V4 CSA via chunked partition-merge top-k. Code: https://github.com/RightNow-AI/StreamIndex

add comment on HN
Also discussed: May 2026 (4 points, 0 comments) · May 2026 (5 points, 0 comments)