about
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression (arxiv.org)
1 point by wondersky 31 days ago | hide | past | pdf | discuss on HN

In plain words: Compressing a long conversation's memory into fixed-size blocks to save space makes token positions line up badly with block edges, creating weak spots. Retrieval accuracy swung by up to 40 percentage points with a token's position in its block, even when averages looked fine.

Abstract

Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position relative to compression-window boundaries. We uncover a systematic asymmetry in models using such compression: the same information can be easy to retrieve at one phase and difficult at another. We call this periodic variation in retrieval performance phase sensitivity. In large open-weight models with such compression, long-context retrieval accuracy can differ by up to 40 percentage points across phases, revealing periodic weak spots that average benchmark scores can conceal. To investigate this behavior, we pretrain a family of transformers from scratch across multiple KV-compression designs, reproducing phase sensitivity across the variants. Mechanistic analysis using causal interventions in these models reveals phase specialization: different attention components contribute asymmetrically to retrieving information at different source phases. We further analyze idealized retrieval models, showing how gradient flow dynamics may favor sharp phase specialization. Evaluating models with chunked KV-cache compression thus requires measuring across compression phases: high average accuracy can coexist with systematic positional failures.

Xingyu Zhu, Pu, Yi, Ziheng Cheng, Ang Lv, Jing Liu, Lexing Ying, Yiyuan Ma, Xin Dong
arXiv:2609.36322 · cs.LG, cs.AI · submitted Sep 28, 2026
abstract · pdf · 65 pages, 20 figures, pre-print

add comment on HN