ML Newsnew | past | best | rss | submit about
Parameter-free KV cache compression for memory-efficient long-context LLMs (arxiv.org)
65 points by PaulHoule on Mar 27, 2025 | hide | past | pdf | 19 comments on HN

In plain words: It shrinks the memory storing past words by merging less important ones while keeping a small correction so key context survives, needing no retraining. On a 7-billion-parameter model it cut that memory to 5% of normal with similar quality, and tripled speed at 54,000 tokens.

Abstract · ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs

The linear growth of key-value (KV) cache memory and quadratic computational in attention mechanisms complexity pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often incur irreversible information loss or require costly parameter retraining. To this end, we propose ZSMerge, a dynamic KV cache compression framework designed for efficient cache management, featuring three key operations: (1) fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) a residual merging mechanism that preserves critical context through compensated attention scoring, and (3) a zero-shot adaptation mechanism compatible with diverse LLM architectures without requiring retraining. ZSMerge significantly enhances memory efficiency and inference speed with negligible performance degradation across LLMs. When applied to LLaMA2-7B, it demonstrates a 20:1 compression ratio for key-value cache retention (reducing memory footprint to 5\% of baseline) while sustaining comparable generation quality, coupled with triple throughput gains at extreme 54k-token contexts that eliminate out-of-memory failures. The code is available at https://github.com/SusCom-Lab/ZSMerge.

Xin Liu, Xudong Wang, Pei Liu, Guoming Tang
arXiv:2503.10714 · cs.CL, cs.AI · submitted Mar 13, 2025 · updated Oct 5, 2025
abstract · pdf · html

add comment on HN

kristianp on Mar 27, 2025 [–]
Code at https://github.com/SusCom-Lab/ZeroMerge
reply
hinkley on Mar 27, 2025 [–]
This feels like something that could be done in part by hand. We store documents in KV that are often built deterministically by merging two or three pieces of data, one of which is a form of string interpolation (eg, templates).

Effectively if you had a microservice that did extremely light data processing, and you moved the KV store behind it instead of in front of it, you’d achieve a similar aim. A small cache in front of it or even at the upstream would reduce the calculations in the face of thundering herds.

reply
vlovich123 on Mar 27, 2025 | parent [–]
this sounds like you’re thinking KV as in key-value like redis or s3. This paper is about KV cache as in LLM which is for reducing the computational complexity for self-attention. Has nothing to do with what you wrote unless I misunderstood what your wrote (I’m confused what upstream would mean here - the contents of the KV cache are specific to the context provided to the LLM / what the LLM is generating in response).
reply
az226 on Mar 27, 2025 [–]
Is this some joke? They use Llama 2 7B? What year is it?
reply
PaulHoule on Mar 27, 2025 | parent [–]
The best model is the one you can fit in memory.

About as soon as GPT-4 came out I said that OpenAI was doomed on the trajectory it was on because they could not afford to develop a GPT-5, GPT-6, etc.

Real innovation comes out of doing a lot of experiments and that means doing experiments quickly with the resources you have. So you do most of your experiments with non-frontier models, enough to make a good prediction of what would happen if you maxxed out your model size, then you go big. That's how you make everyone else have a "DeepSeek moment".

A company like Apple wants to pick something on the frontier and keep advancing on a straight line. Works great if you want to make an M1, M2, M3, ... ARM chip but that's not how progress works in AI today.

reply
hinkley on Mar 27, 2025 | parent [–]
Will we see models built on b-trees to deal with memory requirements? Have we already?
reply
sujayakar on Mar 27, 2025 | parent [–]
Deepseek is already using SSDs for their KV cache: https://github.com/deepseek-ai/3FS
reply
vlovich123 on Mar 27, 2025 | parent [–]
You are deeply misunderstanding what the KV cache referred to here is. It’s not for storing data. This is the KV cache that’s part of the model to reduce quadratic compute complexity into linear for self attention. This is not stored on SSD - it’s in VRAM (or CPU if you’re not using a GPU)
reply
boroboro4 on Mar 27, 2025 | parent [–]
They, in fact, mention inference kv cache as use case in readme. The most advanced kv caching uses hierarchy of gpu ram/regular ram/ssd. Seems like they were able to use their storage abstraction for last tier.
reply
magicalhippo on Mar 28, 2025 | parent [–]
https://github.com/deepseek-ai/3FS?tab=readme-ov-file#3-kvca...

KVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers. The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s

reply
vlovich123 on Mar 28, 2025 | parent [–]
That's because DeepSeek uses MLA which apparently does allow offloading the KV cache. That doesn't apply to all models, particularly the open-weight models that are primarily GQA AFAIK.
reply
boroboro4 on Mar 28, 2025 | parent [–]
Any models allow offloading KV cache, it’s not a matter of model architecture but only of the implementation. The only somewhat difference can be for non transformer models. But for all attention models it’s the same – blob of data per each token. It’s much worse for older models with MHA because their KV cache is just too big, and it’s better for DeepSeek because their KV cache is the smallest. But it’s alright for current generation of GQA models as well.
reply
vlovich123 on Mar 28, 2025 | parent [–]
Are you sure about that? GQA applies self-attention to every KV cache entry. If you're offloading, then you're having to dynamically page in all the KV cache entries into the GPU which is quite slow since the CPU/GPU link only has so much bandwidth. My understanding is that MLA reduces the size of the KV cache & doesn't necessarily attend to every KV token at every step which is why offloading to disk works (i.e. most of the tokens can remain on disk without ever being loaded into the GPU).
reply
boroboro4 on Mar 28, 2025 | parent [–]
Offloading in this case doesn’t mean keeping the kv cache on the disk/in storage all the time, it means keeping it there when request isn’t in process of generation. While request being generated kv cache is indeed in vram.

As for MLA - Deepseek is, just like others, attend to all historical tokens. The only difference instead of having actual KV entries it has lower dimension KV entries, which are being projected into full blown KV entries on the fly during attention. It’s similar to GQA, just instead of just duplication KV entries by size of groups it applies linear transformation.

reply
vlovich123 on Mar 28, 2025 | parent [–]
Ah OK. So this is for resuming chat context cheaply. What I said is still correct - 3FS is not part of the inference flow & not relevant to the paper which is about optimizing the KV cache usage at runtime.
reply
monocasa on Mar 27, 2025 | parent [–]
I mean, there's other, better 7B models than Lllama 2 at this point.
reply
x1000 on Mar 27, 2025 | parent [–]
If they had experimented using a newer model (gemma 3, deepseek-1 7b, etc.) and reported better results, would that be because their newer baseline model was better than the llama 2 model used in the previous methods' experiments? A more comprehensive study would include results for as many baseline models as possible. But there are likely other researchers in the lab all waiting to use those expensive GPUs for their experiments as well.
reply
josephg on Mar 27, 2025 | parent [–]
Sure. But papers take a really long time to write and go through peer review. I think my paper on collaborative editing took about 4 months from the point where we were done writing to the point at which it appeared on arxiv.

This research was almost certainly done well before Gemma 3 and Deepseek were released.

reply
krasin on Mar 27, 2025 | parent [–]
> Is this some joke? They use Llama 2 7B? What year is it?

They use llama2 to demonstrate that their compression method works. There are potential cases:

1. The method works on all / most LLMs. In this case, it does not matter on which model they demonstrated the effect.

2. The method only works on llama2, but not on other models. Given that they published the code, I expect that people will quickly test the method on many other models, so we will know that soon. And yet - there would be a scientific significance even if it works only on llama2, as it would mean that there's some special and good in that architecture.

But I would bet it's #1 - the method works on most of the models and they just picked whatever they had already had code bindings to, to save the effort.

reply
About | RSS | RSS (all) | HN arXiv