about
GAAC: Lossless KV Cache Compression for Faster LLM Inference (arxiv.org)
2 points by wslh 17 days ago | hide | past | pdf | 3 comments on HN

In plain words: Large AI models store past conversation in a KV cache that eats memory; this squeezes it by re-encoding the floating-point exponents, which carry redundant bits, and rebuilding values exactly during attention. It shrinks the cache 1.4 times losslessly and speeds up decoding.

Abstract · GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices

Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. GAAC adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving. Across five tested models, GAAC achieves 1.405-1.407$\times$ lossless BF16 KV compression. We also study compression performance on low-precision tensors FP8 E4M3 caches ( 1.09-1.14$\times$) and E5M2 caches (1.25-1.30$\times$). On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\times$. In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. On Apple M4, GAAC achieves 1.02-1.25$\times$ attention speedups, demonstrating portability beyond NVIDIA GPUs.

Yipin Guo, Arun M George, Siddharth Joshi
arXiv:2610.38210 · cs.AI · submitted Oct 28, 2026
abstract · pdf · html

add comment on HN

I searched around, but couldn't find source code. :(
v2 added the repo link to the abstract: https://github.com/Intelligent-Microsystems-Lab/GAAC
I couldn't find it either but there is something related: https://github.com/lichangshiwhu/Adacode