about
Optimization of RAG systems using quantization and dimensionality reduction (arxiv.org)
3 points by PaulHoule on May 19, 2025 | hide | past | pdf | discuss on HN

In plain words: They tested cheaper number formats and ways to shorten the vectors that retrieval systems store to find information. Combining half-length vectors with a compact floating-point format made storage eight times smaller with almost no accuracy loss, better than the usual 8-bit integer format.

Abstract · Optimization of embeddings storage for RAG systems using quantization and dimensionality reduction techniques

Retrieval-Augmented Generation enhances language models by retrieving relevant information from external knowledge bases, relying on high-dimensional vector embeddings typically stored in float32 precision. However, storing these embeddings at scale presents significant memory challenges. To address this issue, we systematically investigate on MTEB benchmark two complementary optimization strategies: quantization, evaluating standard formats (float16, int8, binary) and low-bit floating-point types (float8), and dimensionality reduction, assessing methods like PCA, Kernel PCA, UMAP, Random Projections and Autoencoders. Our results show that float8 quantization achieves a 4x storage reduction with minimal performance degradation (<0.3%), significantly outperforming int8 quantization at the same compression level, being simpler to implement. PCA emerges as the most effective dimensionality reduction technique. Crucially, combining moderate PCA (e.g., retaining 50% dimensions) with float8 quantization offers an excellent trade-off, achieving 8x total compression with less performance impact than using int8 alone (which provides only 4x compression). To facilitate practical application, we propose a methodology based on visualizing the performance-storage trade-off space to identify the optimal configuration that maximizes performance within their specific memory constraints.

Naamán Huerga-Pérez, Rubén Álvarez, Rubén Ferrero-Guillén, Alberto Martínez-Gutiérrez, Javier Díez-González
arXiv:2505.00105 · cs.IR, cs.CL, cs.DB · submitted Apr 30, 2025
abstract · pdf · html · 13 pages, 9 figures, 1 table

add comment on HN