about
Learned Multi-dimensional Indexes – Result with big impact for key/value stores? (arxiv.org)
17 points by Traudl on Dec 5, 2019 | hide | past | pdf | 1 comment on HN

In plain words: Flood builds an in-memory index that learns from the data and queries to arrange rows and the index together automatically, instead of hand-tuned trees or fixed sort orders. On real data it ran filtered range scans up to 1000 times faster than those.

Abstract · Learning Multi-dimensional Indexes

Scanning and filtering over multi-dimensional tables are key operations in modern analytical database engines. To optimize the performance of these operations, databases often create clustered indexes over a single dimension or multi-dimensional indexes such as R-trees, or use complex sort orders (e.g., Z-ordering). However, these schemes are often hard to tune and their performance is inconsistent across different datasets and queries. In this paper, we introduce Flood, a multi-dimensional in-memory index that automatically adapts itself to a particular dataset and workload by jointly optimizing the index structure and data storage. Flood achieves up to three orders of magnitude faster performance for range scans with predicates than state-of-the-art multi-dimensional indexes or sort orders on real-world datasets and workloads. Our work serves as a building block towards an end-to-end learned database system.

Vikram Nathan, Jialin Ding, Mohammad Alizadeh, Tim Kraska
arXiv:1912.01668 · cs.DB, cs.DS, cs.LG · submitted Dec 3, 2019
abstract · pdf · html

add comment on HN

This could have huge implications for key/value stores and cloud storage systems as it provides a mechanism to access data by more than one key. The numbers to Amazon's z-order encoding also look very interesting. Of course, updates/inserts might still be a problem, but many of the other techniques (e.g, Amazon) are also static.