about
M3: Scaling Up Machine Learning via Memory Mapping (arxiv.org)
2 points by yzmtf2008 on Apr 12, 2016 | hide | past | pdf | discuss on HN

In plain words: Instead of spreading data across computers, one machine uses memory mapping, where the system swaps data in and out of RAM, so algorithms like logistic regression and k-means can handle up to 190GB. It beat a four-computer Spark cluster and matched an eight-computer one.

Abstract

To process data that do not fit in RAM, conventional wisdom would suggest using distributed approaches. However, recent research has demonstrated virtual memory's strong potential in scaling up graph mining algorithms on a single machine. We propose to use a similar approach for general machine learning. We contribute: (1) our latest finding that memory mapping is also a feasible technique for scaling up general machine learning algorithms like logistic regression and k-means, when data fits in or exceeds RAM (we tested datasets up to 190GB); (2) an approach, called M3, that enables existing machine learning algorithms to work with out-of-core datasets through memory mapping, achieving a speed that is significantly faster than a 4-instance Spark cluster, and comparable to an 8-instance cluster.

Dezhi Fang, Duen Horng Chau
arXiv:1604.03034 · cs.LG, cs.DC · submitted Apr 11, 2016
abstract · pdf · html · 2 pages, 1 figure, 1 table

add comment on HN