In plain words: A tool that trains deep networks across a Spark cluster by splitting data and averaging simple weight updates, so it works even when machines talk slowly. Unlike systems needing fast links and tuning, it sped up ImageNet training as more machines were added.
Abstract
Training deep networks is a time-consuming process, with networks for object recognition often requiring multiple days to train. For this reason, leveraging the resources of a cluster to speed up training is an important area of work. However, widely-popular batch-processing computational frameworks like MapReduce and Spark were not designed to support the asynchronous and communication-intensive workloads of existing distributed deep learning systems. We introduce SparkNet, a framework for training deep networks in Spark. Our implementation includes a convenient interface for reading data from Spark RDDs, a Scala interface to the Caffe deep learning framework, and a lightweight multi-dimensional tensor library. Using a simple parallelization scheme for stochastic gradient descent, SparkNet scales well with the cluster size and tolerates very high-latency communication. Furthermore, it is easy to deploy and use with no parameter tuning, and it is compatible with existing Caffe models. We quantify the dependence of the speedup obtained by SparkNet on the number of machines, the communication frequency, and the cluster's communication overhead, and we benchmark our system's performance on the ImageNet dataset.
Philipp Moritz, Robert Nishihara, Ion Stoica, Michael I. Jordan
arXiv:1511.06051 · stat.ML, cs.DC, cs.LG, cs.NE, math.OC · submitted Nov 19, 2015 · updated Feb 28, 2016
abstract · pdf · html · 12 pages, 7 figures