ML Newsnew | past | best | rss | submit about
Optimization Methods for Large-Scale Machine Learning (arxiv.org)
38 points by Bootvis on Jun 21, 2016 | hide | past | pdf | 7 comments on HN

In plain words: It surveys how training huge machine-learning models turns into optimization problems, using text classification and deep networks as examples. It finds simple stochastic gradient methods—small noisy steps—succeed where classic gradient techniques fail, and points to less noisy steps and slope-change estimates as next improvements.

Abstract

This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.

Léon Bottou, Frank E. Curtis, Jorge Nocedal
arXiv:1606.04838 · stat.ML, cs.LG, math.OC · submitted Jun 15, 2016 · updated Feb 8, 2018
abstract · pdf · html

add comment on HN
Also discussed: Jun 2016 (3 points, 0 comments)
arcanus on Jun 21, 2016 [–]
Nocedal's text, 'Numerical Optimization' is the standard for that field.

As he notes, I've always been surprised that more techniques in ML do not leverage the Hessian to get quadratic convergence rates.

Nevertheless, the most interesting tidbit of this text, speaking as a Computational Scientist, was,

'Much more could be said about this rapidly evolving field. Perhaps most importantly, we have neither discussed nor analyzed at length the opportunities offered by parallel and distributed computing'

The scalability of these algorithms, in particular across distributed memory systems (e.g. MPI) at extreme scale will be an extremely important question. I'm very interested in attempting to scale these networks to tens or hundreds of thousands of processing cores. With heroic scale systems now often eclipsing millions of cores, there is quite a bit of room to scale up, if the algorithms are indeed robust.

reply
x0x0 on Jun 21, 2016 | parent [–]
The Hessian is too expensive and too big.

lbfgs is quite common for eg regression w/o l1 penalties.

MPI is not great at even high hundreds of cores; it's too much work to build redundancy / retry / restart / clean failure in. You really need a framework that helps with this.

reply
Xcelerate on Jun 21, 2016 | parent [–]
> MPI is not great at even high hundreds of cores

I've used MPI on Titan for thousands of cores. It's essentially what MPI was invented for. I also know people who perform QMC simulations using all of the cores on the machine at once using software built upon MPI.

reply
arcanus on Jun 21, 2016 | parent [–]
> The Hessian is too expensive and too big.

Not necessarily, often derivatives are analytically known in ML.

> MPI is not great at even high hundreds of cores

? You realize that Sequia, which I have run on, has codes that scale to all two million processors.

reply
oliwaw on Jun 21, 2016 | parent [–]
>Not necessarily, often derivatives are analytically known in ML.

The focus here is largely on deep neural networks. In this domain, the Hessian cannot be computed and SGD (with minor variants) continues to be the golden standard.

reply
x0x0 on Jun 21, 2016 | parent [–]
Sure, you can build highly reliable supercomputers. But the majority of us are running on google style networks of mostly-reliable boxes. I worked for a company that built distributed ML software, and even at eg hundreds of boxes you will regularly see failures. Customers will be very unhappy unless your code tolerates that.
reply
jey on Jun 21, 2016 | parent [–]
Stochastic gradient actually performs better in terms of generalization performance! https://arxiv.org/abs/1509.01240

The intuition for it is that when optimizing a machine learning objective all the way to machine precision, you at some point cross an (unknown) threshold where you are over-optimizing the parameters to the particular model class you're using, but that's probably too much faith in your model specification. So stochastic optimization and early stopping (before gradient is zero) provides a form of regularization.

reply
About | RSS | RSS (all) | HN arXiv