about
Provably Faster Gradient Descent via Long Steps (arxiv.org)
6 points by mihaitodor on Jul 15, 2023 | hide | past | pdf | 2 comments on HN

In plain words: Instead of checking gradient descent one step at a time, the analysis looks at many steps together, allowing occasional oversized steps that briefly raise the value being minimized. These long steps are proven to reach the minimum faster than the usual steady step size.

Abstract

This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster $O(1/T\log T)$ rate for gradient descent is also motivated along with simple numerical validation.

Benjamin Grimmer
arXiv:2307.06324 · math.OC, cs.LG, math.NA · submitted Jul 12, 2023 · updated Feb 5, 2024
abstract · pdf · html · 20 pages

add comment on HN