about
Don't Unroll Adjoint: Differentiating SSA-Form Programs (arxiv.org)
1 point by ChrisRackauckas on Oct 20, 2018 | hide | past | pdf | discuss on HN

In plain words: Instead of recording every step a program takes and replaying it backwards to find gradients, this tool rewrites the compiler's internal code form to build the backward pass directly. It handles loops, recursion, and data structures while giving ordinary compilers code that runs fast.

Abstract

This paper presents reverse-mode algorithmic differentiation (AD) based on source code transformation, in particular of the Static Single Assignment (SSA) form used by modern compilers. The approach can support control flow, nesting, mutation, recursion, data structures, higher-order functions, and other language constructs, and the output is given to an existing compiler to produce highly efficient differentiated code. Our implementation is a new AD tool for the Julia language, called Zygote, which presents high-level dynamic semantics while transparently compiling adjoint code under the hood. We discuss the benefits of this approach to both the usability and performance of AD tools.

Michael Innes
arXiv:1810.07951 · cs.PL · submitted Oct 18, 2018 · updated Mar 9, 2019
abstract · pdf · html

add comment on HN
Also discussed: Feb 2019 (87 points, 16 comments) · Oct 2018 (1 point, 0 comments) · Oct 2018 (2 points, 0 comments)