In plain words: Instead of recording every step a program takes and replaying it backwards to find gradients, this tool rewrites the compiler's internal code form to build the backward pass directly. It handles loops, recursion, and data structures while giving ordinary compilers code that runs fast.
Abstract
This paper presents reverse-mode algorithmic differentiation (AD) based on source code transformation, in particular of the Static Single Assignment (SSA) form used by modern compilers. The approach can support control flow, nesting, mutation, recursion, data structures, higher-order functions, and other language constructs, and the output is given to an existing compiler to produce highly efficient differentiated code. Our implementation is a new AD tool for the Julia language, called Zygote, which presents high-level dynamic semantics while transparently compiling adjoint code under the hood. We discuss the benefits of this approach to both the usability and performance of AD tools.
Michael Innes
arXiv:1810.07951 · cs.PL · submitted Oct 18, 2018 · updated Mar 9, 2019
abstract · pdf · html