about
Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming (arxiv.org)
1 point by sebg on Aug 16, 2021 | hide | past | pdf | discuss on HN

In plain words: Instead of chaining bulk array operations, arrays are treated as memoized functions over typed index sets, so element-by-element code compiles into fast parallel in-place updates and can still compute gradients automatically. Against a fast low-level array language, it ran with little performance penalty.

Abstract · Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming

We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, "pointful" style with little performance penalty.

Adam Paszke, Daniel Johnson, David Duvenaud, Dimitrios Vytiniotis, Alexey Radul, Matthew Johnson, Jonathan Ragan-Kelley, Dougal Maclaurin
arXiv:2104.05372 · cs.PL · submitted Apr 12, 2021
abstract · pdf · html · 31 pages with appendix, 11 figures. A conference submission is still under review

add comment on HN
Also discussed: Apr 2021 (3 points, 0 comments) · Apr 2021 (1 point, 0 comments)