about
Liquid Time-Constant Networks (arxiv.org)
1 point by rbanffy on Aug 17, 2024 | hide | past | pdf | discuss on HN

In plain words: Each memory unit is a linear system whose speed of change shifts with the network's state, and a differential-equation solver steps it forward instead of the usual fixed nonlinear update. The design stays stable and beats classical and modern recurrent networks on time-series prediction.

Abstract · Liquid Time-constant Networks

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics and compute their expressive power by the trajectory length measure in latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs. Code and data are available at https://github.com/raminmh/liquid_time_constant_networks

Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu
arXiv:2006.04439 · cs.LG, cs.NE, stat.ML · submitted Jun 8, 2020 · updated Dec 14, 2020
abstract · pdf · html · Accepted to the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)

add comment on HN