about
Implicit Neural Representations with Periodic Activation Functions (2020) (arxiv.org)
3 points by peter_d_sherman 8 days ago | hide | past | pdf | 1 comment on HN

In plain words: Instead of the usual kinks, this network uses sine waves as its building blocks, so one function can describe an image, sound, or wave and how it changes. It captures fine detail and derivatives that standard networks blur, and solves physics equations like Poisson's.

Abstract · Implicit Neural Representations with Periodic Activation Functions

Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and demonstrate the representation of images, wavefields, video, sound, and their derivatives. Further, we show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations (yielding signed distance functions), the Poisson equation, and the Helmholtz and wave equations. Lastly, we combine Sirens with hypernetworks to learn priors over the space of Siren functions.

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein
arXiv:2006.09661 · cs.CV, cs.LG, eess.IV · submitted Jun 17, 2020
abstract · pdf · html · Project website: https://vsitzmann.github.io/siren/ Project video: https://youtu.be/Q2fLWGBeaiI

add comment on HN

>"We propose SIREN, a simple neural network architecture for implicit neural representations that uses the sine as a periodic activation function: [...]

Interestingly, any derivative of a SIREN is itself a SIREN, as the derivative of the sine is a cosine, i.e., a phase-shifted sine (see supplemental).

Therefore, the derivatives of a SIREN inherit the properties of SIRENs, enabling us to supervise any derivative of SIREN with “complicated” signals. In our experiments, we demonstrate that when a SIREN is supervised using a constraint Cm involving the derivatives of φ, the function φ remains well behaved, which is crucial in solving many problems, including boundary value problems (BVPs).

We will show that SIRENs can be initialized with some control over the distribution of activations, allowing us to create deep architectures.

Furthermore, SIRENs converge significantly faster than baseline architectures, fitting, for instance, a single image in a few hundred iterations, taking a few seconds on a modern GPU, while featuring higher image fidelity..."

SIRENs look interesting... the idea of using Sine as an Activation Function seems like a brilliant one!