about
Baking Neural Radiance Fields for Real-Time View Synthesis (arxiv.org)
3 points by Hard_Space on Apr 4, 2021 | hide | past | pdf | discuss on HN

In plain words: A trained NeRF runs a neural network hundreds of times per pixel, too slow for live use. Baking it into a sparse 3D grid of stored values lets a laptop GPU draw views at over 30 frames per second, keeping detail and shiny surfaces.

Abstract

Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rendering photorealistic images of the scene from unobserved viewpoints. However, NeRF's computational requirements are prohibitive for real-time applications: rendering views from a trained NeRF requires querying a multilayer perceptron (MLP) hundreds of times per ray. We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) that enables real-time rendering on commodity hardware. To achieve this, we introduce 1) a reformulation of NeRF's architecture, and 2) a sparse voxel grid representation with learned feature vectors. The resulting scene representation retains NeRF's ability to render fine geometric details and view-dependent appearance, is compact (averaging less than 90 MB per scene), and can be rendered in real-time (higher than 30 frames per second on a laptop GPU). Actual screen captures are shown in our video.

Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, Paul Debevec
arXiv:2103.14645 · cs.CV, cs.GR · submitted Mar 26, 2021
abstract · pdf · html · Project page: https://nerf.live

add comment on HN