about
Differentiable Surface Rendering via Non-Differentiable Sampling (arxiv.org)
2 points by lnyan on Aug 12, 2021 | hide | past | pdf | discuss on HN

In plain words: To fit 3D shapes to images, it grabs surface points with ordinary rasterization, then paints them as depth-aware dots so gradients flow at silhouette edges. It skips differentiable meshing and, for the first time, renders a learned 3D scene as a solid surface, not fog.

Abstract

We present a method for differentiable rendering of 3D surfaces that supports both explicit and implicit representations, provides derivatives at occlusion boundaries, and is fast and simple to implement. The method first samples the surface using non-differentiable rasterization, then applies differentiable, depth-aware point splatting to produce the final image. Our approach requires no differentiable meshing or rasterization steps, making it efficient for large 3D models and applicable to isosurfaces extracted from implicit surface definitions. We demonstrate the effectiveness of our method for implicit-, mesh-, and parametric-surface-based inverse rendering and neural-network training applications. In particular, we show for the first time efficient, differentiable rendering of an isosurface extracted from a neural radiance field (NeRF), and demonstrate surface-based, rather than volume-based, rendering of a NeRF.

Forrester Cole, Kyle Genova, Avneesh Sud, Daniel Vlasic, Zhoutong Zhang
arXiv:2108.04886 · cs.GR, cs.CV · submitted Aug 10, 2021
abstract · pdf · html · Accepted to ICCV 2021

add comment on HN