In plain words: Instead of running a tiny network once for every point along each camera ray, this system feeds groups of points through one network at once, making rendering much faster. A training trick that needs no pre-trained model keeps the colors and densities accurate.
Abstract · MIMO-NeRF: Fast Neural Rendering with Multi-input Multi-output Neural Radiance Fields
Neural radiance fields (NeRFs) have shown impressive results for novel view synthesis. However, they depend on the repetitive use of a single-input single-output multilayer perceptron (SISO MLP) that maps 3D coordinates and view direction to the color and volume density in a sample-wise manner, which slows the rendering. We propose a multi-input multi-output NeRF (MIMO-NeRF) that reduces the number of MLPs running by replacing the SISO MLP with a MIMO MLP and conducting mappings in a group-wise manner. One notable challenge with this approach is that the color and volume density of each point can differ according to a choice of input coordinates in a group, which can lead to some notable ambiguity. We also propose a self-supervised learning method that regularizes the MIMO MLP with multiple fast reformulated MLPs to alleviate this ambiguity without using pretrained models. The results of a comprehensive experimental evaluation including comparative and ablation studies are presented to show that MIMO-NeRF obtains a good trade-off between speed and quality with a reasonable training time. We then demonstrate that MIMO-NeRF is compatible with and complementary to previous advancements in NeRFs by applying it to two representative fast NeRFs, i.e., a NeRF with sample reduction (DONeRF) and a NeRF with alternative representations (TensoRF).
Takuhiro Kaneko
arXiv:2310.01821 · cs.CV, cs.AI, cs.GR, cs.LG, eess.IV · submitted Oct 3, 2023
abstract · pdf · html · Accepted to ICCV 2023. Project page: https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/mimo-nerf/