about
HyperFields: Towards Zero-Shot Generation of NeRFs from Text (arxiv.org)
2 points by PaulHoule on Nov 4, 2023 | hide | past | pdf | discuss on HN

In plain words: A single network learns to turn a text description straight into a 3D scene model, instead of the usual way of slowly optimizing a fresh model for each scene. It covers over 100 scenes and fine-tunes 5 to 10 times faster.

Abstract

We introduce HyperFields, a method for generating text-conditioned Neural Radiance Fields (NeRFs) with a single forward pass and (optionally) some fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF distillation training, which distills scenes encoded in individual NeRFs into one dynamic hypernetwork. These techniques enable a single network to fit over a hundred unique scenes. We further demonstrate that HyperFields learns a more general map between text and NeRFs, and consequently is capable of predicting novel in-distribution and out-of-distribution scenes -- either zero-shot or with a few finetuning steps. Finetuning HyperFields benefits from accelerated convergence thanks to the learned general map, and is capable of synthesizing novel scenes 5 to 10 times faster than existing neural optimization-based methods. Our ablation experiments show that both the dynamic architecture and NeRF distillation are critical to the expressivity of HyperFields.

Sudarshan Babu, Richard Liu, Avery Zhou, Michael Maire, Greg Shakhnarovich, Rana Hanocka
arXiv:2310.17075 · cs.CV · submitted Oct 26, 2023 · updated Jun 13, 2024
abstract · pdf · html · Accepted to ICML 2024, Project page: https://threedle.github.io/hyperfields/

add comment on HN