about
Neural Spline Fields for Burst Image Fusion and Layer Separation (arxiv.org)
1 point by PaulHoule on Jan 2, 2024 | hide | past | pdf | discuss on HN

In plain words: A burst is fit with a two-layer model—what's behind and what blocks it—made from networks that turn coordinates into curves, fusing them into one image while splitting the layers. Discarding the blocking layer removes occlusions, reflections, and shadows, beating dedicated removal tools.

Abstract

Each photo in an image burst can be considered a sample of a complex 3D scene: the product of parallax, diffuse and specular materials, scene motion, and illuminant variation. While decomposing all of these effects from a stack of misaligned images is a highly ill-conditioned task, the conventional align-and-merge burst pipeline takes the other extreme: blending them into a single image. In this work, we propose a versatile intermediate representation: a two-layer alpha-composited image plus flow model constructed with neural spline fields -- networks trained to map input coordinates to spline control points. Our method is able to, during test-time optimization, jointly fuse a burst image capture into one high-resolution reconstruction and decompose it into transmission and obstruction layers. Then, by discarding the obstruction layer, we can perform a range of tasks including seeing through occlusions, reflection suppression, and shadow removal. Validated on complex synthetic and in-the-wild captures we find that, with no post-processing steps or learned priors, our generalizable model is able to outperform existing dedicated single-image and multi-view obstruction removal approaches.

Ilya Chugunov, David Shustin, Ruyu Yan, Chenyang Lei, Felix Heide
arXiv:2312.14235 · cs.CV · submitted Dec 21, 2023
abstract · pdf · html · project website: https://light.princeton.edu/publication/nsf

add comment on HN