about
3D scene reconstruction in adverse weather conditions via Gaussian splatting (arxiv.org)
53 points by PaulHoule on Jan 28, 2025 | hide | past | pdf | 14 comments on HN

In plain words: It rebuilds a 3D scene from rainy or snowy photos by first wiping out airborne flakes and drops, then masking spots where water sits on the lens. Normal rebuilds bake weather into the scene; this one recovers clear views, beating the best current methods.

Abstract · WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting

3D Gaussian Splatting (3DGS) has gained significant attention for 3D scene reconstruction, but still suffers from complex outdoor environments, especially under adverse weather. This is because 3DGS treats the artifacts caused by adverse weather as part of the scene and will directly reconstruct them, largely reducing the clarity of the reconstructed scene. To address this challenge, we propose WeatherGS, a 3DGS-based framework for reconstructing clear scenes from multi-view images under different weather conditions. Specifically, we explicitly categorize the multi-weather artifacts into the dense particles and lens occlusions that have very different characters, in which the former are caused by snowflakes and raindrops in the air, and the latter are raised by the precipitation on the camera lens. In light of this, we propose a dense-to-sparse preprocess strategy, which sequentially removes the dense particles by an Atmospheric Effect Filter (AEF) and then extracts the relatively sparse occlusion masks with a Lens Effect Detector (LED). Finally, we train a set of 3D Gaussians by the processed images and generated masks for excluding occluded areas, and accurately recover the underlying clear scene by Gaussian splatting. We conduct a diverse and challenging benchmark to facilitate the evaluation of 3D reconstruction under complex weather scenarios. Extensive experiments on this benchmark demonstrate that our WeatherGS consistently produces high-quality, clean scenes across various weather scenarios, outperforming existing state-of-the-art methods. See project page:https://jumponthemoon.github.io/weather-gs.

Chenghao Qian, Yuhu Guo, Wenjing Li, Gustav Markkula
arXiv:2412.18862 · cs.CV, cs.AI · submitted Dec 25, 2024 · updated Feb 12, 2025
abstract · pdf · html

add comment on HN

Is it possible to see microwave ovens on the ground with a Rydberg antenna array for in-cockpit drone Remote ID signal location?
Recently I heard about an approach where they train a model to behave as a Lidar, using only camera inputs. Sounds like a cool, modular approach, but not sure about safety.
You can train those models all you want, the sensors still have some physical limitations. A regular camera operating in the visible spectrum won't be able to penetrate dense fog, where lidar or radar may still work.
True, but it's the same for a human driver.
Sure, but we should aim for superhuman driving capabilities, especially in conditions humans struggle.
Maybe, but the car can also simply refuse to drive if the conditions are bad.
Of course it can refuse (and should do so if it's unsafe), but that's not the goal, is it?
Driving in conditions unsafe for humans is not a current goal of anyone I'm aware of.
Of course, but Lidar is more expensive.
Meh, i believe Tesla will be wrong on this one. Yes, they will achieve good autonomous cars without lidar and radar, but someone else will build a slightly safer autonomous driving experience with those added modalities. The costs will plummet as the sensors become ubiquitous and eventually they will be required by legislation.
Yup, I strongly agree with you too. Infact In fact I mentioned that in my interview for the Tesla Autopilot team lol I guess that’s why they didn’t move me forward in the process.
haha, bold move.
EV grade LiDAR is now under $1000 a pop. In retail. For car manufacturers it must be under half of that.
Tesla FSD used to do this with occupancy networks [0]. It's basically a "software lidar". Afaik they don't use it any more in their newest FSD stack, expect for visualizations maybe.

[0] https://www.youtube.com/watch?v=jPCV4GKX9Dw&t=440s