NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections


Dor Verbin1, Pratul Srinivasan1, Peter Hedman1, Ben Mildenhall1, Ben Attal2, Richard Szeliski1, Jon Barron1
1Google Research 2Carnegie Mellon University

SIGGRAPH Asia 2024
Paper Code and data [coming soon]

Abstract

Neural Radiance Fields (NeRFs) typically struggle to reconstruct and render highly specular objects, whose appearance varies quickly with changes in viewpoint. Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. Moreover, these techniques rely on large computationally-expensive neural networks to model outgoing radiance, which severely limits optimization and rendering speed.

We address these issues with an approach based on ray tracing: instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts reflection rays from these points and traces them through the NeRF representation to render feature vectors which are decoded into color using a small inexpensive network. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing shiny objects, and that it is the only existing NeRF method that can synthesize photorealistic specular appearance and reflections in real-world scenes, while requiring comparable optimization time to current state-of-the-art view synthesis models.

Results and comparisons

Here we display side-by-side videos comparing our method to top-performing baselines across different captured scenes.

Select a scene and a baseline method below:


Interactive visualization. Hover or tap to move the split.
Notice how our method synthesizes accurate reflections of the houses and plants that smoothly move over the car's surface, while baseline methods produce fuzzy reflections that fade in and out depending on the viewpoint.
Our method renders convincing reflections of the distant scene beyond the windows as well as the nearby paintings and bowl.
Our method is even able to render convincing reflections of the tree and lamppost, which are never directly viewed by the observed images.
Our method synthesizes reflections with more high-frequency details, and is able to render convincing reflections of near-field content. Notice the accurate interreflections of the shiny spheres and statue head.
Although semi-transparent but reflective surfaces (such as windows) can be challenging for our method, it is able to simulate convincing reflections across the body and windshield of the car.
Notice how our method renders specularities that are consistent across views and smoothly move over the black reflective surface instead of fading in and out depending on the viewpoint.
In this relatively diffuse scene without significant view-dependant appearance, our method's rendered views are on par with Zip-NeRF's, and noticeably better than other baselines.
In this relatively diffuse scene without significant view-dependant appearance, our method's rendered views are on par with Zip-NeRF's, and noticeably better than other baselines.

Ablation study

Here we display side-by-side videos comparing our full method to versions of our method where key components have been ablated. See more details in the paper.

Select an ablation below:


Interactive visualization. Hover or tap to move the split.
Using only a single reflection ray, using Zip-NeRF's 3D Jacobian instead of our 2D directional Jacobian, or omitting downweighting all cause aliasing in the rendered reflections. This prevents optimization from effectively reconstructing accurate scene geometry and reflected content, resulting in blurred reflections with jagged aliasing artifacts. Omitting our method's tracing of nearby scene content significantly degrades the renderings of near-field reflections, such as the reflections of the statue and ground tiles on the reflective spheres.

Synthetic results

While we focus on large-scale real world scenes, our method still performs well on simpler synthetic datasets.