Volumetric Rendering with Baked Quadrature Fields

(ECCV 2024)

Gopal Sharma1, Daniel Rebain1, Kwang Moo Yi1, Andrea Tagliasacchi2,3,4
1University of British Columbia, 2Google DeepMind 3University of Toronto 4Simon Fraser University
Paper arXiv Code

TL;DR: Our approach generates a mesh using a quadrature field. We associate texture with the mesh and render the mesh using ray tracing, which can produce volumetric effects such as fur.

Abstract

We propose a novel Neural Radiance Field (NeRF) representation for non-opaque scenes that enables fast inference by utilizing textured polygons. Despite the high-quality novel view rendering that NeRF provides, a critical limitation is that it relies on volume rendering that can be computationally expensive and does not utilize the advancements in modern graphics hardware. Many existing methods fall short when it comes to modelling volumetric effects as they rely purely on surface rendering. We thus propose to model the scene with polygons, which can then be used to obtain the quadrature points required to model volumetric effects, and also their opacity and colour from the texture. To obtain such polygonal mesh, we train a specialized field whose zero-crossings would correspond to the quadrature points when volume rendering, and perform marching cubes on this field. We then perform ray-tracing and utilize the ray-tracing shader to obtain the final colour image. Our method allows an easy integration with existing graphics frameworks allowing rendering speed of over 100 frames-per-second for a 1920×1080 image, while still being able to represent non-opaque objects.

Approach

teaser
In short, We start with a pre-trained network to train a quadrature field that learns the placement of quadrature points. The extracted mesh from the quadrature field is fine-tuned using a deformation field (deformation is shown using red colour on the deformed mesh). Lastly, the neural features are baked into a texture map and the mesh, which can be rendered with WebGL.

The core of our approach consists of learning a quadrature field \(\sin (\omega \mathcal{F}(x)\), the zero-crossings of which gives us quadrature points deterministically. We train this field such that the field gives more quadrature points in regions where volumetric weights are higher. The relative number of zero crossings can be controlled by the frequency term omega as shown above.

Results

Visualization of reconstructed nerf-synthetic dataset.

Visualization of reconstructed nerf-synthetic dataset.

Our approach can represent transparency using continuous opacity.

Original Image

Our approach can represent hair like material better than the baselines.

Original Image

Our approach can represent transparency better than surface based baking approaches like MobileNeRF.

BibTeX

@article{quadfields,
  author    = {Gopal Sharma, Daniel Rebain, Andrea Tagliasacchi, Kwang Moo Yi},
  title     = {Volumetric Rendering with Baked Quadrature Fields},
  journal   = {ECCV},
  year      = {2024},
}