ReLU Fields: The Little Non-linearity That Could

ReLU-Fields

In this work, we present the following contributions:

1. we propose a minimal extension to grid-based signal representations, which we refer to as ReLU Fields.
2. we show that this representation is simple, does not require any neural networks, is directly differentiable (and hence easy to optimize), and is fast to optimize and evaluate (i.e. render).
3. we empirically validate our claims by showing applications where ReLU Fields plug in naturally: first, image-based 3D scene reconstruction; and second, implicit modeling of 3D geometries.

Method explanation

Representing a ground-truth function (blue) in a 1D (a) and 2D (b) grid cell using the linear basis (yellow) and a ReLUField (pink). The reference has a c1-discontinuity inside the domain that a linear basis cannot capture. A ReLUField will pick two values \(y_1\) and \(y_2\), such that their interpolation, after clamping will match the sharp c1-discontinuity in the ground-truth (blue) function. Please note that a hard-clip at max 1 is also needed usually for rendering as is done in both the examples shown here.

It's just a little ReLU

We look for a representation of \(n\)-valued signals on an \(m\)-dimensional coordinate domain \(\mathbb{R}^m\). For simplicity, we explain the method for \(m=3\). Our representation is strikingly simple. We consider a regular (\(m=3\))-dimensional (\(r \times r \times r\))-grid \(G\) composed of \(r\) voxels along each side. Each voxel has a certain size defined by its diagonal norm in the (\(m=3\))-dimensional space and holds an \(n\)-dimensional vector at each of its (\(2^{m=3}=8\)) vertices. Importantly, even though they have matching number of dimensions, these values do not have a direct physical interpretation (e.g. color, density, or occupancy), which always have some explicitly-defined range, e.g. \([0,1]\) or \(\left[0,+\infty\right)\). Rather, we store unbounded values on the grid; and thus for technical correctness, we call these grids "feature"-grids instead of signal-grids. The features at grid vertices are then interpolated using \(m=3\)-linear (trilinear) interpolation, and followed by a single non-linearity: the \(ReLU\) function, i.e. \(\mathtt{ReLU}(x) = \max(0, x)\) which maps negative input values to 0 and all other values to themselves. Note that this approach does not have any MLP or other neural-network that interprets the features, instead they are simply clipped before rendering. Intuitively, during optimization, these feature-values at the vertices can go up or down such that the ReLU clipping plane best aligns with the c1-discontinuities within the ground-truth signal. Above figure illustrates this concept.

Experimental evaluation

2D Pixel Fields didactic example

3D Occupancy Fields

Quantitative Scores (volumetric-IoU)

Evaluation results on modeling 3D geometries as occupancy fields. Metric used is Volumetric-IoU. The baseline MLP is our implementation of OccupancyNetworks.

3D occupancy field qualitative results

3D Radiance Fields

Quantitative Scores (PSNR & LPIPS)

Evaluation results on 3D synthetic scenes. Metrics used are PSNR (↑) / LPIPS (↓). The column NeRF-TF^* quotes PSNR values from prior work, and as such we do not have a comparable runtime for this method.

Training timelapse (Figure version)

Qualitative comparison between NeRF-PT, Grid and ReLUField . Grid-based versions converge much faster, and we can see significant sharpness improvements of ReLUField over Grid , for example in the leaves of the plant.

Training timelapse (video-loop version)

Spiral path ReLU-Fields results

360-degree comparison results