We present an algorithm that directly factorizes raw RGB-D monocular video using keyframe annotations to produce object-level neural representation with explicit motion trajectories and possibly deformation information. The decoupling subsequently enables different object manipulation and novel view synthesis applications to produce authored videos. We do not make use of any template but instead use an end-to-end optimization to enable factorization. Note that the human deforms/moves in this sequence.

Scene Reconstruction (Training Cameras)

Here we compare our object factorization results. Please use mouse dragging to browse results.
For the comparisons (iMap and Nice-SLAM), we use depth order to generate the composed full-scene results.

Scene Reconstruction (Validation Cameras)

Here we show the reconstruction results using the validation cameras.

Object Reconstruction (Training Cameras)

Here we compare our object factorization results. Please use mouse dragging to browse results.

Object Reconstruction (Validation Cameras)

Here we compare our object factorization results. Please use mouse dragging to browse results.

Quantitative Evaluation on Our Synthetic Dataset

(Top/Bottom) Quantitative color/depth novel view rendering results on validation cameras. Ours largely produces better reconstruction, validating that our joint optimization captures better scene geometry.

Dynamic Scene Reconstruction - Real-world Data

We also test on real RGB-D monocular captures using the camera \(k_0\) in the BeHave dataset.

Scene Reconstruction (Training Cameras)

Here we compare our object factorization results. For the comparisons (iMap and Nice-SLAM), we use depth order to generate the composed full-scene results. Please use mouse dragging to browse results.

Object Reconstruction

Here we compare our object factorization results. Please use mouse dragging to browse results.

Method

Starting from a monocular RGB-D sequence \(\{I(t)\}\), we extract a \(\textbf{factored neural representation}\) \(\mathcal{F}\) that contains separate neural models for the background and each of the moving objects along with their trajectories. For any object tagged as \(\textit{nonrigid}\), we also optimize a corresponding deformation block (e.g., human). First, in an initialization phase, We assume access to keyframe annotation (segmentation and AABBs) over time, propagate them to neighboring frames via dense visual tracking and optical flow, and estimate object trajectories. Then, we perform end-to-end optimization using a customized neural volume rendering block. The factored representation enables a variety of applications involving novel view synthesis and object manipulations.

Given a factored representation \( F(t)\) and any query ray \(r\) from the current camera, we first intersect each objects' bounding box \(B_i\) to obtain a sampling range and then compute an uniform sampling for each of the intervals. For each such sample \(s\), we lookup feature attributes by re-indexing using local coordinate \(T_i^{-1} s \), resort the samples across the different objects based on (sample) depth values, and then volume render to get a rendered attribute. Background is modeled as the 0-th object. For objects with active \(\textit{non-rigid}\) flags, we also invoke the corresponding deformation block. The neural representations and the volume rendering functions are jointly trained.

We provide our network architecture below. For the SDF MLP networks, we use geometric initialization, weighted normalization, Softplus activations, and a skip-connection at the 4-th layer. The input coordinates and view directions are lifted to a high dimensional space using positional encoding. For rigid objects, we use \(\mathbb{SE}3\) representation, \ie a quaternion and a translation vector. For non-rigid objects, we use bijective deformation blocks proposed by Cai et. al. with weighted normalization and Softplus activations as well. For the color MLP network, we use ReLU activation.

Bibtex

@article{,
    title={Factored Neural Representation for Scene Understanding},
    author={Wong, Yu-Shiang and Mitra, Niloy},
    journal={arXiv preprint arXiv:},
    year={2023}
} 

Related Work

• Sucar, E., Liu, S., Ortiz, J., & Davison, A. J. (2021). iMAP: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

• Zhu, Z., Peng, S., Larsson, V., Xu, W., Bao, H., Cui, Z., ... & Pollefeys, M. (2022). Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

• Cai, H., Feng, W., Feng, X., Wang, Y., & Zhang, J. (2022). Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera. arXiv preprint arXiv:2206.15258.