Neural Convolutional Surfaces (NCS) can faithfully represent a given ground-truth shape while disentangling coarse geometry from fine details, leading to a highly-accurate representation of the shape. Compared to other state-of-the-art methods NGLOD and ACORN, NCS achieves significantly more accurate results for the same memory footprint.

Abstract

This work is concerned with representation of shapes while disentangling fine, local and possibly repeating geometry, from global, coarse structures. Achieving such disentanglement leads to two unrelated advantages: i) a significant compression in the number of parameters required to represent a given geometry; ii) the ability to manipulate either global geometry, or local details, without harming the other. At the core of our approach lies a novel pipeline and neural architecture, which are optimized to represent one specific atlas, representing one 3D surface. Our pipeline and architecture are designed so that disentanglement of global geometry from local details is accomplished through optimization, in a completely unsupervised manner. We show that this approach achieves better neural shape compression than the state of the art, as well as enabling manipulation and transfer of shape details.

Overview

Surfaces \(S\) are represented by two models, a coarse model \(g_\phi^c\) that encodes a coarse version \(\tilde{S}\) of the surface and allows computing a local reference frame \(\mathbf{F}\), and fine model \(g_\psi^d\) that encodes geometric detail as offsets \(\hat{p}\) from the coarse surface, in coordinates of the local reference frame.

Representation quality

The reconstruction quality of our method, compared with ACORN and NGLOD for two models, using the same number of network parameters on each method model size (100K parameters in this example). Our result exhibits higher accuracy and reconstruction of fine details, while not exhibiting artifacts such as artificial edges or rasterization.

Editing

Our NCS provides a natural decomposition of shapes into coarse shapes and fine details. Boosting or suppressing the fine details and reconstructing the shapes, naturally results in exaggeration or smoothing of surface features.

Detail transfer

Our architecture intrinsically decomposes shapes into coarse base models and associated geometric details, which allows us to transfer learned details from one model to another base shape. In this case, from one pair of pants (a), to another pair of pants. The fitter coarse model for each of the two pairs is shown at the top. Here we compare our detail transfer results with those of a concurrent work IDF. To transfer details we replace the source coarse model with the target coarse one, and reconstruct the shape. Note, this is possible because the global geometry images, source and target, are aligned. In case of misalignment, an inter-surface map between the coarse models could be computed using, e.g., NSM.

Interpretability

Our method yields interpretable convolutional kernels: we select one spike (highlighted) on the dino, identify the CNN features that are strongly active in its region, and then identify other regions where the same features are active. High correlation (hotter colors) implies regions with similar geometric details.

Bibtex

@inproceedings{morreale2022neural,
  title={Neural Convolutional Surfaces},
  author={Morreale, Luca and Aigerman, Noam and Guerrero, Paul and Kim, Vladimir G. and Mitra, Niloy J.},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}