Learning monocular 3D reconstruction of articulated categories from motion
Abstract
Monocular 3D reconstruction of articulated object categories is challenging due to the lack of training data and the inherent ill-posedness of the problem. In this work we use video self-supervision, forcing the consistency of consecutive 3D reconstructions by a motion-based cycle loss. This largely improves both optimization-based and learning-based 3D mesh reconstruction. We further introduce an interpretable model of 3D template deformations that controls a 3D surface through the displacement of a small number of local, learnable handles. We formulate this operation as a structured layer relying on mesh-laplacian regularization and show that it can be trained in an end-to-end manner. We finally introduce a per-sample numerical optimisation approach that jointly optimises over mesh displacements and cameras within a video, boosting accuracy both for training and also as test time post-processing. While relying exclusively on a small set of videos collected per category for supervision, we obtain state-of-the-art reconstructions with diverse shapes, viewpoints and textures for multiple articulated object categories.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2021
- DOI:
- 10.48550/arXiv.2103.16352
- arXiv:
- arXiv:2103.16352
- Bibcode:
- 2021arXiv210316352K
- Keywords:
-
- Computer Science - Computer Vision and Pattern Recognition;
- Computer Science - Graphics;
- Computer Science - Machine Learning
- E-Print:
- Accepted to CVPR2021. For project website see https://fkokkinos.github.io/video_3d_reconstruction/