Neural Re-Simulation for Generating Bounces in Single Images

1University College London         2Adobe Research        

ICCV 2019

We take as input a single still image depicting a scene and output a video depicting a virtual object dynamically interacting with the scene through bouncing. Here, we consider a ball as our virtual object. We achieve this by our Dynamic Object Generation Network which takes as inputs estimated depth and an initial forward trajectory of the virtual object from the PyBullet and outputs a 'corrected' trajectory via a neural re-simulation step. To visualize all the trajectories in this paper, we composited the virtual object at each time step with the input image; warmer colors indicate earlier time steps.

Abstract

We introduce a method to generate videos of dynamic virtual objects plausibly interacting via collisions with a still image's environment. Given a starting trajectory, physically simulated with the estimated geometry of a single, static input image, we learn to 'correct' this trajectory to a visually plausible one via a neural network. The neural network can then be seen as learning to 'correct' traditional simulation output, generated with incomplete and imprecise world information, to obtain context-specific, visually plausible re-simulated output, a process we call neural re-simulation. We train our system on a set of 50k synthetic scenes where a virtual moving object (ball) has been physically simulated. We demonstrate our approach on both our synthetic dataset and a collection of real-life images depicting everyday scenes, obtaining consistent improvement over baseline alternatives throughout.

Video
Bibtex
@ARTICLE{InnamoratiEtAl:DynamicBounce:ICCV:2019,
    author = {Innamorati, Carlo and Russell, Bryan and Kaufman, Danny and Mitra, Niloy J.},
    title = {Neural Re-Simulation for Generating Bounces in Single Images},
    journal = {{ICCV}},
    year = 2019
}
Acknowledgements

This work was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 642841 and by the ERC Starting Grant SmartGeometry (StG-2013-335373).

Links
Paper draft
11MB
arXiv
Supplementary
(soon)
Code
(soon)