3D scene reconstruction from single image

Reading Time: 2 minutes

This paper by Facebook research on how to use neural networks to analyze one image of a scene, segment it into the seen 3D models within it and automatically create meshes\voxels from that single image.
Link to paper: https://arxiv.org/abs/1906.02739

Why single image?

Using multiple image will bring better results and reconstruction accuracy, so why use single images only?

It’s easier

Training datasets are more available for single image. The architecture of the neural network is easier to model and explain when it is a single image, it requires less computational resources to train over single image.

It’s more interesting

Once good reconstruction accuracy is reached with a single image, we know that the structure of the neural network is good. It is then possible to change this structure to add more images as input, be it changing the neural network itself, changing the input vector which it receives, averaging over the output of the network or other combination methods. So, actually, multi-image reconstruction is a subgroup of single image reconstruction.

AI and humans

Sometimes people get afraid that AI will replace us all. Well, if it will be able to reach singularity (can read more here https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html ) then yes, it could happen. But, neural networks which transfer 2D images into 3D models is not what’s going to bring this change.
Current technological developments allow to minimize repetitive tasks of humans, and actually facilitate more time, money and energy for creative valuable tasks for humans.

This might lead to a change in the workforce structure in the future, creating new jobs and making older jobs obsolete, but so did the invention of the car (which almost eliminated the use for coachmen but allowed for more accessible transportation and creation of jobs for taxi\bus\truck drivers), the invention of the telegram and many more examples.

Imagine giving a 2D\3D artist a tool in which he can draw whatever shape he likes in 2D and a software can create a corresponding 3D representation, this might open new possibilities both for modelling, for art, for VR\AR, for printing and for other industries that might pop up in the future. Or, just in the short term, making 3D scanning and modelling a much cheaper and faster processes for makers.

Ideas for future research

• AI which gets as input a point cloud (instead of image) and reconstructs and accurate 2D mesh
• Camera and lighting pose and parameters estimation