Computer Vision, 3D and the connecting factor - AI

3D scene reconstruction from single image

by Peter
Reading Time: 2 minutes

This paper by Facebook research on how to use neural networks to analyze one image of a scene, segment it into the seen 3D models within it and automatically create meshes\voxels from that single image.
Link to paper: https://arxiv.org/abs/1906.02739

Example of 3D scene reconstruction
Example of scene 3D reconstruction

Why single image?

Using multiple image will bring better results and reconstruction accuracy, so why use single images only?

It’s easier

Training datasets are more available for single image. The architecture of the neural network is easier to model and explain when it is a single image, it requires less computational resources to train over single image.

It’s more interesting

Once good reconstruction accuracy is reached with a single image, we know that the structure of the neural network is good. It is then possible to change this structure to add more images as input, be it changing the neural network itself, changing the input vector which it receives, averaging over the output of the network or other combination methods. So, actually, multi-image reconstruction is a subgroup of single image reconstruction.

AI and humans

Sometimes people get afraid that AI will replace us all. Well, if it will be able to reach singularity (can read more here https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html ) then yes, it could happen. But, neural networks which transfer 2D images into 3D models is not what’s going to bring this change.
Current technological developments allow to minimize repetitive tasks of humans, and actually facilitate more time, money and energy for creative valuable tasks for humans.

This might lead to a change in the workforce structure in the future, creating new jobs and making older jobs obsolete, but so did the invention of the car (which almost eliminated the use for coachmen but allowed for more accessible transportation and creation of jobs for taxi\bus\truck drivers), the invention of the telegram and many more examples.

Imagine giving a 2D\3D artist a tool in which he can draw whatever shape he likes in 2D and a software can create a corresponding 3D representation, this might open new possibilities both for modelling, for art, for VR\AR, for printing and for other industries that might pop up in the future. Or, just in the short term, making 3D scanning and modelling a much cheaper and faster processes for makers.

Ideas for future research

  • AI which gets as input a point cloud (instead of image) and reconstructs and accurate 2D mesh
  • Camera and lighting pose and parameters estimation

3 comments

Implicit-Decoder part 1 - 3D reconstruction - 2d3d.ai October 11, 2019 - 15:34

[…] 3D scene reconstruction from single image […]

Reply
Implicit-Decoder part 2 – 3D generation - 2d3d.ai November 16, 2019 - 20:21

[…] Implicit-Decoder part 1 – 3D reconstruction – 2d3d.ai on 3D scene reconstruction from single image […]

Reply
back_to_code October 9, 2020 - 15:52

Just thing about making scene graphs using these. Then using Graphs you can do magic. (Google it)

Reply

Leave a Comment

You may also like