Depth Map Upsampling via Multi-Modal Generative Adversarial Network

Daniel Stanley Tan, Jun-Ming Lin, Yu-Chi Lai, Joel Ilao, Kai-Lung Hua*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review


Autonomous robots for smart homes and smart cities mostly require depth perception in order to interact with their environments. However, depth maps are usually captured in a lower resolution as compared to RGB color images due to the inherent limitations of the sensors. Naively increasing its resolution often leads to loss of sharpness and incorrect estimates, especially in the regions with depth discontinuities or depth boundaries. In this paper, we propose a novel Generative Adversarial Network (GAN)-based framework for depth map super-resolution that is able to preserve the smooth areas, as well as the sharp edges at the boundaries of the depth map. Our proposed model is trained on two different modalities, namely color images and depth maps. However, at test time, our model only requires the depth map in order to produce a higher resolution version. We evaluated our model both quantitatively and qualitatively, and our experiments show that our method performs better than existing state-of-the-art models.
Original languageEnglish
Article number1587
Number of pages13
Issue number7
Publication statusPublished - 1 Apr 2019
Externally publishedYes


  • Depth upsampling
  • Encoder-decoder networks
  • Generative adversarial networks


Dive into the research topics of 'Depth Map Upsampling via Multi-Modal Generative Adversarial Network'. Together they form a unique fingerprint.

Cite this