Multimodal 3D Façade Reconstruction Using 3D LiDAR and Images

Abstract

3D urban reconstruction is an active research area which is significant for many applications, such as virtual tours and urban planning. 3D facade reconstruction is a fundamental task in this research area. 3D real-time LiDAR is an ideal choice for capturing the 3D data in large-scale 3D urban reconstruction because of its fast working speed and mobility. However, the resulting 3D point cloud is very sparse and is not directly suitable for 3D reconstruction. In this paper, we aim to provide a new method for 3D facade reconstruction, which is based on the fusion of 3D terrestrial LiDAR and images. Our novelty lies in the extraction of semantic information from images and a state-of-the-art depth completion method used to estimate the missing depth information in LiDAR scans. The basic idea of our 3D facade reconstruction method is an iterative optimization process of sparse depth maps. More specifically, the input LiDAR point cloud is first converted to a depth map of the input image. Then, facade elements made of glass (e.g., windows) are extracted by using an instance segmentation technique. Once these elements are located, their exact depth values can be estimated by using weak architectural constraints. Finally, a data-driven depth completion methods based on CNN is used for predicting the rest of missing depth data in the semi-dense depth map, which generates the final completed dense depth map. Besides, we present a new dataset used for 3D facade reconstruction. The dataset contains LiDAR scans and images of the 15 different facades in the City Campus of the University of Auckland. All the 2D and 3D data in this dataset are calibrated. The proposed 3D facade reconstruction method was evaluated on this new dataset. The experimental results show that our approach can generate photorealistic 3D reconstruction models for some particular styles of facades. Also, our method is able to identify facade elements such as windows and doors to enhance 3D models of buildings semantically.