3D reconstruction and semantic mapping are of great importance for many tasks and applications, such as consumer robots, augmented reality, digital heritage, and autonomous vehicles. Despite the drastic advancements in solving the 3D reconstruction problem, it is still challenging to reconstruct accurate 3D models and create semantic maps. Within this dissertation, contributions are made to take steps closer towards optimal 3D reconstruction and semantic mapping.

It is crucial to have an easy performance evaluation method for advancing 3D mapping systems. Thus, in Chapter 5, we propose dense map posterior (DMP) for 3D reconstruction and mapping performance evaluation that can work without any ground-truth data. With this metric, one can evaluate 3D mapping systems on any public or new data without worrying about the availability of ground-truth data captured by expensive and bulky equipment. We also show that the DMP can be used beyond the evaluation of final results. It can act as a supervisory figure-of-merit signal during 3D reconstruction processes.

To improve 3D reconstruction results, we propose a novel 3D reconstruction system that corrects surface loops with sparse feature-based bundle adjustment. In the system, fast 3D surface-based loop detection is done by a GPU-accelerated random sample consensus algorithm with optimized randomness supported by fractional calculus, which is in Chapter 3. Then, to solve a low-precision problem in surface loop detection, in Chapter 4, an online method for loop sifting is proposed for real-time feedback to users. For the best 3D reconstruction performance, an offline method for loop sifting and majorization is proposed in Chapter 6. State-of-the-art performance is observed in experiments on public and our datasets.

In Chapter 7, an exploration of semantic mapping is carried out. A simple and effective real-time 3D semantic mapping method is proposed. In addition, a benchmark suite with a dataset derived from the KITTI dataset and three novel metrics are developed for semantic mapping evaluation.

Finally, conclusions and future works are presented in the last chapter.