The work presented in this thesis deals with the modelling and the estimation of parametric models in surface fitting, image registration, and 3D reconstruction. We have presented various contributions related to different topics in Computer Vision: range surface fitting, feature-based and direct image registration, 3D reconstruction of a deforming surface. Several aspects have been considered from the design of parametric models to the estimation of the parameters and the hyperparameters themselves. Although these contributions may appear quite different, they are linked together in different ways. First, all of our contributions consider a deformable environment. This is interesting since over the past few decades Computer Vision has produced, for the topics we have considered, useful and effective results but mainly in rigid environments. Second, it is clear that our contributions all point at a single objective: the reconstruction of arbitrary deformable surfaces. Indeed, surface fitting is a general problem which is useful to convert a sparse set of 3D points into a smooth and analytical representation of the same surface. Image registration (whatever approach is used) is generally one of the early steps in 3D surface reconstruction algorithms. In this thesis, we surely not have solved the ultimate goal of Computer Vision that would be the 3D reconstruction of arbitrary surfaces in deformable environments but our contributions constitute a step towards this goal.

Synthesis of our contributions.

As said previously, we have made contributions linked to several topics of Computer Vision. These contributions include the following items.

Surface fitting.
We have proposed several methods to fit a smooth parametric surface to range data. These algorithms are efficient in the sense that they provide accurate results in reasonable computational times. This has been made possible by a correct modelling of the considered data. For instance, we have taken into account the fact that the typical noise of a depth map acquired with a ToF camera is heteroskedastic. We have also exploited the particular properties of the tensor-product surface model to achieve good computational performance when the data are arranged on a regular grid.

Image registration.
We have made several contributions related to image registration ranging to a general model of warp able to model perspective effects and deformable environment to specific technique to parameter and hyperparameter estimations. More precisely, we have exploited the properties of the NURBS to efficiently model deformations appearing under perspective image conditions. Concerning the estimation of the parameters, we have proposed contributions related with the two main approaches to image registration: the direct approach and the feature-based approach. In the direct approach, we have presented a new algorithm that allows one to discard the delicate problem related to the region of interest. This has been made possible by an adequate modelling of the so-called off-target pixels which consisted in saying that such pixels may be seen as classical outliers. Therefore, a standard robust estimation framework based on M-estimators allowed us to have a unified processing of all the pixels. In feature-based image registration, we have proposed a new approach to tune any hyperparameters that may intervene in such problems.

3D Surface reconstruction.
The purpose of our last contribution has been to reconstruct a 3D surface in a deformable environment from a monocular video. It is well known that such a problem is intrinsically ill-posed since potentially there exists an infinite number of surfaces having the same projection in an image. We thus considered common assumptions which consisted in saying that the surface to reconstruct was inextensible and that a reference shape was known. Given these assumptions, we proposed two algorithms to solve the reconstruction problem. First, we proposed a point-wise algorithm, i.e. an algorithm that reconstructed only a sparse set of 3D points. This first algorithm is formulated as an SOCP problem. Second, we proposed a method that directly reconstruct a smooth parametric surface. It is formulated as a standard least-squares minimization problem. Our main contribution was to propose a new term enforcing the inextensibility of the reconstructed surface.

Hyperparameter selection.
The automatic selection of hyperparameters has been an important transversal topic in the work presented in this thesis. We have emphasized the fact that choosing proper hyperparameters is important to get correct results. We have also shown that this choice may not be always trivial. In this thesis, hyperparameters have been handled in several contexts. We have proposed two different ways of automatically tuning the hyperparameters in range surface fitting: the L-Tangent Norm criterion and an adaptation of Morozov's discrepancy principle. In feature-based image registration, we have proposed a new framework to automatically select any hyperparameter. It relies on the fact that in feature-based image registration, the point correspondences are not the only available data: there is also the photometric information contained in the images. Therefore, we have proposed to use the point correspondences as a training set for the natural parameters and the photometric information as a test set for the hyperparameters.

What's next?

The ultimate goal of automatically reconstructing an arbitrary deforming surface has not been reached yet. There is still a long way to go before any surface in any context could be reconstructed from images. Even though the design of such a ultimate algorithm would be a nice proceeding to this thesis, we also focus on shorter term goals.

Range surface fitting.
Even though we have proposed efficient algorithms for fitting surfaces on range data, there are some problems which still await to be solved. In particular, handling discontinuities is a major issue in range surface fitting. Several approaches may be considered, such as using a different parametric surface model that would be able to properly model discontinuities. Such a model would require one to design specific parameter estimation techniques.

Hyperparameters in feature-based image registration.
We have proposed a generic framework to automatically determine proper hyperparameters in feature-based image registration. However, the criteria resulting of our idea are generally difficult to optimize. In particular, they may have several local minima and they may not be continuous. An automatic procedure to minimize the proposed criterion would be a natural conclusion to this work. Given the nature of the cost function to optimize, combinatorial optimization based on metaheuristics (such as simulated annealing, Kangaroo's algorithm, or genetic algorithms (71)) would be a good starting point to solve this problem.

Monocular reconstruction of smooth surface.
We have proposed an effective way to reconstruct smooth and inextensible surfaces from a monocular sequence of images. This has been made possible by introducing a term in the cost function that penalizes departure from pure inextensibility. Although efficient, this approach has an important computational cost. Other ways of enforcing inextensibility should be explored. In particular, noticing that an inextensible surface has a null Gaussian curvature may be a good starting point. Besides, not all surfaces are inextensible. Other types of constraints should also be envisaged.

Regularization terms.
As we have showed in this thesis, it is often required to use a regularization term in order to have well-posed problems. We often used regularization terms based on the bending energy, regardless of the problem's nature. Other regularization terms could be useful. In particular, it would be interesting to study regularization terms of order higher than two. It would also be interesting to use multiple regularization terms and select the most appropriate one based on the data.

Contributions to Parametric Image Registration and 3D Surface Reconstruction (Ph.D. dissertation, November 2010) - Florent Brunet
Webpage generated on July 2011
PDF version (11 Mo)