Experimental Results

Experiments on Synthetic Data

In this section, we experiment several aspects of different reconstruction algorithms. We first use synthetic piece of papers, such as those of figure 7.2, randomly generated using the code provided by (142). The piece of papers are square and 200mm wide. The input images are simulated by projecting the deformed piece of paper with a virtual camera placed at approximately 1 meter of the paper sheet and with a focal length of 36mm. A set of $n_c$ point correspondences are generated by taking random locations on the 3D surface. A zero mean Gaussian noise with standard deviation of 1 pixel is added to the point correspondences. There are no self-occlusion in the data.

Figure 7.2: Example of randomly generated piece of paper. Left: 3D surface. Middle: template image. Right: input image. The blue dots are examples of point correspondences.

Several algorithms are compared in our experiments:

• SOCPimg: our point-wise method described in section 7.3.2;
• FFDref: our smooth reconstruction algorithm described in section 7.4.2;
• FFDinit: the initial solution of our smooth reconstruction algorithm, as described in section 7.4.2;
• Salz: the convex formulation proposed in (166). This method is similar to SOCPimg except for the noise that is not handled the same way. In (166), the author minimizes a cost function that includes a `reprojection error' in order to cope with the noise. In SOCPimg, the noise is handled with hard constraints.
• PerrioInit: the `upper depth bound' approach of (143,144) which is a point-wise algorithm that iteratively enforces the inextensibility constraints ;
• PerrioRef: the `refined approach' of (143,144) which minimizes a cost function resulting in a refined estimation of the 3D points obtained with PerrioInit.

Reconstruction Errors

The discrepancy between the reconstructed and the ground truth surfaces are quantified with two measures, depending on the surface model used by the algorithms. The point-wise reconstruction error (PWRE), denoted $e_p$, can be used for all the algorithms. It is defined by:

$$e_p = \frac{1}{n_c} \sum_{i=1}^{n_c} \Vert \mathbf{Q}_i - \hat{\mathbf{Q}}_i \Vert.$$ (7.16)

For algorithms that uses more complex surface models, such as triangular meshes or FFD, we measures the surface reconstruction error (SRE), denoted $e_s$. It is the difference between the reconstructed surface  $\mathcal {W}_{\mbox{\boldmath ${\ell}$}}$ and the ground truth surface  $\hat{\mathcal {W}}$:

$$e_s = \iint \big\Vert \mathcal {W}_{\mbox{\boldmath${\ell}$}}(\mathbf{q}) - \hat{\mathcal {W}}(\mathbf{q}) \big\Vert \mathrm d\mathbf{q}.$$ (7.17)

In this experiment, we use $1{,}000$ randomly generated paper sheets with 150 points correspondences. Figure 7.3 (a) shows the PWRE for all the algorithms and figure 7.3 (b) shows the SRE for the algorithms that use a complex surface model. The main result of this experiment is that our approach FFDref gives the smallest reconstruction errors (PWRE and SRE). Globally, the methods that use complex surface models get better results than the point-wise approaches.

Figure 7.3: Comparison of the reconstruction errors for different algorithms. The central red line is the median. The limits of the blue box are the 25th and the 75th percentiles. The black `whiskers' cover approximately 99.3% of the experiment outcomes. The green crosses are the maximal errors over the 1000 trials.

(a) Point-wise reconstruction error		(b) Surface reconstruction error

Length of Geodesics

When a reconstructed 3D surface is reconstructed in a truly inextensible way, the transformation of the straight line linking two points in the template image must be the geodesic linking the corresponding two 3D points on the surface. In particular, the length of these two paths must be identical. Testing this hypothesis for our algorithms FFDinit and FFDref is the goal of this experiment. To do so, we use the same data than in the previous experiment. For each surface, we choose randomly $10{,}000$ pairs of points in the template image. For each pair of points $(\mathbf{g}_i,\mathbf{g}_j)$, the length  $l^{3D}_{ij}$ of the deformed path linking the 3D points  $\mathcal {W}_{\mbox{\boldmath ${\ell}$}}(\mathbf{g}_i)$ and  $\mathcal {W}_{\mbox{\boldmath ${\ell}$}}(\mathbf{g}_j)$ on the surface is approximated with the following formula:

$$l^{3D}_{ij} = \sum_{k=1}^{n_g} \left\Vert \mathcal {W}_{\mbox{\bo... ...style{\frac{k-1}{n_g}} \Vert \mathbf{g}_j - \mathbf{g}_i \Vert\big)\right\Vert,$$ (7.18)

where $n_g$ is the number of intermediate points used for the approximation (we use $n_g=200$ since we experimentally observed that the approximation stabilizes for values of $n_g$ greater than 180). The lengths of the deformed paths are plotted against their reference length in the template image in figure 7.4 (a) for FFDinit and in figure 7.4 (b,c) for FFDref. Figure 7.4 (b) and figure 7.4 (c) show that, with the surfaces reconstructed with FFDref, the length of the deformed paths are almost equal to the length they should have if they were actual geodesics. In other words, our approach FFDref reconstructs 3D surfaces which are truly inextensible. On the other hand, figure 7.4 (a) shows that the initial solution FFDinit (which is just an FFD fitted to a sparse set of reconstructed 3D points) seems to be much less inextensible.

Figure 7.4: Plot of the length of deformed paths against the length they should have if the reconstructed surface was truly inextensible. The red diagonal line is the place where all the blue points should be for inextensible surfaces.

(a) FFDinit		(b) FFDref		(c) Magnification of (b)

Let  $l^{2D}_{ij}$ be the Euclidean distance between the points  $\mathbf{g}_i$ and  $\mathbf{g}_j$. Table 7.2 gives some statistics on the relative error between the computed length  $l^{3D}_{ij}$ and the reference length  $l^{2D}_{ij}$, i.e. the quantity  $(l^{2D}_{ij} - l^{2D}_{ij}) / l^{3D}_{ij}$. These numbers confirm the results seen in figure 7.4.

Table 7.2: Statistics on the relative errors between the length of transformed paths and the length they should have.

	Mean	Std deviation	Median	Minimun	Maximum
FFDinit	$0.0119$	$0.0417$	$0.0036$	$-1.9689$	$0.8931$
FFDref	$2.0084 \times 10^{-5}$	$7.1965 \times 10^{-4}$	$5.8083 \times 10^{-6}$	$-0.0505$	$0.3396$

Gaussian curvature

The Gaussian curvature is the product of the two principal curvature (which are the reciprocal of the radius of the osculating circle). For an inextensible surface, the Gaussian curvature is null. In this experiment, we check if this property is satisfied by the smooth surfaces reconstructed with FFDinit and FFDref. We used the same $1{,}000$ reconstructed surfaces than in the previous experiment. The Gaussian curvature, denoted $\kappa$, is computed for $10{,}000$ randomly chosen points on the surface with the formula $\kappa = \frac{\det(\mathbf{I} \mathbf{I})}{\det(\mathbf{I})}$, where  $\mathbf{I}$ and  $\mathbf{I}\mathbf{I}$ are the first and the second fundamental forms of the parametric surface (86). The results of this experiment are reported in table 7.3. It shows that, in average, the Gaussian curvature of the surfaces reconstructed using FFDref are consistently close to 0. It also shows that FFDref gives Gaussian curvatures which are 100 times smaller than the ones obtained with FFDinit. These results demonstrate that the surfaces reconstructed with our approach FFDref are indeed inextensible. Note that this kind of experiment cannot be achieved if a smooth surface is not available.

Table 7.3: Statistics on the (absolute value of the) Gaussian curvatures for $1{,}000$ reconstructed surfaces and $10{,}000$ points per surface.

	Mean	Std deviation	Median	Minimun	Maximum
FFDinit	$4.9458 \times 10^{-4}$	$0.0875$	$9.7302 \times 10^{-5}$	$7.5122\times 10^{-14}$	$258.2379$
FFDref	$5.0046\times 10^{-6}$	$7.1320 \times 10^{-4}$	$1.7333 \times 10^{-6}$	$2.2325 \times 10^{-14}$	$1.5199$

Experiments on Real Data

The algorithms used in the synthetic experiments of section 7.5.1 are applied to real data in figure 7.5 and figure 7.6. These figures shows that our approaches gives good results on real data. In particular, figure 7.5 shows that our method FFDref outperforms the other approaches in the presence of a self-occlusion. This comes from the fact that FFDref requires the surface to be inextensible everywhere, even if there are no point correspondences (which is the case on the self-occluded part of the paper sheet). An accurate stereo reconstruction of the surface in figure 7.6 were available. We compare in table 7.4 the average 3D errors between the surfaces reconstructed with a monocular approach to the stereo reconstruction. Again, our method FFDref is the one giving the best results.

Figure 7.5: Illustration of monocular reconstruction algorithms in the presence of a self-occlusion (the point correspondences were automatically extracted using (74)). Note how our algorithm FFDref is able to recover a reasonable shape for the occluded part.

Figure 7.6: Illustration of the results obtained with several monocular reconstruction algorithms. First row: input image along with a reprojection of the reconstructed 3D surface. Second row: reconstructed surface from a different point of view. Note that the stereo reconstruction (first column) is not a monocular algorithm: it is just used to assert the quality of the other reconstructed surfaces (see table 7.4).

Table 7.4: Average 3D error (in millimeters) with respect to the stereo reconstruction of the surface for the surfaces of figure 7.6.

PerrioRef	SOCPimg	Salz	FFDinit	FFDref
2.388	2.261	4.743	2.259	1.991