Affine Interpretation of the BS-Warps

Notation.

In this chapter, we denote  $\mathcal {W}_B$ the BS-Warp. As a reminder of section 2.3.2, a BS-Warp is defined as:

$$\mathcal {W}_B(\mathbf{q} ; \mathbf{x}) = \sum_{i=1}^m \sum_{j=1}^n \mathbf{p}_{ij} N_i(x) N_j(y),$$ (6.2)

where the functions $N_i : \mathbb{R}\rightarrow \mathbb{R}$ are the B-Spline basis functions and the $\mathbf{p}_{ij} = (p_{ij}^x, p_{ij}^y)^\mathsf{T}$ are the control points (grouped in the parameter vector  $\mathbf{x} \in \mathbb{R}^{2mn}$). The scalars $m$ and $n$ are the number of control points along the $x$ and $y$ directions respectively. Note that we consider as coincident the knot sequences used to define the B-Spline basis functions.

Affine interpretation.

If we consider that the observed surface is modelled by a threedimensional tensor-product B-Spline, the BS-Warp corresponds to the transformation between the two images under affine imaging conditions (see figure 6.1 for an illustration).

Figure 6.1: A BS-Warp can be seen as the result of a threedimensional B-Spline surface projected under affine conditions.

Let  $\mathcal {R} : \mathbb{R}^2 \rightarrow \mathbb{R}^3$ be the 2D-3D map between the first image and the threedimensional surface:

$$\mathbf{Q} = \mathcal {R}(\mathbf{q} ; \mathbf{x}) = \sum_{i=1}^m \sum_{j=1}^n \bar{\mathbf{p}}_{i,j} N_i(x) N_j(y),$$ (6.3)

with $\bar{\mathbf{p}}_{i,j} = (\bar{p}_{i,j}^x \: \bar{p}_{i,j}^y \: \bar{p}_{i,j}^z)^\mathsf{T}\in \mathbb{R}^3$ the 3D control points of the surface. Let  $\mathcal {A}: \mathbb{R}^3 \rightarrow \mathbb{R}^2$ be the affine projection of the surface into the second image:

$$\mathcal {A}(\mathbf{Q}) = \mathsf{A} \mathbf{Q},$$ (6.4)

with  $\mathsf{A}$ the matrix which models the affine projection, assuming that the 3D surface is expressed within the coordinate frame of the second camera:

$$\mathsf{A} = \begin{pmatrix} a_x & 0 & 0 0 & a_y & 0 \end{pmatrix}.$$ (6.5)

Given these notations, the warped point  $\mathbf{q}'$ can be written  $\mathcal {A} ( \mathcal {R} ( \mathbf{q} ) )$ which, after expansion, gives:

$$\mathbf{q}' = \sum_{i=1}^m \sum_{j=1}^n \begin{pmatrix}\bar{p}_{i,j}^x \bar{p}_{i,j}^y \end{pmatrix} N_i(x) N_j(y).$$ (6.6)

Equation (6.6) matches the definition of equation (6.2) of a BS-Warp.

BS-Warps are not suited for perspective imaging conditions.

As we just demonstrated, BS-Warps are obtained under affine imaging conditions. However, this does not prove that they are not suited for perspective imaging conditions. In this paragraph, we experimentally illustrate that BS-Warps are indeed not suited for perspective imaging conditions. This comes from the fact that the division appearing in a perspective projection is not present in the BS-Warp model.

The bad behavior of the BS-Warp in the presence of perspective effects is illustrated in figure 6.2. In this experiment, we simulate a set of point correspondences by transforming a regular grid with an homography parametrized by a scalar $a$ that controls the amount of perspective effect. The $3 \times 3$ matrix of this homography, $\mathsf{H}_a$, is given by:

$$\mathsf{H}_a \propto {\frac{1}{a}} \begin{pmatrix} (a+1)^2/4 & ... ...(a^2-1)/4 0 & a(a+1)/2 & 0 -(a^2-1)/4 & 0 & (a+1)^2/4 \end{pmatrix},$$ (6.7)

where $\propto$ indicates equality up to scale. The larger $\vert a-1\vert$, the more important the perspective effect. Figure 6.2 c clearly shows that the transformation can be correctly modelled by a BS-Warp only when $a=1$, i.e. when the perspective effect is barely existant. Figure 6.2 d shows that the number of control points of a BS-Warp must be significantly large in order to correctly model a perspective effect.

Figure 6.2: Bad behavior of the BS-Warp in the presence of perspective effects. (a) Data points on a regular grid (first image). (b) Transformed points simulating a perspective effect with an homography (second image). (c) Influence of the perspective effect on a BS-Warp with 16 control points. (d) The perspective effect ( $a=\frac{5}{2}$) can be modeled with a BS-Warp but it requires a large amount of control points.

(a)	(b)	(c)	(d)