Introduction to medical image registration Pietro Gori

(1)

Introduction to medical image registration

Pietro Gori

Enseignant-chercheur Equipe IMAGES - Télécom ParisTech pietro.gori@telecom-paristech.fr

October 10, 2018

(2)

Summary

1 Introduction

2 Geometric global transformations

3 Image warping and interpolations

4 Intensity based registration

5 Landmark based registration Affine registration

Procrustes superimposition

Non-linear registration (small displacement) Non-linear registration (diffeomorphism)

(3)

Medical image geometric registration

Definition

Geometric: find theoptimal parameters of ageometric

transformationto spatially align two different images of the same object. It establishesspatial correspondence between the pixels of the source (or moving) image with the ones of the target image

Photometric: Modify the intensity of the pixels and not their position Applications

Compare two (or more) images of the same modality (e.g. T1-w MRI of the brain of two different subjects)

Combine information from multiple modalities (e.g. PET, DWI, T1-w MRI of the brain of the same subject)

Longitudinal studies (e.g. monitor anatomical or functional changes of the brain over time)

(4)

Medical Image registration - Applications

(5)

Image registration components

Dimensionality: 2D/2D, 3D/3D, 2D/3D Transformation (linear / non-linear)

Similarity metric(e.g. intensities, landmarks, edges, surfaces) Optimization procedure

Interaction (automatic / semi-automatic / interactive) Modalities(mono-modal / multi-modal)

Subjects(intra-subject / inter-subject / atlas construction)

(6)

Introduction

LetI and J be the source and target images. They show the same anatomical object, most of the time with a different field of view and resolution (sampling)

I(x, y) andJ(u, v) represent the intensity values of the pixels located onto two regular grids: {x, y} ∈Ω_I and{u, v} ∈Ω_J

For the same subject, the same anatomical pointz can be in position x_z, y_z in I and inu_z, v_z inJ

(7)

Mathematical definition

Both I andJ are functions:

I(x, y) : Ω_I⊆R² → R

(x, y) → I(x, y)

We look for a geometric transformationT, which is a 2D warping parametric function that belongs to a certain familyΓ:

T_φ(x, y) : R² → R²

(x, y;φ) → T_φ(x, y)

φis the vector of parameters of T. We look for a transformation that

(8)

Mathematical definition

Most of the time,I andJ are simply seen as matrices whose coordinates (x, y) and(u, v) are thus integer-valued (number of line and column)

The values of the intensities of the pixels can be real numbers R (better for computations) or integer, usually in the range[0,255] for a gray-scale image

There are several kind of transformations:

modèle global modèle par morceaux

(régional)

modèle local

(9)

Summary

1 Introduction

(10)

Global transformations

Global means that the transformation is the same for any points p Scaling - multiply each coordinate by a scalar

"

u v

#

=

"

sx 0 0 s_y

# "

x y

#

(1)

(11)

Global transformations

Rotation- WRT origin. Let p= [x y]^T andp⁰ = [u v]^T

"

x y

#

=

"

rcos(α) rsin(α)

#

"

u v

#

=

"

rcos(α+θ) rsin(α+θ)

#

=

"

r(cos(α) cos(θ)−sin(α) sin(θ)) r(sin(α) cos(θ) + cos(α) sin(θ))

#

=

"

xcos(θ)−ysin(θ) ycos(θ) +xsin(θ)

#

=

"

cos(θ) −sin(θ) sin(θ) cos(θ)

# "

x y

#

R=

"

cos(θ) −sin(θ) sin(θ) cos(θ)

#

det(R) = 1 R⁻¹=R^T

(12)

Global transformations

Reflection

Horizontal (Y-axis)

"

u v

#

=

"

rcos(π−α) rsin(π−α)

#

=

"

−1 0 0 1

# "

x y

#

Vertical (X-axis)

"

u v

#

=

"

rcos(³₂π+α) rsin(³₂π+α)

#

=

"

1 0

0 −1

# "

x y

#

det(R) =−1 R⁻¹=R^T

(13)

Global transformations

Shear- Transvection in french

"

u v

#

=

"

1 λx

λy 1

# "

x y

#

(14)

2D Linear transformations

"

u v

#

=

"

a b c d

# "

x y

#

Linear transformations are combinations of:

scaling rotation reflection shear

Properties of linear transformations:

origin is always transformed to origin parallel lines remain parallel

ratios are preserved lines remain lines

(15)

Translation

It does not have a fixed point→ no matrix multiplication

"

u v

#

=

"

x y

# +

"

t_x t_y

#

=

"

x+t_x y+t_y

#

(2)

(16)

Homogeneous coordinates - 2D affine transformation

Instead than 2D matrices we use 3D matrices.

Affine transformation: combination of linear transformations and translations

Letp= [x y]^T and p⁰ = [u v]^T, we obtain p⁰ =Tp





 u v 1





=







a b tx

c d ty

0 0 1







| {z }

T





 x y 1







Properties of affine transformations:

origin isnot always transformed to origin parallel lines remain parallel

ratios are preserved lines remain lines

(17)

2D Projective transformations (homographies)





 u v 1





=







a b g c d h e f i











 x y 1







Properties of projective transformations:

origin isnot always transformed to origin parallel lines donot necessarily remain parallel ratios arenotpreserved

lines remain lines

(18)

Examples

(19)

Summary

1 Introduction

(20)

Forward warping

T_φ(x, y) : R² → R²

(x, y;φ) → T_φ(x, y)

Ideally, ifΩ_I andΩ_J were continuous domains, we would simply map (x, y)to (u, v) =T_φ(x, y)and then compare I(x, y) with J(u, v) However, Ω_I andΩ_J are regular grids ! What if T_φ(x, y) is not on the grid Ω_J ? → Splatting, add contribution to neighbor pixels.

(21)

Inverse warping

Find the pixel intensities for the deformed imageI_T starting fromΩ_J: (x, y) =T⁻¹_φ (u, v)

Assign toI_T(u, v)the pixel intensity in I(x, y) What ifT⁻¹_φ (u, v) is not on Ω_I ? → Interpolation !

(22)

Interpolation

Goal: estimate the intensity value on points not located onto the regular grids

nearest neighbor bilinear

cubic lanczos ...

(23)

Interpolation

Nearest neighbor: J(u, v) =I(round(x),round(y)) Bilinear: f(x,q)−f(x,y)

∆y = ^f(x,y+1)−f(x,q)

1−∆y →

f(x, q) = (1−∆y)f(x, y) +f(x, y+ 1)∆y. Similarly,

f(x+ 1, q) = (1−∆y)f(x+ 1, y) +f(x+ 1, y+ 1)∆y. Then, f(p, q) =f(x, q)(1−∆x) +f(x+ 1, q)∆x

(24)

Interpolation examples

Every time we deform an image, we need to interpolate it. For instance, this is the result on Lena after 10 rotations of 36 degrees:

Figure 1:Original image - Nearest Neighbour - Bilinear

Source : http://bigwww.epfl.ch/demo/jaffine/index.html (Michael Unser)

(25)

Deformation algorithm

Recipe:

Given a source imageI and a global affine transformationT, compute the forward warping of the extremities of I. This gives the bounding box of the deformed image IT

Given the bounding box ofI_T, create a new grid within it with, for instance, the same characteristics of Ω_J to allow comparison between IT andJ

Use the inverse warping and interpolation to compute the intensity values at the grid points of the warped imageI_T (we avoid holes) Be careful! During the inverse warping, points that are mapped outside Ω_I are rejected.

(26)

Transformation parameters

Given a transformationT defined by a set of parameterθ and two imagesI andJ, how do we estimateθ ?

By minimizing a cost function:

θ^∗= arg min

θ

d(IT, J) (3)

The similarity measure dmight be based on the pixel intensities and/or on corresponding geometric objects such as control points (i.e. landmarks), curves or surfaces

(27)

Transformation parameters

Given a transformationT defined by a set of parameterθ and two imagesI andJ, how do we estimateθ ?

By minimizing a cost function:

θ^∗= arg min

θ

d(IT, J) (3)

The similarity measure dmight be based on the pixel intensities and/or on corresponding geometric objects such as control points (i.e.

landmarks), curves or surfaces

(28)

Summary

1 Introduction

(29)

Intensity based registration

Same modality

Sum of squared intensity differences (SSD). Best measure whenI and J only differ by Gaussian noise. Very sensitive to “outliers” pixels, namely pixels whose intensity difference is very large compared to others.

d(I_T, J) =^X

u

X

v

(J(u, v)−I(T⁻¹_φ (u, v)))²= (J(u, v)−I_T(u, v))² (4) Correlation coefficient. Assumption is that there is a linear

relationship between the intensity of the images

d(IT, J) =

P

u

P

v(J(u, v)−J¯)(IT(u, v)−I¯T) q

P P

(J(u, v)−J¯)²^P ^P (I (u, v)−I¯ )² (5)

(30)

Intensity based registration

Multi modality

Mutual information. We first need to define the joint histogram between I_T and J. The value at location (a, b) is equal to the number of intensity values that have intensityain IT(u, v)and intensity bin J(u, v). For example, a joint histogram which has the value of 2 in the position (7,3) means that we have found two locations (u, v) where the intensity of the first image was 7

(IT(u, v) = 7) and the intensity of the second was 3 (J(u, v) = 3).

By dividing by the total number of pixels, we obtain a joint probability density function (pdf) pIT,J.

The definition of joint entropy is:

H(IT, J) =−^X

a

X

b

pIT,J(a, b) log(pIT,J(a, b)) (6) whereaand bare defined within the range of intensities in IT andJ respectively

(31)

Intensity based registration

Figure 2:Joint histogram of a) same modality (IRM) b) different modality

(32)

Intensity based registration

The definition of Mutual information is:

M(I_T, J) =H(I_T) +H(J)−H(I_T, J) (7) whereH(IT) =−^P_apIT(a) log(pIT(a))and

H(J) =−^P_bpJ(b) log(pJ(b)). The two marginal pdf pIT andpJ are simply the accumulations over the columns and rows ofp_I_T_,J

respectively. It results:

M(IT, J) =^X

a

X

b

p_I_T_,J(a, b) log p_I_T_,J(a, b)

p_I_T_,J(a)p_J(b) (8)

(33)

Intensity based registration

M(I_T, J) =H(I_T) +H(J)−H(I_T, J) =H(J)−H(J|I_T) (9) where the conditional entropy

H(J|I_T) =−^P_a^P_bp_I_T_,J(a, b) logp_J|I_T(b|a).

Mutual information measures the amount of uncertainty about J minus the uncertainty aboutJ when IT is known, that is to say, how much we reduce the uncertainty aboutJ after observing IT. It is maximized when the two images are aligned. [8]

Maximizing M means finding a transformations Tthat makesIT the best predictor for J. Or, equivalently, knowing the intensity I_T(u, v) allows us to perfectly predictJ(u, v).

(34)

Optimization procedure

Pixel-based similarity measures need an iterativeapproach where an initial estimate of the transformation is gradually refined using the gradient and, depending on the method, also the Hessian of the similarity measure with respect to the parametersθ

Possible algorithms: gradient descent, Newton-Raphson, Levenberg-Marquardt

Problem of the “local minima”→ stochastic optimization, line search, trust region, multi-resolution (first low resolution and then higher resolution)

Validation →visual inspection, alignment of manually segmented objects, value of similarity measure

(35)

Summary

1 Introduction

(36)

Anatomical landmarks

Definition: anatomical landmark

An anatomical landmark is a point precisely defined onto an anatomical structure which establishes a correspondence among the population of homologous anatomical objects

Figure 3:Example of manually labeled landmarks

(37)

Anatomical landmarks

Given a set ofN landmarksi defined onI andN corresponding landmarksj defined onJ, we seek to minimize:

θ^∗= arg min

θ N

X

p=1

||T(i_p)−j_p||² (10)

whereT(i_p) means that we apply the deformationT to thep-th landmarkip

We suppose that all landmarks belong toR² (it would be similar for R³)

The metric is the Euclidean norm (or Frobenius when using matrices)

(38)

Landmark based affine registration

We define:

T(ip) =Aip+t ∀i∈[1, N] (11) Thus,θ=A, t

A^∗, t^∗= arg min

A,t

f(A, t) =

N

X

p=1

||Ai_p+t−jp||² (12) From which it results:

∂f

∂t = 2

N

X

p=1

(Ai_p+t−j_p) = 0→t^∗ = ¯j−A¯i (13) We notice that if we center the data (i.e. ˜ip=ip−¯iand˜jp=jp−¯j) then t^∗ = 0. The criterion thus becomesf =^P^N_p=1||A˜ip−˜jp||²

(39)

Landmark based affine registration

Now we differentiate wrtA. :

∂f

∂A =

N

X

p=1

∂||A˜ip||²

∂A −2∂hA˜ip,˜jpi

∂A

=2

N

X

p=1

A˜i_p˜i^T_p −˜j_p˜i^T_p = 2

N

X

p=1

(A˜i_p−˜j_p)˜i^T_p = 0

(14)

It results:

A^∗ =





N

X

p=1

˜jp˜i^T_p









N

X

p=1

˜ip˜i^T_p





−1

(15)

The matrix ^P^N ˜i ˜i^Tis invertible if the landmarks are not all

(40)

Landmark based affine registration

From a computational point of view, it is easier to use homogeneous coordinates:

T^∗= arg min

T

||xT−y||²_F (16) where we define







x₁ y₁ 1 0 0 0 ...

xN yN 1 0 0 0 0 0 0 x₁ y₁ 1

...

0 0 0 xN yN 1







| {z }

x





 a₁ a₂ t1

a₃ a₄ t2







| {z }

T

−





 u₁

... uN

v₁ ... vN







| {z }

y

(17)

T^∗= (x^Tx)⁻¹x^Ty (18)

(41)

Procrustes superimposition (similarity transformation)

We seek to minimize:

(s, R, t)^∗ = arg min

s,R,t N

X

p=1

||sRi_p+t−jp||²₂ (19) wheresis a uniform scaling factor (scalar) andR is a rotation matrix.

The translation vector tis, as before, equal tot^∗= ¯j−_N¹ ^P^N_p=1sRi_p. Thus, by centering the data, we obtain:

(s, R)^∗= arg min

s,R

f(s, R) =

N

X

p=1

||sR˜i_p−˜j_p||²₂=||sXR−Y||²_F (20)

Remember that: ^P^N ||˜ip−˜jp||² =||X−Y||² where

(42)

Procrustes superimposition (similarity transformation)

We minimize wrt s:

f(s, R)∝s²||XR||²_F −2shXR, Yi_F

∂f(s, R)

∂s = 2s||X||²_F −2hXR, Yi_F s^∗ = hXR, Yi_F

||X||²_F

(21)

where we use the fact thatR^TR=RR^T =I. Substituting intof: R^∗ = arg min

R

f(R) =−(hXR, Yi_F)²

||X||²_F

= arg max

R

|hXR, Yi_F|=|hR, X^TYi_F|

= arg max

R

(22)

where we use the SVD decomposition X^TY =UΣV^T and the definition of the trace hXR, Yi_F =Tr(R^TX^TY) =hR, X^TYi_F

(43)

Procrustes superimposition (similarity transformation)

Notice thatZ =U^TRV is an orthogonal matrix since it is the product of orthogonal matrices. ThusZ^TZ =I andz_j^Tz_j = 1. It follows thatzij ≤1.

R^∗ = arg max

R

|hZ,Σi_F|=|Tr(Σ^TZ)|=

=|Tr "

σ₁ 0 0 σ2

# "

z₁₁ z₁₂ z21 z22

#!

|=

2

X

d=1

σdzdd≤

2

X

d=1

σd

(23)

The maximum is obtained whenz_dd= 1 ∀d, which means when:

Z =U^TRV =I →R^∗=U V^T (24) In order to be sure thatR is a rotation matrix (det(R) = 1), we compute [5]:

"

1 0 ^#

(44)

Procrustes superimposition (similarity transformation)

To recap [5]:

R^∗ =U SV^T s^∗ = hR, Y X^Ti_F

||X||²_F = Tr(SΣ)

||X||²_F t^∗ = ¯j− 1

N

X

p=1

sRip

(26)

(45)

Non-linear registration (small displacement)

We define the deformation of a pixel at location z= (x, y) as:

T(z) =z+v(z) with v(z) =

N

X

p=1

K(z, ip)αp (27)

whereK(z, ip) is a kernel, for instanceK(z, ip) = exp (−^||z−i_λ2^p^||²²) andαp is a 2D vector which need to be estimated.

The displacement at any pointz depends on the displacement of the neighbor landmarks

(46)

Non-linear registration (small displacement)

We minimize

α^∗ = arg min

α f(α;λ) =

N

X

p=1

||i_p+v(i_p)−j_p||²₂

=

N

X

p=1

||i_p+ (

N

X

d=1

K(ip, id)αd)−jp||²₂

=||i+Kα−j||²_F

(28)

whereK=







1 K(i₁, i₂) ... K(i₁, i_N) K(i₂, i₁) 1 ... K(i₂, i_N)

... ... ... ...

K(i_N, i₁) K(i_N, i₂) ... 1





 and α= [α^T₁;...;α^T_N]

(47)

Non-linear registration (small displacement)

By differentiating wrtα:

∂||i+Kα−j||²_F

∂α = 2K^T(i+Kα−j) = 0 α^∗=K⁻¹(j−i)

(29) The matrix Kmight not always be invertible (ifλis too big for instance). We need to regularize it. A possible solution is to use a Tikhonov matrix such asα^TKα, thus obtaining:

α^∗= arg min

α f(α;λ, γ) =||i+Kα−j||²_F +γα^TKα (30)

∂f(α;λ, γ)

∂α = 2K^T(i+Kα−j) + 2γKα= 0

(31)

(48)

Non-linear registration (small displacement)

Why only small displacement ? → We could approximate the inverse of T(z) as T⁻¹(z⁰) =z⁰−v(z⁰) obtaining:

T(T⁻¹(z⁰)) =z⁰−v(z⁰) +v(z⁰−v(z⁰))6=z⁰ (32) The error is small only ifv(z⁰−v(z⁰))−v(z⁰) is small, which is the case only when the displacement is small !

We might have intersections, holes or tearing in area where the displacement is large

(49)

Small-displacement registration

Figure 4:First row: forward and inverse transformation. The first one is a one-to-one mapping whereas the second one presents intersections. Second row:

(50)

Large-displacement registration: diffeomorphism

Instead than using small-displacement transforms we should use diffeomorphisms

A diffeomorphism is a differentiable (smooth and continuous) bijective transformation (one-to-one) with differentiable inverse (i.e.

nonzero Jacobian determinant)

Using diffeomorphic transformations we can preserve the topology and spatial organization, namely no intersection, folding or shearing may occur

(51)

Diffeomorphism

Figure 5:First row: forward and inverse diffeomorphic transformation (both are one-to-one). Second row: composition of forward and inverse transformations.

(52)

Diffeomorphism

One of the most used algorithm in medical imaging to create diffeomorphic deformations is called LDDMM: Large Deformation Diffeomorphic Metric Mapping

Deformations are built by integrating a time-varying vector fieldvt(x) overt∈[0,1]wherev_t(x) represents the instantaneous velocity of any pointx at time t(and no more a displacement vector !)

(53)

Diffeomorphism

Callingφ_t(x) the position of a point at time twhich was located in x at time t= 0, its evolution is given by: ^∂φ_∂t^t^(x) =vt(φt(x))with φ₀(x) =x

Integrating ^∂φ_∂t^t^(x) =v_t(φ_t(x))between t∈[0,1]produces a flow of

(54)

Diffeomorphism

(55)

Diffeomorphism

Figure 6:Image taken from T. Mansi - MICCAI - 2009

(56)

Diffeomorphism

http://www.deformetrica.org/

(57)

Diffeomorphism

The flow of diffeomorphisms produces a dense deformation of the entire 3D space. We know how to deform every point in the space.

The last diffeomorphism is parametrised by the initial velocityv0

From a mathematical point of view, to register a source image or meshI to a target image or mesh J we minimize:

arg min

v0

D(φ1(I), T) +γReg(v0) (33) whereDis a data term, Regis a regularization term and γ their trade-off. We use an optimization scheme (e.g. gradient descent) to estimate the optimal deformation parameters v0.

(58)

References

1 J. Modersitzki (2004). Numerical methods for image registration.

Oxford university press

2 J. V. Hajnal, D. L.G. Hill, D. J. Hawkes (2001). Medical image registration. CRC press

3 G. Wolberg (1990). Digital Image Warping. IEEE

4 The Matrix Cookbook

5 S. Umeyama (1991). Least-squares estimation of transformation parameters.... IEEE TPAMI

6 S. Durrleman et al. (2014). Morphometry of anatomical shape complexes ... . NeuroImage.

7 J. Ashburner (2007). A fast diffeomorphic image registration algorithm. NeuroImage

8 J. P. W. Pluim (2003). Mutual information based registration of medical images: a survey. IEEE TMI