Inverse Problems

Gabriel Peyré

www.numerical-tours.com

Low Complexity Regularization of

Inverse Problems

Cours #1

Inverse Problems

Overview of the Course

• Course #1: Inverse Problems

• Course #2: Recovery Guarantees

• Course #3: Proximal Splitting Methods

Overview

• Inverse Problems

• Compressed Sensing

• Sparsity and L1 Regularization

y = x₀ + w 2 R^P

Inverse Problems

Recovering x₀ R^N from noisy observations

y = x₀ + w 2 R^P

Examples: Inpainting, super-resolution, . . .

Inverse Problems

Recovering x₀ R^N from noisy observations

x₀

x = (p_✓k)_16k6K

Inverse Problems in Medical Imaging

Magnetic resonance imaging (MRI):

x = (p_✓k)_16k6K

x = ( ˆf(!))_!2⌦

Inverse Problems in Medical Imaging

ˆ x

Magnetic resonance imaging (MRI):

Other examples: MEG, EEG, . . .

x = (p_✓k)_16k6K

x = ( ˆf(!))_!2⌦

Inverse Problems in Medical Imaging

ˆ x

Inverse Problem Regularization

observations y parameter

Estimator: x(y) depends only on Observations: y = x₀ + w 2 R^P .

Inverse Problem Regularization

observations y parameter

Example: variational methods

Estimator: x(y) depends only on

x(y) 2 argmin

x2R^N

1

2||y x||² + J(x)

Data fidelity Regularity Observations: y = x₀ + w 2 R^P .

J(x₀)

Regularity of x₀

Inverse Problem Regularization

observations y parameter

Example: variational methods

Estimator: x(y) depends only on

x(y) 2 argmin

x2R^N

1

2||y x||² + J(x)

Data fidelity Regularity Observations: y = x₀ + w 2 R^P .

Choice of : tradeoff

||w||

Noise level

J(x₀)

Regularity of x₀

x^? 2 argmin

x2R^Q, x=y

J(x)

Inverse Problem Regularization

observations y parameter

Example: variational methods

Estimator: x(y) depends only on

x(y) 2 argmin

x2R^N

1

2||y x||² + J(x)

Data fidelity Regularity Observations: y = x₀ + w 2 R^P .

No noise: 0⁺, minimize Choice of : tradeoff

||w||

Noise level

J(x₀)

Regularity of x₀

This course: Performance analysis.

Fast computational scheme.

x^? 2 argmin

x2R^Q, x=y

J(x)

Inverse Problem Regularization

observations y parameter

Example: variational methods

Estimator: x(y) depends only on

x(y) 2 argmin

x2R^N

1

2||y x||² + J(x)

Data fidelity Regularity Observations: y = x₀ + w 2 R^P .

No noise: 0⁺, minimize Choice of : tradeoff

||w||

Noise level

Overview

• Inverse Problems

• Compressed Sensing

• Sparsity and L1 Regularization

˜ x₀

Compressed Sensing

[Rice Univ.]

P measures N micro-mirrors

˜ x₀

y[i] = hx₀, '_ii

Compressed Sensing

[Rice Univ.]

P/N = 0.16 P/N = 0.02 P/N = 1

P measures N micro-mirrors

˜ x₀

y[i] = hx₀, '_ii

Compressed Sensing

[Rice Univ.]

Physical hardware resolution limit: target resolution f R^N. micro

mirrors array

resolution CS hardware

˜

x₀ 2 L² x₀ 2 R^{N y 2 RP}

CS Acquisition Model

CS is about designing hardware: input signals ˜f L²(R²).

Physical hardware resolution limit: target resolution f R^N. micro

mirrors array

resolution

CS hardware

,

.. .

Operator

˜

x₀ 2 L² x₀ 2 R^{N y 2 RP}

CS Acquisition Model

CS is about designing hardware: input signals ˜f L²(R²).

, ,

x₀

Overview

• Inverse Problems

• Compressed Sensing

• Sparsity and L1 Regularization

Q

N Dictionary = ( _m)_m R^{Q N} , N Q.

Redundant Dictionaries

m = e^{i ·, m}

frequency Fourier:

Q

N Dictionary = ( _m)_m R^{Q N} , N Q.

Redundant Dictionaries

Wavelets:

m = (2 ^jR x n)

m = (j, , n)

scale

orientation

position

m = e^{i ·, m}

frequency Fourier:

Q

N Dictionary = ( _m)_m R^{Q N} , N Q.

Redundant Dictionaries

= 1 = 2

Wavelets:

m = (2 ^jR x n)

m = (j, , n)

scale

orientation

position

m = e^{i ·, m}

frequency Fourier:

DCT, Curvelets, bandlets, . . .

Q

N Dictionary = ( _m)_m R^{Q N} , N Q.

Redundant Dictionaries

= 1 = 2

Synthesis: f = _m x_{m m} = x.

x

f

Image f = x Coefficients x

Wavelets:

m = (2 ^jR x n)

m = (j, , n)

scale

orientation

position

m = e^{i ·, m}

frequency Fourier:

DCT, Curvelets, bandlets, . . .

Q

N Dictionary = ( _m)_m R^{Q N} , N Q.

Redundant Dictionaries

=

= 1 = 2

Ideal sparsity: for most m, x_m = 0.

J₀(x) = # {m \ x_m = 0}

Sparse Priors

Image f₀ Coefficients x

Ideal sparsity: for most m, x_m = 0.

J₀(x) = # {m \ x_m = 0}

Sparse approximation: f = x where argmin

x2R^N ||f₀ x||² + T ²J₀(x)

Sparse Priors

Image f₀ Coefficients x

Ideal sparsity: for most m, x_m = 0.

J₀(x) = # {m \ x_m = 0}

Sparse approximation: f = x where

Orthogonal : = = Id_N

x_m = f₀, _m if | f₀, _m | > T,

0 otherwise. S_T

f = S_T (f₀) argmin

x2R^N ||f₀ x||² + T ²J₀(x)

Sparse Priors

Image f₀ Coefficients x

Ideal sparsity: for most m, x_m = 0.

J₀(x) = # {m \ x_m = 0}

Sparse approximation: f = x where

Orthogonal : = = Id_N

x_m = f₀, _m if | f₀, _m | > T,

0 otherwise. S_T

Non-orthogonal : NP-hard.

f = S_T (f₀) argmin

x2R^N ||f₀ x||² + T ²J₀(x)

Sparse Priors

Image f₀ Coefficients x

Image with 2 pixels:

q = 0

J₀(x) = # {m \ x_m = 0}

J₀(x) = 0 null image.

J₀(x) = 1 sparse image.

J₀(x) = 2 non-sparse image.

x₂

Convex Relaxation: L1 Prior

x₁

Image with 2 pixels:

q = 0 q = 1/2 q = 1 q = 3/2 q = 2

J_q(x) =

m

|x_m|^q

J₀(x) = # {m \ x_m = 0}

J₀(x) = 0 null image.

J₀(x) = 1 sparse image.

J₀(x) = 2 non-sparse image.

x₂

Convex Relaxation: L1 Prior

q priors: (convex for q 1)

x₁

Image with 2 pixels:

q = 0 q = 1/2 q = 1 q = 3/2 q = 2

J_q(x) =

m

|x_m|^q

J₀(x) = # {m \ x_m = 0}

J₀(x) = 0 null image.

J₀(x) = 1 sparse image.

J₀(x) = 2 non-sparse image.

x₂

J₁(x) =

m

|x_m|

Convex Relaxation: L1 Prior

Sparse ¹ prior:

q priors: (convex for q 1)

x₁

L1 Regularization

coefficients x₀ R^N

L1 Regularization

coefficientsx₀ R^{N f0 =}image^x0 R^Q

L1 Regularization

observations w

coefficients image

K

x₀ R^{N f0 = x0} R^Q y = Kf₀ + w R^P

L1 Regularization

observations

= K ⇥ ⇥ R^{P N} w

coefficients image

K

x₀ R^{N f0 = x0} R^Q y = Kf₀ + w R^P

Fidelity Regularization

xminR^N

1

2||y x||² + ||x||¹

L1 Regularization

Sparse recovery: f = x where x solves

observations

= K ⇥ ⇥ R^{P N} w

coefficients image

K

x₀ R^{N f0 = x0} R^Q y = Kf₀ + w R^P

x argmin

x=y m

|x_m| x

x = y

Noiseless Sparse Regularization

Noiseless measurements: y = x₀

x argmin

x=y m

|x_m| x

x = y

x argmin

x=y m

|x_m|²

Noiseless Sparse Regularization

x

x = y Noiseless measurements: y = x₀

x argmin

x=y m

|x_m| x

x = y

x argmin

x=y m

|x_m|²

Noiseless Sparse Regularization

Convex linear program.

Interior points, cf. [Chen, Donoho, Saunders] “basis pursuit”.

Douglas-Rachford splitting, see [Combettes, Pesquet].

x

x = y Noiseless measurements: y = x₀

Regularization Data fidelity

y = x₀ + w Noisy measurements:

x argmin

x R^Q

1

2||y x||² + ||x||¹

Noisy Sparse Regularization

Regularization

Data fidelity Equivalence

|| x =

y||

y = x₀ + w Noisy measurements:

x argmin

x R^Q

1

2||y x||² + ||x||¹

x argmin

|| x y|| ||x||¹

Noisy Sparse Regularization

x

Iterative soft thresholding

Forward-backward splitting

Algorithms:

Regularization

Data fidelity Equivalence

|| x =

y||

y = x₀ + w Noisy measurements:

x argmin

x R^Q

1

2||y x||² + ||x||¹

x argmin

|| x y|| ||x||¹

Noisy Sparse Regularization

Nesterov multi-steps schemes.

see [Daubechies et al], [Pesquet et al], etc

x

K

y = Kf₀ + w Measures:

(Kf)_i =

⇢ 0 if i 2 ⌦, f_i if i /2 ⌦.

Inpainting Problem