Septemberthe6 ,2012 Journ´eeGDRRobotiqueMatthieuGeist Anoverviewof ‘ -regularizationforvaluefunctionapproximation

Introduction Background Algorithms Summary

An overview of ` ₁ -regularization for value function approximation

Journ ´ee GDR Robotique

Matthieu Geist

(Sup ´elec, IMS-MaLIS Research Group) matthieu.geist@supelec.fr

September the 6

, 2012

Matthieu Geist Regularizing LSTD 1/26

Introduction Background Algorithms Summary

MDP Setting

1

Introduction MDP Setting

2

Background LSTD

`

-regularization

3

Algorithms Lasso-TD

`

-PBR

`

-LSTD Dantzig LSTD

4

Summary

Introduction Background Algorithms Summary

MDP Setting

(Bertsekas & Tsitsiklis, 1996; Sutton & Barto, 1998)

Controller : a

= π(s

) Reward : r

= r (s

, a

)

System dynamics : s

∼ P(.|s

, a

)

Goal : Given a policy π : S → A compute its value v

(s) = E

"

X

t=0

γ

r (s

, π(s

))

s

= s, π

#

(0 < γ < 1)

Matthieu Geist Regularizing LSTD 3/26

Introduction Background Algorithms Summary

MDP Setting

The value function v satisfies the Bellman equation v = r + γ Pv ⇔ v = T v

We look for ˆ v(s) = P

j=1

θ

φ

(s) or v ˆ = Φθ, where Φ =







φ(s

)

.. . φ(s

)





 = (φ

. . . φ

) and θ =





 θ

.. . θ







T is only known through samples (s

, r

, s

)

where s

∼ µ :

Φ = ˜





 φ(s

)

.. . φ(s

)





 , Φ ˜

=





 φ(s

)

.. . φ(s

)





 , ˜ r =





 r

.. . r





 ,

We consider the situation where n p

Introduction Background Algorithms Summary

LSTD

`₁-regularization

1

Introduction MDP Setting

2

Background LSTD

`

-regularization

3

Algorithms Lasso-TD

`

-PBR

`

-LSTD Dantzig LSTD

4

Summary

Matthieu Geist Regularizing LSTD 5/26

Introduction Background Algorithms Summary

LSTD

`₁-regularization

LSTD

One looks for v ˆ in the feature space satisfying v ˆ = Π

T ˆ v.

The solution v ˆ = Φθ

can be characterized as ( ω

= arg min

kr + γPΦθ − Φωk

θ

= arg min

kΦθ − Φω

k

and approximated by its empirical counterpart :

( ω

= arg min

k˜ r + γ Φ ˜

θ − Φωk ˜

θ

= arg min

k Φθ ˜ − Φω ˜

k

Introduction Background Algorithms Summary

LSTD

`₁-regularization

LSTD (alternate writing)

We look for v ˆ = Φθ

that solves v ˆ = Π

T v ˆ . It is known that Π

= Φ(Φ

D

Φ)

Φ

D

, and (after algebra) one can show that θ

= A

b (Linear system of size p) ⇔ θ

= arg min

θ

kAθ − bk

where

A = Φ

D

(I − γP)Φ b = Φ

D

r

can be approximated through their empirical counterpart : A ˜ = 1

n Φ ˜

( ˜ Φ − γ Φ ˜

) b ˜ = 1

n Φ ˜

˜ r

Matthieu Geist Regularizing LSTD 7/26

Introduction Background Algorithms Summary

LSTD

`₁-regularization

LSTD (another alternate writing)

Recall the first writing :

( ω

= arg min

k˜ r + γ Φ ˜

θ − Φωk ˜

θ

= arg min

k Φθ ˜ − Φω ˜

k

.

The second equation is minimized to zero for θ = ω

(the fixed point of Π

T exists).

Therefore, the second equation rewrites θ

= arg min

ω

k˜ r + γ Φ ˜

θ

− Φωk ˜

(fixed-point equation, θ

in both sides).

Introduction Background Algorithms Summary

LSTD

`₁-regularization

Least-squares regression and ` ₂ -penalization

regression (γ = 0), find θ such that ˜ r ≈ Φθ ˜ ; natural (LS) objective function :

θ

= arg min

θ

k˜ r − Φθk ˜

analytical solution : θ

= ( ˜ Φ

Φ) ˜

Φ˜ ˜ r this is an empirical projection : Φθ ˜

= ˜ Π

˜ r , (so Π ˜

= ˜ Φ( ˜ Φ

Φ) ˜

Φ ˜

)

`

-penalized LS :

θ

= arg min

θ

k˜ r − Φθk ˜

+ λkθk

analytical solution : θ

= ( ˜ Φ

Φ + ˜ λI)

Φ˜ ˜ r

this is a penalized empirical projection : Φθ ˜

= ˜ Π

˜ r

Matthieu Geist Regularizing LSTD 9/26

Introduction Background Algorithms Summary

LSTD

`₁-regularization

` ₁ -penalization

Lasso (Tibshirani, 1996) (`

-penalized LS) : θ

= arg min

θ

k˜ r − Φθk ˜

+ λkθk

promotes sparsity

no analytical solution, but efficient computation through the regularization path (θ

piecewise linear) (Efron et al. , 2004) this is a penalized empirical projection : Φθ ˜

= ˜ Π

˜ r Dantzig Selector (Candes & Tao, 2007) :

θ

= arg min

θ

kθk

subject to k Φ ˜

(˜ r − Φθ)k ˜

≤ λ promotes sparsity

this is actually a linear program

many (Lasso) extensions, but lets keep things simple...

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

1

Introduction MDP Setting

2

Background LSTD

`

-regularization

3

Algorithms Lasso-TD

`

-PBR

`

-LSTD Dantzig LSTD

4

Summary

Matthieu Geist Regularizing LSTD 11/26

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Algorithm

LSTD (3

writing) :

θ

= arg min

ω

k˜ r + γ Φ ˜

θ

− Φωk ˜

Lasso-TD (Kolter & Ng, 2009) :

θ

= arg min

ω

k˜ r + γ Φ ˜

θ

− Φωk ˜

+ λkωk

(not a standard lasso problem)

alternative writing :

( ω

= arg min

k˜ r + γ Φ ˜

θ − Φωk ˜

+ λkωk

θ

= arg min

k Φθ ˜ − Φω ˜

k

another alternative writing : v ˜

= ˜ Π

T ˆ ˜ v

(with ˜ v

= ˜ Φθ

)

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Properties

not a standard lasso ⇒ originally requires an adhoc solver can be framed as an LCP (Linear Complementary

Problem) (Johns et al. , 2010) : more generic solvers requires A ˜ to be a P-matrix (not necessarily true in the off policy case : Π ˜

T ˆ may have zero or multiple fixed points) sparsity oracle inequality (Ghavamzadeh et al. , 2011) :

inf

kv − ˜ v

k

≤ 1 1 − γ inf

θ

(

kv − ˆ v

k

+ O

r kθk

ln p n

!)

remark : how to choose λ ? (no cross-validation !)

Matthieu Geist Regularizing LSTD 13/26

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Algorithm (Geist & Scherrer, 2011; Hoffman et al. , 2011)

lasso-td :

( ω

= arg min

k˜ r + γ Φ ˜

θ − Φωk ˜

+ λkωk

θ

= arg min

k Φθ ˜ − Φω ˜

k

why not :

( ω

= arg min

k˜ r + γ Φ ˜

θ − Φωk ˜

θ

= arg min

k Φθ ˜ − Φω ˜

k

+ λkθk

? (this is `

-PBR)

alternative writing : θ

= arg min

θ

k Π ˜

(˜ v

− T ˆ v ˜

)k

+ λkθk

(recall v ˜

= ˜ Φθ)

(`

-penalized Projected Bellman Residual)

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Properties

`

-PBR is a standard Lasso problem : θ

= arg min

θ

k˜ y − Ψθk ˜

+λkθk

, y ˜ = ˜ Π

˜ r , Ψ = ˜ ˜ Φ−γ Π ˜

Φ ˜

(easy extension to other penalization)

weaker conditions than Lasso-TD (off-policy is fine) but huge computational cost if p n (projection in O(p

)) no finite sample analysis

Matthieu Geist Regularizing LSTD 15/26

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Algorithm

2

formulation of LSTD : Aθ ˜

= ˜ b, A ˜ = 1

n Φ ˜

( ˜ Φ − γ Φ ˜

), ˜ b = 1 n Φ ˜

˜ r equivalently :

θ

= arg min

θ

k Aθ ˜ − ˜ bk

`

-LSTD (Pires, 2011; Pires & Szepesv ´ari, 2012) : θ

= arg min

θ

k Aθ ˜ − bk ˜

+λkθk

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Properties

“standard” Lasso problem (use any solver) no problem in the off-policy case

built-in (theoretical) λ-selection scheme finite sample analysis :

inf

kAθ

− bk

≤ O kθ

k

r p

n ln 1 δ

!

w.p. 1 − δ;

(recall A = lim

A ˜ and b = lim

b, ˜ Aθ

= b)

Matthieu Geist Regularizing LSTD 17/26

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Algorithm

2

formulation of LSTD : θ

= arg min

θ

k Aθ ˜ − ˜ bk

Dantzig-LSTD (Geist et al. , 2012) :

θ

= arg min

θ

kθk

subject to k Aθ ˜ − ˜ bk

≤ λ.

this is a linear program :

u,θ∈

min

1

u subject to

( −u ≤ θ ≤ u

−λ1 ≤ Aθ ˜ − b ˜ ≤ λ1

Introduction Background Algorithms Summary

Lasso-TD

`₁-PBR

`₁-LSTD Dantzig LSTD

Properties

standard Linear Program

no problem in the off-policy case connexion to Lasso-TD :

k Aθ ˜

− bk ˜

≤ λ.

heuristic model selection scheme finite sample analysis :

inf

kAθ

− bk

≤ O kθ

k

r 1

n ln p δ

!

w.p. 1 − δ;

(recall A = lim

A ˜ and b = lim

b, ˜ Aθ

= b) empirically : Lasso-TD, D-LSTD ≥ `

-LSTD, `

-PBR

Matthieu Geist Regularizing LSTD 19/26

Introduction Background Algorithms Summary

1

Introduction MDP Setting

2

Background LSTD

`

-regularization

3

Algorithms Lasso-TD

`

-PBR

`

-LSTD Dantzig LSTD

4

Summary

Introduction Background Algorithms Summary

ω

= arg min

ω

k R ˜ + γ Φ ˜

θ − Φωk ˜

+ λ

kωk

θ

= arg min

θ

k Φθ ˜ − Φω ˜

k

+ λ

kθk

LSTD : v ˆ

= ˜ Π

T ˆ ˆ v

Matthieu Geist Regularizing LSTD 21/26

Introduction Background Algorithms Summary

ω

= arg min

ω

k R ˜ + γ Φ ˜

θ − Φωk ˜

+λ

kωk

θ

= arg min

θ

k Φθ ˜ − Φω ˜

k

+λ

kθk

`

-LSTD (Farahmand et al. , 2008) : θ

= arg min kˆ v

1,2

− Π ˜

T ˆ v ˆ

1,2

k

+ λ

kθk

Introduction Background Algorithms Summary

ω

= arg min

ω

k R ˜ + γ Φ ˜

θ − Φωk ˜

+λ

kωk

θ

= arg min

θ

k Φθ ˜ − Φω ˜

k

+ λ

kθk

Lasso-TD : v

1

= ˜ Π

T ˆ v

1

Matthieu Geist Regularizing LSTD 21/26

Introduction Background Algorithms Summary

ω

= arg min

ω

k R ˜ + γ Φ ˜

θ − Φωk ˜

+λ

kωk

θ

= arg min

θ

k Φθ ˜ − Φω ˜

k

+λ

kθk

`

-PBR : θ

= arg min kˆ v

1,2

− Π ˜

T ˆ v ˆ

1,2

k

+ λ

kθk

Introduction Background Algorithms Summary

based on the residual Aθ ˜ − b ˜

`

-LSTD :

θ

= arg min

θ

kθk

subject to k Aθ ˜ − bk ˜

≤ λ Dantzig-LSTD :

θ

= arg min

θ

kθk

subject to k Aθ ˜ − bk ˜

≤ λ

Matthieu Geist Regularizing LSTD 22/26

Introduction Background Algorithms Summary

Thank you for your attention !

Questions ?

Final Remark. This talk focused on projected-fixed-point-based

regularization, but it is also possible to consider residual approaches,

with a potential bias problem. The first (as far as I know) algorithm to

use `

-penalization for value function estimation is actually a (biased)

residual approach (Loth et al. , 2007).

Introduction Background Algorithms Summary

References I

Bertsekas, Dimitri P., & Tsitsiklis, John N. 1996.

Neuro-Dynamic Programming.

Athena Scientific.

Bradtke, S. J., & Barto, A. G. 1996.

Linear Least-Squares algorithms for temporal difference learning.

Machine Learning,22, 33–57.

Candes, E., & Tao, T. 2007.

The Dantzig selector : statistical estimation when p is much larger than n.

Annals of Statistics,35(6), 2313–2351.

Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. 2004.

Least Angle Regression.

Annals of Statistics,32(2), 407–499.

Farahmand, A., Ghavamzadeh, M., Szepesv ´ari, C., & Mannor, S. 2008.

Regularized Policy Iteration.

In : Proc. of NIPS 21.

Geist, M., & Scherrer, B. 2011.

`₁-penalized projected Bellman residual.

In : Proc. of EWRL 9.

Geist, Matthieu, Scherrer, Bruno, Lazaric, Alessandro, & Ghavamzadeh, Mohammad. 2012.

A Dantzig Selector Approach to Temporal Difference Learning.

In : International Conference on Machine Learning (ICML).

Matthieu Geist Regularizing LSTD 24/26

Introduction Background Algorithms Summary

References II

Ghavamzadeh, M., Lazaric, A., Munos, R., & Hoffman, M. 2011.

Finite-Sample Analysis of Lasso-TD.

In : Proc. of ICML.

Hoffman, M. W., Lazaric, A., Ghavamzadeh, M., & Munos, R. 2011.

Regularized Least Squares Temporal Difference learning with nested`<sub>2</sub>and`₁penalization.

In : Proc. of EWRL 9.

Johns, J., Painter-Wakefield, C., & Parr, R. 2010.

Linear Complementarity for Regularized Policy Evaluation and Improvement.

In : Proc. of NIPS 23.

Kolter, J. Z., & Ng, A. Y. 2009.

Regularization and Feature Selection in Least-Squares Temporal Difference Learning.

In : Proc. of ICML.

Loth, Manuel, Davy, Manuel, & Preux, Philippe. 2007.

Sparse Temporal Difference Learning using LASSO.

In : IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning.

Pires, B. A. 2011.

Statistical analysis of`₁-penalized linear estimation with applications.

M.Phil. thesis, University of Alberta.

Pires, Bernardo A., & Szepesv ´ari, Csaba. 2012.

Statistical linear estimation with penalized estimators : an application to reinforcement learning.

In : Proceedings of the 29th International Conference on Machine Learning (ICML 2012).

Introduction Background Algorithms Summary

References III

Sutton, R. S., & Barto, A. G. 1998.

Reinforcement Learning : an Introduction.

The MIT Press.

Tibshirani, R. 1996.

Regression Shrinkage and Selection via the Lasso.

Journal of the Royal Statistical Society,58(1), 267–288.

Matthieu Geist Regularizing LSTD 26/26