Supplement to "Robust linear least squares regression"

HAL Id: hal-00624459

https://hal.archives-ouvertes.fr/hal-00624459

Submitted on 17 Sep 2011

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Supplement to ”Robust linear least squares regression”

Jean-Yves Audibert, Olivier Catoni

To cite this version:

Jean-Yves Audibert, Olivier Catoni. Supplement to ”Robust linear least squares regression”. Annals of Statistics, Institute of Mathematical Statistics, 2011, 39 (5), 19 p. �10.1214/11-AOS918SUPP�.

�hal-00624459�

“ROBUST LINEAR LEAST SQUARES REGRESSION”

By Jean-Yves Audibert

, and Olivier Catoni

This supplementary material provides the proofs of Theorems 2.1, 2.2 and 3.1 of the article “Robust linear least squares regression”.

CONTENTS

1 Proofs of Theorems 2.1 and 2.2 . . . . 1

1.1 Proof of Theorem 2.1 . . . . 8

1.2 Proof of Theorem 2.2 . . . . 9

2 Proof of Theorem 3.1 . . . . 11

References . . . . 19 1. Proofs of Theorems 2.1 and 2.2. To shorten the formulae, we will write X for ϕ(X), which is equivalent to considering without loss of generality that the input space is R

and that the functions ϕ

, . . . ,ϕ

are the coordinate functions. Therefore, the function f

maps an input x to h θ, x i . With a slight abuse of notation, R(θ) will denote the risk of this prediction function.

Let us first assume that the matrix Q

= Q + λI is positive definite. This indeed does not restrict the generality of our study, even in the case when λ = 0, as we will discuss later (Remark 1.1).

Consider the change of coordinates X = Q

X.

Let us introduce

R(θ) = E

( h θ, X i − Y )

,

∗

Universit´e Paris-Est, LIGM, Imagine, 6 avenue Blaise Pascal, 77455 Marne-la-Vall´ee, France, E-mail:

†

CNRS/´ Ecole Normale Sup´erieure/INRIA, LIENS, Sierra – UMR 8548, 23 avenue d’Italie, 75214 Paris cedex 13, France.

‡

Ecole Normale Sup´erieure, CNRS – UMR 8553, D´epartement de Math´ematiques et ´ Applications, 45 rue d’Ulm, 75230 Paris cedex 05, France, E-mail:

§

INRIA Paris-Rocquencourt - CLASSIC team.

AMS 2000 subject classifications: 62J05, 62J07

Keywords and phrases: Linear regression, Generalization error, Shrinkage, PAC- Bayesian theorems, Risk bounds, Robust statistics, Resistant estimators, Gibbs posterior distributions, Randomized estimators, Statistical learning theory

1

so that

R(Q

θ) = R(θ) = E

( h θ, X i − Y )

. Let

Θ =

Q

θ; θ ∈ Θ . Consider

r(θ) = 1 n

X

h θ, X

i − Y

, (1.1)

r(θ) = 1 n

X

h θ, X

i − Y

, (1.2)

θ

= arg min

θ∈Θ

R(θ) + λ k Q

θ k

, (1.3)

θ ˆ ∈ arg min

θ∈Θ

r(θ) + λ k θ k

, (1.4)

θ

= Q

θ ˆ ∈ arg min

θ∈Θ

r(θ) + λ k Q

θ k

. (1.5)

For α > 0, let us introduce the notation W

(θ) = α n

h θ, X

i − Y

− h θ

, X

i − Y

o , W (θ) = α n

h θ, X i − Y

− h θ

, X i − Y

o .

For any θ

∈ R

and β > 0, let us consider the Gaussian distribution centered at θ

ρ

(dθ) = β

2π

exp

− β

2 k θ − θ

k

dθ.

Lemma 1.1. For any η > 0 and α > 0, with probability at least 1 − exp( − η), for any θ

∈ R

,

− n Z

log n 1 − E

W (θ) + E

W (θ)

/2 o

ρ

(dθ)

≤ − X

Z log n

1 − W

(θ) + W

(θ)

/2 o

ρ

(dθ) + K (ρ

, ρ

) + η, where K (ρ

, ρ

) is the Kullback-Leibler divergence function :

K (ρ

, ρ

) = Z

log dρ

dρ

(θ)

ρ

(dθ).

Proof. Since E

Z

ρ

(dθ) Y

1 − W

(θ) + W

(θ)

/2 1 − E

W (θ) + E

W (θ)

/2

!

≤ 1, with probability at least 1 − exp( − η)

log Z

ρ

(dθ) Y

1 − W

(θ) + W

(θ)

/2 1 − E

W (θ) + E

W (θ)

/2

!

≤ η.

We conclude the proof using the convex inequality (see [2], [3, Proposition 1.4.2] or [1, page 159])

log R

ρ

(dθ) exp h(θ)

≥ R

ρ

(dθ)h(θ) − K (ρ

, ρ

).

Let us compute some useful quantities K (ρ

, ρ

) = β

2 k θ

− θ

k

, (1.6)

R ρ

(dθ) W (θ)

= α R

ρ

(dθ) h θ − θ

, X i

+ W (θ

)

= W (θ

) + α k X k

β , (1.7)

R ρ

(dθ) h θ − θ

, X i

= 3 k X k

β

, (1.8)

(1.9) R

ρ

(dθ)

W (θ)

= α

R

ρ

(dθ) h θ − θ

, X i

h θ + θ

, X i − 2Y

= α

R

ρ

(dθ) h

h θ − θ

+ θ

− θ

, X i h θ − θ

+ θ

+ θ

, X i − 2Y i

= R

ρ

(dθ) h

α h θ − θ

, X i

+ 2α h θ − θ

, X i h θ

, X i − Y

+ W (θ

) i

= R

ρ

(dθ) h

α

h θ − θ

, X i

+ 4α

h θ − θ

, X i

h θ

, X i − Y

+ W (θ

)

+ 2α h θ − θ

, X i

W (θ

) i

= 3α

k X k

β

+ 2α k X k

β

h 2α h θ

, X i − Y

+ W (θ

) i

+ W (θ

)

. Using the fact that

2α h θ

, X i − Y

+ W (θ

) = 2α h θ

, X i − Y

+ 3W (θ

),

and that for any real numbers a and b, 6ab ≤ 9a

+ b

, we get

Lemma 1.2.

R ρ

(dθ) W (θ)

= W (θ

) + α k X k

β , (1.10)

R ρ

(dθ)

W (θ)

= W (θ

)

+ 2α k X k

β

h 2α h θ

, X i − Y

+ 3W (θ

) i + 3α

k X k

β

(1.11)

≤ 10W (θ

)

+ 4α

k X k

β h θ

, X i − Y

+ 4α

k X k

β

, (1.12)

and the same holds true when W is replaced with W

and (X, Y ) with (X

, Y

).

Another important thing to realize is that E

k X k

= E

Tr X X

= E

Tr Q

XX

Q

= E

Tr Q

XX

= Tr

Q

E (XX

)

= Tr Q

(Q

− λI)

= d − λ Tr(Q

) = D . (1.13)

We can weaken Lemma 1.1 (page 2) noticing that for any real number x, x − x

2 ≤ − log

1 − x + x

2

= log

1 + x + x

/2 1 + x

/4

≤ log

1 + x + x

2

≤ x + x

2 . We obtain with probability at least 1 − exp( − η)

n E

W (θ

) + nα

β E k X k

− 5n E

W (θ

)

− E

( 2nα

k X k

β h θ

, X i − Y

+ 2nα

k X k

β

)

≤ X

i=1

(

W

(θ

) + 5W

(θ

)

+ α k X

k

β + 2α

k X

k

β h θ

, X

i − Y

+ 2α

k X

k

β

)

+ β

2 k θ

− θ

k

+ η.

Noticing that for any real numbers a and b, 4ab ≤ a

+ 4b

, we can then bound

α

W (θ

)

= h θ

− θ

, X i

h θ

+ θ

, X i − 2Y

= h θ

− θ

, X i

h

h θ

− θ

, X i + 2 h θ

, X i − Y i

= h θ

− θ

, X i

+ 4 h θ

− θ

, X i

h θ

, X i − Y + 4 h θ

− θ

, X i

h θ

, X i − Y

≤ 2 h θ

− θ

, X i

+ 8 h θ

− θ

, X i

h θ

, X i − Y

. Theorem 1.3. Let us put

D b = 1 n

X

k X

k

(let us remind that D = E k X k

from (1.13)), B

= 2 E h

k X k

h θ

, X i − Y

i , B b

= 2

n X

h k X

k

h θ

, X

i − Y

i , B

= 2 E h

k X k

i , B b

= 2

n X

k X

k

, B

= 40 sup n

E

h u, X i

h θ

, X i − Y

: u ∈ R

, k u k = 1 o , B b

= sup

40 n

X

h u, X

i

h θ

, X

i − Y

: u ∈ R

, k u k = 1 o , B

= 10 sup n

E h

h u, X i

i

: u ∈ R

, k u k = 1 o , B b

= sup

10 n

X

h u, X

i

: u ∈ R

, k u k = 1

. With probability at least 1 − exp( − η), for any θ

∈ R

,

n E

W (θ

)

−

nα

(B

+ B b

) + β 2

k θ

− θ

k

− nα

(B

+ B b

) k θ

− θ

k

≤ X

W

(θ

) + nα

β ( D b − D) + nα

β (B

+ B b

) + nα

β

(B

+ B b

) + η.

Let us now assume that θ

∈ Θ and let us use the fact that Θ is a convex set and that θ

= arg min

R(θ) + λ k Q

θ k

. Introduce θ

= arg min

R(θ) + λ k Q

θ k

. As we have

R(θ) + λ k Q

θ k

= k θ − θ

k

+ R(θ

) + λ k Q

θ

k

,

the vector θ

is uniquely defined as the projection of θ

on Θ for the Eu- clidean distance, and for any θ

∈ Θ

(1.14) α

E

W (θ

)

+ λ k Q

θ

k

− λ k Q

θ

k

= R(θ

) − R(θ

) + λ k Q

θ

k

− λ k Q

θ

k

= k θ

− θ

k

− k θ

− θ

k

= k θ

− θ

k

+ 2 h θ

− θ

, θ

− θ

i ≥ k θ

− θ

k

. This and the inequality

α

X

i=1

W

(θ

) + nλ k Q

θ

k

− nλ k Q

θ

k

≤ 0 leads to the following result.

Theorem 1.4. With probability at least 1 − exp( − η), R(ˆ θ) + λ k θ ˆ k

− inf

θ∈Θ

R(θ) + λ k θ k

= α

E

W (θ

)

+ λ k Q

θ

k

− λ k Q

θ

k

is not greater than the smallest positive non degenerate root of the following polynomial equation as soon as it has one

n 1 −

α(B

+ B b

) +

o

x − α(B

+ B b

)x

= 1

β max( D b − D, 0) + α

β (B

+ B b

) + α

β

(B

+ B b

) + η

nα .

Proof. Let us remark first that when the polynomial appearing in the

theorem has two distinct roots, they are of the same sign, due to the sign of

its constant coefficient. Let Ω be the event of probability at least 1 b − exp( − η)

described in Theorem 1.3 (page 5). For any realization of this event for which the polynomial described in Theorem 1.4 does not have two distinct positive roots, the statement of Theorem 1.4 is void, and therefore fulfilled. Let us consider now the case when the polynomial in question has two distinct positive roots x

< x

. Consider in this case the random (trivially nonempty) closed convex set

Θ = b n

θ ∈ Θ : R(θ) + λ k θ k

≤ inf

θ^′∈Θ

R(θ

) + λ k θ

k

+

o . Let θ

∈ arg min

r(θ) + λ k θ k

and θ

∈ arg min

r(θ) + λ k θ k

. We see from Theorem 1.3 that

(1.15) R(θ

) + λ k θ

k

< R(θ

) + λ k θ

k

+ x

+ x

2 ,

because it cannot be larger from the construction of Θ. On the other hand, b since Θ b ⊂ Θ, the line segment [θ

, θ

] is such that [θ

, θ

] ∩ Θ b ⊂ arg min

r(θ)+

λ k θ k

. We can therefore apply equation (1.15) to any point of [θ

, θ

] ∩ Θ, b which proves that [θ

, θ

] ∩ Θ is an open subset of [θ b

, θ

]. But it is also a closed subset by construction, and therefore, as it is non empty and [θ

, θ

] is connected, it proves that [θ

, θ

] ∩ Θ = [θ b

, θ

], and thus that θ

∈ Θ. This can be applied to any choice of b θ

∈ arg min

r(θ) + λ k θ k

and θ

∈ arg min

r(θ) + λ k θ k

, proving that arg min

r(θ) + λ k θ k

⊂ arg min

r(θ) + λ k θ k

and therefore that any θ

∈ arg min

r(θ) + λ k θ k

is such that

R(θ

) + λ k θ

k

≤ inf

θ∈Θ

R(θ) + λ k θ k

+ x

.

because the values between x

and x

are excluded by Theorem 1.3.

The actual convergence speed of the least squares estimator ˆ θ on Θ will depend on the speed of convergence of the “empirical bounds” B b

towards their expectations. We can rephrase the previous theorem in the following more practical way:

Theorem 1.5. Let η

, η

, . . . , η

be positive real numbers. With proba- bility at least

1 − P D > D b + η

− X

P B b

− B

> η

− exp( − η

), R(ˆ θ) + λ k θ ˆ k

− inf

R(θ) + λ k θ k

is smaller than the smallest non de-

generate positive root of

(1.16) n 1 −

α(2B

+ η

) +

o

x − α(2B

+ η

)x

= η

β + α

β (2B

+ η

) + α

β

(2B

+ η

) + η

nα , where we can optimize the values of α > 0 and β > 0, since this equation has non random coefficients. For example, taking for simplicity

α = 1

8B

+ 4η

, β = nα

2 , we obtain

x − 2B

+ η

4B

+ 2η

x

= 16η

(2B

+ η

)

n + 8B

+ 4η

n

+ 32(2B

+ η

)(2B

+ η

)

n

+ 8η

(2B

+ η

)

n .

1.1. Proof of Theorem 2.1. Let us now deduce Theorem 2.1 from Theo- rem 1.5. Let us first remark that with probability at least 1 − ε/2

D b ≤ D + r B

εn , because the variance of D b is less than B

2n . For a given ε > 0, let us take η

=

r B

εn , η

= B

, η

= B

, η

= B

and η

= B

. We get that R

(ˆ θ) − inf

R

(θ) is smaller than the smallest positive non degenerate root of

x − B

2B

x

= 48B

n

r B

nε + 12B

n + 288B

B

n

+ 24 log(3/ε)B

n ,

with probability at least 1 − 5 ε

6 − X

P B b

> B

+ η

.

According to the weak law of large numbers, there is n

such that for any n ≥ n

,

X

P B b

> B

+ η

≤ ε/6.

Thus, increasing n

and the constants to absorb the second order terms, we see that for some n

and any n ≥ n

, with probability at least 1 − ε, the excess risk is less than the smallest positive root of

x − B

2B

x

= 13B

n + 24 log(3/ε)B

n .

Now, as soon as ac < 1/4, the smallest positive root of x − ax

= c is 2c

1 + √

1 − 4ac . This means that for n large enough, with probability at least 1 − ε,

R

(ˆ θ) − inf

θ

R

(θ) ≤ 15B

n + 25 log(3/ε)B

n ,

which is precisely the statement of Theorem 2.1, up to some change of no- tation.

1.2. Proof of Theorem 2.2. Let us now weaken Theorem 1.4 in order to make a more explicit non asymptotic result and obtain Theorem 2.2. From now on, we will assume that λ = 0. We start by giving bounds on the quantity defined in Theorem 1.3 in terms of

B = sup

f∈span{ϕ1,...,ϕd}−{0}

k f k

/ E [f (X)]

. Since we have

k X k

= k Q

X k

≤ dB, we get

d b = 1 n

X

k X

k

≤ dB, B

= 2 E h

k X k

h θ

, X i − Y

i

≤ 2dB R(f

), B b

= 2

n X

i=1

h k X

k

h θ

, X

i − Y

i

≤ 2dB r(f

), B

= 2 E

h k X k

i

≤ 2d

B

, B b

= 2

n X

i=1

k X

k

≤ 2d

B

, B

= 40 sup n

E

h u, X i

h θ

, X i − Y

: u ∈ R

, k u k = 1 o

≤ 40B R(f

), B b

= sup

40 n

X

h u, X

i

h θ

, X

i − Y

: u ∈ R

, k u k = 1 o

≤ 40B r(f

),

B

= 10 sup n E h

h u, X i

i

: u ∈ R

, k u k = 1 o

≤ 10B

, B b

= sup

10 n

X

h u, X

i

: u ∈ R

, k u k = 1

≤ 10B

. Let us put

a

= 2dB + 4dBα[R(f

) + r(f

)] + η

αn + 16B

d

αn

, a

= 3/4 − 40αB[R(f

) + r(f

)],

and

a

= 20αB

.

Theorem 1.4 applied with β = nα/2 implies that with probability at least 1 − η the excess risk R( ˆ f

) − R(f

) is upper bounded by the smallest positive root of a

x − a

x

= a

as soon as a

> 4a

a

. In particular, setting ε = exp( − η) when (1.17) holds, we have

R( ˆ f

) − R(f

) ≤ 2a

a

+ p

a

− 4a

a

≤ 2a

a

. We conclude that

Theorem 1.6. For any α > 0 and ε > 0, with probability at least 1 − ε, if the inequality

(1.17) 80 (2 + 4α[R(f

) + r(f

)])Bd + log(ε

)

n +

4Bd n

!

<

3

4B − 40α[R(f

) + r(f

)]

holds, then we have (1.18)

R( ˆ f

) − R(f

) ≤ J (2 + 4α[R(f

) + r(f

)])Bd + log(ε

)

n +

4Bd n

! , where J = 8/(3α − 160α

B [R(f

) + r(f

)])

Now, the Bienaym´e-Chebyshev inequality implies P r(f

) − R(f

) ≥ t

≤ E r(f

) − R(f

)

t

≤ E [Y − f

(X)]

/nt

.

Under the finite moment assumption of Theorem 2.2, we obtain that for any ε ≥ 1/n, with probability at least 1 − ε,

r(f

) < R(f

) + p

E [Y − f

(X)]

. From Theorem 1.6 and a union bound, by taking

α =

80B[2R(f

) + p

E [Y − f

(X)]

, we get that with probability 1 − 2ε,

R( ˆ f

) − R(f

) ≤ J

B 3Bd + log(ε

)

n +

4Bd n

! ,

with J

= 640

2R(f

) + q E

[Y − f

(X)]

. This concludes the proof of Theorem 2.2.

Remark 1.1. Let us indicate now how to handle the case when Q is degenerate. Let us consider the linear subspace S of R

spanned by the eigenvectors of Q corresponding to positive eigenvalues. Then almost surely Span { X

, i = 1, . . . , n } ⊂ S. Indeed for any θ in the kernel of Q, E h θ, X i

= 0 implies that h θ, X i = 0 almost surely, and considering a basis of the ker- nel, we see that X ∈ S almost surely, S being orthogonal to the kernel of Q.

Thus we can restrict the problem to S, as soon as we choose θ ˆ ∈ span

X

, . . . , X

∩ arg min

θ

X

h θ, X

i − Y

,

or equivalently with the notation X = (ϕ

(X

))

and Y = [Y

]

, θ ˆ ∈ im X

∩ arg min

θ

k X θ − Y k

This proves that the results of this section apply to this special choice of the empirical least squares estimator. Since we have R

= ker X ⊕ im X

, this choice is unique. Finally, we also have that inequality (2.3) of the paper still holds by replacing d by rank(Q).

2. Proof of Theorem 3.1. We use the same notations as in Section 1.

We write X for ϕ(X), therefore, the function f

maps an input x to h θ, x i . We consider the change of coordinates

X = Q

X.

Thus, from (1.13), we have E k X k

= D. We will use R(θ) = E

( h θ, X i − Y )

, so that R(Q

θ) = E

( h θ, X i − Y )

= R(f

). Let Θ =

Q

θ; θ ∈ Θ , and consider

θ

= arg min

θ∈Θ

n

R(θ) + λ k Q

θ k

o . With these notations,

θ ˜ = Q

θ

, σ =

q E

h θ

, X i − Y

, χ = sup

u∈R^d

E h u, X i

E h u, X i

, κ = E k X k

E k X k

= E k X k

D ,

κ

= E

h θ

, X i − Y

σ

,

and T = k Θ k = max

θ,θ^′∈Θ

k θ − θ

k . For α > 0, we introduce

J

(θ) = h θ, X

i − Y

, J(θ) = h θ, X i − Y L

(θ) = α h θ, X

i − Y

, L(θ) = α h θ, X i − Y

W

(θ) = L

(θ) − L

(θ

), W (θ) = L(θ) − L(θ

), and

r

(θ, θ

) = λ

k Q

θ k

− k Q

θ

k

+ 1

nα X

i=1

ψ L(θ) − L(θ

) .

Let ¯ θ = Q

θ ˆ ∈ Θ. We have

(2.1) − r

(θ

, θ) = ¯ r

(¯ θ, θ

) ≤ max

θ¹∈Θ

r

(¯ θ, θ

) ≤ γ + max

θ¹∈Θ

r

(θ

, θ

),

where the quantity γ = max

θ¹∈Θ

r

(¯ θ, θ

) − inf

θ∈Θ

max

θ¹∈Θ

r

(θ, θ

) can be made arbitrary small by a proper choice of the estimator. Using an upper bound r

(θ

, θ

) that holds uniformly in θ

, we will control both left and right hand sides of (2.1).

To achieve this, we will upper bound (2.2) r

(θ

, θ

) = λ

k Q

θ

k

− k Q

θ

k

+ 1

nα X

i=1

ψ

− W

(θ

) by the expectation of a distribution depending on θ

of a quantity that does not depend on θ

, and then use the PAC-Bayesian argument to control this expectation uniformly in θ

. The distribution depending on θ

should therefore be taken such that for any θ

∈ Θ, its Kullback-Leibler divergence with respect to some fixed distribution is small (at least when θ

is close to θ

).

Let us start with the following result.

Lemma 2.1. Let f, g : R → R be two Lebesgue measurable functions such that f (x) ≤ g(x), x ∈ R . Let us assume that there exists h ∈ R such that x 7→ g(x) + hx

/2 is convex. Then for any probability distribution µ on the real line,

f Z

xµ(dx)

≤ Z

g(x)µ(dx) + min n

sup f − inf f, h

2 V ar(µ) o . Proof. Let us put x

= R

xµ(dx) The function x 7→ g(x) + h

2 (x − x

)

is convex. Thus, by Jensen’s inequality

f (x

) ≤ g(x

) ≤ Z

µ(dx)

g(x) + h

2 (x − x

)

= Z

g(x)µ(dx) + h

2 V ar(µ).

On the other hand

f(x

) ≤ sup f ≤ sup f + Z

g(x) − inf f µ(dx)

= Z

g(x)µ(dx) + sup f − inf f.

The lemma is a combination of these two inequalities.

The above lemma will be used with f = g = ψ, where ψ is the increasing influence function

ψ(x) =

 

 

 

 

 



− log(2), x ≤ − 1,

log(1 + x + x

/2), − 1 ≤ x ≤ 0,

− log(1 − x + x

/2), 0 ≤ x ≤ 1,

log(2), x ≥ 1.

Since we have for any x ∈ R

− log

1 − x + x

2

= log

1 + x +

1 +

< log

1 + x + x

2

, the function ψ satisfies for any x ∈ R

− log

1 − x + x

2

< ψ(x) < log

1 + x + x

2

. Moreover

ψ

(x) = 1 − x

1 − x +

, ψ

(x) = x(x − 2)

2 1 − x +

≥ − 2, 0 ≤ x ≤ 1, showing (by symmetry) that the function x 7→ ψ(x) + 2x

is convex on the real line.

For any θ

∈ R

and β > 0, we consider the Gaussian distribution with mean θ

and covariance β

I :

ρ

(dθ) = β

2π

exp

− β

2 k θ − θ

k

dθ.

From Lemmas 1.2 and 2.1 (with µ the distribution of − W

(θ) +

when θ is drawn from ρ

and for a fixed pair (X

, Y

)), we can see that

ψ

− W

(θ

)

= ψ Z

ρ

(dθ)

− W

(θ) + α k X

k

β

≤ Z

ρ

(dθ)ψ

− W

(θ) + α k X

k

β

+ min n

log(4), V ar

L

(θ) o . Let us compute

1

α

V ar

L

(θ)

= V ar

J

(θ) − J

(θ

)

= Z

ρ

(dθ)

J

(θ) − J

(θ

)

− k X

k

β

= Z

ρ

(dθ) h

h θ − θ

, X

i

+ 2 h θ − θ

, X

i J

(θ

) i

− k X

k

β

= 2 k X

k

β

+ 4L

(θ

) k X

k

αβ .

(2.3)

Let ξ ∈ (0, 1), and let us remark that L

(θ

) ≤ L

(θ)

ξ + α h θ − θ

, X

i

1 − ξ . We get

min n

log(4), V ar

L

(θ) o

= min

log(4), 4α k X

k

L

(θ

)

β + 2α

k X

k

β

≤ Z

ρ

(dθ) min

log(4), 4α k X

k

L

(θ)

βξ + 2α

k X

k

β

+ 4α

k X

k

h θ − θ

, X

i

β(1 − ξ)

≤ Z

ρ

(dθ) min

log(4), 4α k X

k

L

(θ)

βξ + 2α

k X

k

β

+ min

log(4), 4α

k X

k

β

(1 − ξ)

. Let us now put a = 3

log(4) < 2.17, b = a + a

log(4) < 8.7 and let us remark that

min

log(4), x + min

log(4), y

≤ log

1 + a min { log(4), x }

+ log(1 + ay)

≤ log 1 + ax + by

, x, y ∈ R

. Thus

min n

log(4), V ar

L

(θ) o

≤ Z

ρ

(dθ) log

1 + 4aα k X

k

L

(θ)

βξ + 2α

k X

k

β

a + 2b 1 − ξ

.

We can then remark that ψ(x) + log(1 + y) = log

exp[ψ(x)] + y exp[ψ(x)]

≤ log

exp[ψ(x)] + 2y

≤ log

1 + x + x

2 + 2y

, x ∈ R , y ∈ R

. Thus, putting c

= a + 2b

1 − ξ , we get

(2.4) ψ

− W

(θ

)

≤ Z

ρ

(dθ) log[A

(θ)], with

A

(θ) = 1 − W

(θ) + α k X

k

β + 1

2

− W

(θ) + α k X

k

β

+ 8aα k X

k

L

(θ)

βξ + 4c

α

k X

k

β

. Similarly, we define A(θ) by replacing (X

, Y

) by (X, Y ). Since we have

E

exp X

i=1

log[A

(θ)] − n log[ E A(θ)]

= 1,

from the usual PAC-Bayesian argument, we have with probability at least 1 − ε, for any θ

∈ R

,

Z

ρ

(dθ) X

i=1

log[A

(θ)]

− n Z

ρ

(dθ) log[A(θ)] ≤ K (ρ

, ρ

) + log(ε

)

≤ β k θ

− θ

k

2 + log(ε

).

From (2.2) and (2.4), with probability at least 1 − ε, for any θ

∈ R

, we get r

(θ

, θ

) ≤ 1

α log

1 + E Z

ρ

(dθ)

− W (θ) + α k X k

β + 1

2

− W (θ) + α k X k

β

+ 8aα k X k

L(θ)

βξ + 4c

α

k X k

β

+ β k θ

− θ

k

2nα + log(ε

) nα + λ

k Q

θ

k

− k Q

θ

k

.

Moreover from (2.3) and α k X k

β = − L(θ

) + Z

ρ

(dθ)L(θ), we deduce that Z

ρ

(dθ)

− W (θ) + α k X k

β

= V ar

L(θ)

+ W (θ

)

= W (θ

)

+ 4αL(θ

) k X k

β + 2α

k X k

β

. Proposition 2.2. With probability at least 1 − ε, for any θ

∈ R

, r

(θ

, θ

) ≤ 1

α log

1 + E

− W (θ

) + W (θ

)

2 + 2 + 8a/ξ

α k X k

L(θ

) β

+ 1 + 8a/ξ + 4c

α

k X k

β

+ β k θ

− θ

k

2nα + log(ε

) nα + λ

k Q

θ

k

− k Q

θ

k

≤ E

J(θ

)

− J (θ

)

+ 1

2α W (θ

)

+ (2 + 8a/ξ) k X k

L(θ

) β

+ (1 + 8a/ξ + 4c

)α k X k

β

+ β k θ

− θ

k

2nα + log(ε

) nα + λ

k Q

θ

k

− k Q

θ

k

.

Using the triangular inequality and Cauchy-Schwarz’s inequality, we get 1

α

E

W (θ

)

= E n

h θ

− θ

, X i

+ 2 h θ

− θ

, X i J(θ

)

o

≤ n E

h θ

− θ

, X i

+ 2 E

h θ

− θ

, X i

E

J(θ

)

o

(2.5)

≤ (

χ k θ

− θ

k

E

θ

− θ

k θ

− θ

k , X

+ 2 k θ

− θ

k σ p κ

χ

s E

θ

− θ

k θ

− θ

k , X

)

≤ χq

q

+ λ k θ

− θ

k

k θ

− θ

k

r χq

q

+ λ + 2σ √ κ

, and

1 α E

k X k

L(θ

)

= E n

k X kh θ

− θ

, X i + k X k J (θ

)

o

≤ E

k X k

n E

h θ

− θ

, X i

+ E

J (θ

)

o

(2.6)

≤ κD

k θ

− θ

k

r χq

q

+ λ + 2σ √ κ

. Let us put

R(θ) = e R(θ) + λ k Q

θ k

, c

= 4(2 + 8a/ξ),

c

= 4(1 + 8a/ξ + 4c

),

δ = c

κκ

Dσ

n +

2χ

log(ε

)

n + c

κ

D

n

2 √

κ

σ + k Θ k √ χ

1 − 4c

κχD n

.

We have proved the following result.

Proposition 2.3. With probability at least 1 − ε, for any θ

∈ R

, r

(θ

, θ

) ≤ R(θ e

) − R(θ e

) + α

2 χ k θ

− θ

k

2 √

κ

σ + k θ

− θ

k √ χ

+ c

α 4β κD √

κ

σ + k θ

− θ

k √ χ

+ c

ακ

D

4β

+ β k θ

− θ

k

2nα + log(ε

) nα . Let us assume from now on that θ

∈ Θ, our convex bounded parameter set. In this case, as seen in (1.14), we have k θ

− θ

k

≤ R(θ e

) − R(θ e

). We can also use the fact that

√

κ

σ + k θ

− θ

k √ χ

≤ 2κ

σ

+ 2χ k θ

− θ

k

. We deduce from these remarks that with probability at least 1 − ε,

r

(θ

, θ

) ≤

− 1+ αχ 2

2 √

κ

σ+ k Θ k √ χ

+ β

2nα + c

ακDχ 2β

e R(θ

) − R(θ e

)

+ c

ακDκ

σ

2β + c

ακ

D

4β

+ log(ε

) nα . Let us assume that n > 4c

κχD and let us choose

β = nα

2 ,

α = 1 2χ

2 √

κ

σ + k Θ k √ χ

1 − 4c

κχD n

, to get

r

(θ

, θ

) ≤ − R(θ e

) − R(θ e

)

2 + δ.

Plugging this into (2.1), we get R(¯ e θ) − R(θ e

)

2 − δ ≤ r

(¯ θ, θ

) ≤ max

θ1∈Θ

e R(θ

) − R(θ e

) 2

+ γ + δ = γ + δ, hence

R(¯ e θ) − R(θ e

) ≤ 2γ + 4δ.

Computing the numerical values of the constants when ξ = 0.8 gives c

< 95 and c

< 1511.

References.

[1] O. Catoni. Statistical Learning Theory and Stochastic Optimization, Lectures on Prob- ability Theory and Statistics, ´ Ecole d’ ´ Et´ e de Probabilit´ es de Saint-Flour XXXI – 2001, volume 1851 of Lecture Notes in Mathematics. Springer, 2004. Pages 1–269.

[2] M.D. Donsker and S.R.S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time, I. Communications on Pure and Applied Mathematics, 28(1):1–47, 1975.

[3] P. Dupuis and R.S. Ellis. A weak convergence approach to the theory of large deviations.

Wiley-Interscience, 1997.