Kernel Selection in Nonparametric Regression

(1)

HÉLÈNE HALCONRUY* AND NICOLAS MARIE^†

Abstract. In the regression modelY =b(X) +ε, whereX has a densityf, this paper deals with an oracle inequality for an estimator ofbf, involving a kernel in the sense of Lerasle et al. (2016), selected via the PCO method. In addition to the bandwidth selection for kernel-based estimators already studied in Lacour, Massart and Rivoirard (2017) and Comte and Marie (2020), the dimension selection for anisotropic projection estimators off andbfis covered.

Contents

1. Introduction 1

2. Risk bound 2

3. Kernel selection 4

4. Basic numerical experiments 6

Appendix A. Details on kernels sets: proofs of Propositions 2.2, 2.3 and 3.3 8

A.1. Proof of Proposition 2.2 8

Appendix B. Proofs of risk bounds 11

B.1. Preliminary results 11

B.2. Proof of Proposition 2.4 17

B.3. Proof of Theorem 2.5 17

B.4. Proof of Theorem 3.2 18

References 20

MSC2010: 62G05 ; 62G08.

1. Introduction

Considern∈N^∗independentR^d×R-valued (d∈N^∗) random variables(X1, Y1), . . . ,(Xn, Yn), having the same probability distribution assumed to be absolutely continuous with respect to Lebesgue’s measure, and

bsK,`(n;x) := 1 n

n

X

i=1

K(Xi, x)`(Yi) ;x∈R^d,

where`:R→Ris a Borel function andK is a symmetric continuous map fromR^d×R^d intoR. This is an estimator of the functions:R^d→Rdefined by

s(x) :=E(`(Y1)|X1=x)f(x) ;∀x∈R^d,

where f is a density of X1. For` = 1,bsK,`(n;.) coincides with the estimator off studied in Lerasle et al. [11], but for`6= 1, it covers estimators involved in nonparametric regression. Assume that for every i∈ {1, . . . , n},

(1) Yi=b(Xi) +εi

whereε_i is a centered random variable, independent ofX_i, andb:R^d→Ris a Borel function.

Key words and phrases. Nonparametric estimators ; Projection estimators ; Model selection ; Regression model.

1

arXiv:2006.07673v1 [math.ST] 13 Jun 2020

(2)

• If`=Id_R,kis a symmetric kernel and

(2) K(x⁰, x) =

d

Y

q=1

1 hq

k

x⁰_q−x_q hq

withh1, . . . , hd >0

for every x, x⁰ ∈ R^d, then bsK,`(n;.) is the numerator of Nadaraya-Watson’s estimator of the regression functionb. Precisely,sbK,`(n;.)is an estimator ofs=bf. If`6=Id_R, thenbsK,`(n;.)is the numerator of the estimator studied in Einmahl and Mason [5, 6].

• If ` = Id_R, Bmq = {ϕ₁^m^q, . . . , ϕ^mm^q_q} (m_q ∈ N^∗ and q ∈ {1, . . . , d}) is an orthonormal family of L²(R)and

(3) K(x⁰, x) =

d

Y

q=1 m_q

X

j=1

ϕ^m_j ^q(xq)ϕ^m_j ^q(x⁰_q)

for everyx, x⁰∈R^d, thensbK,`(n;.)is the projection estimator onS=span(Bm₁⊗ · · · ⊗ Bm_d)of s=bf.

Now, assume that for everyi∈ {1, . . . , n}, Y_i is defined by the heteroscedastic model

(4) Yi=σ(Xi)εi,

where εi is a centered random variable of variance 1, independent of Xi, and σ : R^d → R is a Borel function. If`(x) =x² for everyx∈R, thensbK,`(n;.)is an estimator ofs=σ²f.

These ten last years, several data-driven procedures have been proposed in order to select the bandwidth of Parzen-Rosenblatt’s estimator (` = 1 and K defined by (2)). First, Goldenshluger-Lepski’s method, introduced in [8], which reaches the adequate bias-variance compromise, but is not completely satisfactory on the numerical side (see Comte and Rebafka [4]). More recently, in [10], Lacour, Massart and Rivoirard proposed the PCO (Penalized Comparison to Overfitting) method and proved an oracle inequality for the associated adaptative Parzen-Rosenblatt’s estimator by using a concentration inequality for the U-statistics due to Houdré and Reynaud-Bouret [9]. Together with Varet, they established the numerical efficiency of the PCO method in Varet et al. [13].

Comte and Marie [3] deals with an oracle inequality and numerical experiments for an adaptative Nadaraya-Watson’s estimator with a numerator and a denominator having distinct bandwidths, both selected via the PCO method. Since the output variable in a regression model has no reason to be bounded, there were significant additional difficulties, bypassed in [3], to establish an oracle inequality for the numerator’s adaptative estimator. Via similar arguments, the present article deals with an oracle inequality forbs

K,`c (n;.), whereKb is selected via the PCO method in the spirit of Lerasle et al. [11]. In addition to the bandwidth selection for kernel-based estimators already studied in [10, 3], it covers the dimension selection for anisotropic projection estimators off,bf (whenY1, . . . , Yn are defined by Model (1)) and σ²f (when Y1, . . . , Yn are defined by Model (4)). As for the bandwidth selection for kernel based estimators, ford >1, the PCO method allows to bypass the numerical difficulties generated by the Goldenshluger-Lepski type method involved in the anisotropic model selection procedures (see Chagny [1]).

In Section 2, some examples of kernels sets are provided and a risk bound for bs_K,`(n;.) is established.

Section 3 deals with an oracle inequality forbs

K,`c (n;.), whereKb is selected via the PCO method. Finally, Section 4 deals with a basic numerical study.

2. Risk bound

Throughout the paper,s∈L²(R^d). LetKn be a set of symmetric continuous maps fromR^d×R^d into R, of cardinal less or equal thann, fulfilling the following assumption.

Assumption 2.1. There exists a deterministic constant mK,`>0, not depending on n, such that

(3)

(1) For everyK∈ K_n,

sup

x⁰∈R^d

kK(x⁰, .)k²₂6m_K,`n.

(2) For everyK∈ Kn,

ksK,`k²₂6m_K,`

with

sK,`:=E(sbK,`(n;.)) =E(K(X1, .)`(Y1)).

(3) For everyK, K⁰∈ Kn,

E(hK(X1, .), K⁰(X₂, .)`(Y₂)i²₂)6m_K,`s_K⁰_,`

with

s_K⁰_,`:=E(kK⁰(X₁, .)`(Y₁)k²₂).

(4) For everyK∈ K_n andψ∈L²(R^d),

E(hK(X1, .), ψi²₂)6m_K,`kψk²₂.

The elements ofKn are called kernels. Let us provide two natural examples of kernels sets.

Proposition 2.2. Consider

K_k(h_min) :=

(

(x⁰, x)7→

d

Y

q=1

1 h_qk

x⁰_q−xq

h_q

; h₁, . . . , h_d∈ {h_min, . . . ,1}

) ,

where k is a symmetric kernel (in the usual sense) and nh^d_min > 1. The kernels set Kk(hmin) fulfills Assumption 2.1 and, for anyK∈ Kk(hmin)such that

K(x⁰, x) =

d

Y

q=1

1 hq

k

x⁰_q−xq

hq

; ∀x, x⁰∈R^d

withh1, . . . , hd∈ {hmin, . . . ,1},

sK,`=kkk^2d₂ E(`(Y1)²)

d

Y

q=1

1 hq

.

Proposition 2.3. Consider

KB1,...,Bn(mmax) :=







(x⁰, x)7→

d

Y

q=1 m_q

X

j=1

ϕ^m_j ^q(xq)ϕ^m_j ^q(x⁰_q) ; m1, . . . , md∈ {1, . . . , mmax}





 ,

where m^d_max ∈ {1, . . . , n} and, for every m∈ {1, . . . , n},Bm ={ϕ^m₁, . . . , ϕ^m_m} is an orthonormal family of L²(R)such that

sup

x⁰∈R m

X

j=1

ϕ^m_j (x⁰)²6m_Bm

withm_B>0not depending on mandn, and

(5) Bm⊂ Bm+1 ;∀m∈ {1, . . . , n−1}

or

(6) m_B:= sup{|E(K(X1, x))| ;K∈ K_B₁_,...,B_n(mmax)andx∈R^d}is finite and doesn’t depend onn.

The kernels setKB₁,...,B_n(mmax)fulfills Assumption 2.1 and, for anyK∈ KB₁,...,B_n(mmax)such that

K(x⁰, x) =

d

Y

q=1 m_q

X

j=1

ϕ^m_j^q(xq)ϕ_j^m^q(x⁰_q) ; ∀x, x⁰∈R^d

(4)

withm₁, . . . , m_n∈ {1, . . . , m_max},

sK,`6m^d_BE(`(Y1)²)

d

Y

q=1

mq.

Remark. Note that Condition (5) (resp. (6)) is close to (resp. the same that) Condition (19) (resp.

(20)) of Lerasle et al. [11], Proposition 3.2. See also Massart [12], Chapter 7 on these conditions. For instance, the trigonometric basis and Hermite’s basis satisfy Condition (5). The regular histograms basis satisfy Condition (6). Indeed, by taking ϕ^m_j = ψ_j^m := √

m1[(j−1)/m,j/m[ for every m ∈ {1, . . . , n} and j∈ {1, . . . , m},

E





d

Y

q=1 m_q

X

j=1

ψ^m_j ^q(X_1,q)ψ_j^m^q(x_q)





=

m1

X

j₁=1

· · ·

m_d

X

j_d=1 d

Y

q=1

m_q1_[(j_q_−1)/m_q_,j_q_/m_q_[(x_q)

!

× Z j₁/m₁

(j1−1)/m1

· · ·

Z j_d/m_d (jd−1)/md

f(x⁰₁, . . . , x⁰_d)dx⁰₁· · ·dx⁰_d

6kfk∞ d

Y

q=1 m_q

X

j=1

1_[(j−1)/m_q,j/m_q[(x)6kfk∞

for everym₁, . . . , m_d ∈ {1, . . . , n}andx∈R^d.

The following proposition provides a suitable control of the variance ofbsK,`(n;.).

Proposition 2.4. Under Assumption 2.1.(1,2,3), if s ∈ L²(R^d) and if there exists α > 0 such that E(exp(α|`(Y1)|))<∞, then there exists a deterministic constantc_2.4>0, not depending onn, such that for everyθ∈]0,1[,

E

sup

K∈Kn

kbsK,`(n;.)−sK,`k²₂−sK,`

n −θ

nsK,`

6c2.4

log(n)⁵ θn . Finally, let us state the main result of this section.

Theorem 2.5. Under Assumption 2.1, ifs∈L²(R^d)and if there existsα >0such thatE(exp(α|`(Y₁)|))<

∞, then there exists a deterministic constant c2.5,c2.5 > 0, not depending on n, such that for every θ∈]0,1[,

E

sup

K∈Kn

nkbsK,`(n;.)−sk²₂−(1 +θ)

ksK,`−sk²₂+s_K,`

n o

6c2.5

log(n)⁵ θn and

E

sup

K∈Kn

ksK,`−sk²₂+sK,`

n − 1

1−θkbsK,`(n;.)−sk²₂

6c2.5

log(n)⁵ θ(1−θ)n.

Remark. Note that the first inequality in Theorem 2.5 gives a risk bound on the estimatorbsK,`(n;.):

E(kbs_K,`(n;.)−sk²₂)6(1 +θ)

ks_K,`−sk²₂+sK,`

n

+c_2.5log(n)⁵ θn

for everyθ ∈]0,1[. The second inequality is useful in order to establish a risk bound on the adaptative estimator defined in the next section (see Theorem 3.2).

3. Kernel selection This section deals with a risk bound on the adaptative estimatorsb

K,`c (n;.), where Kb ∈arg min

K∈Kn

{kbsK,`(n;·)−bsK0,`(n;·)k²₂+pen(K)}, K₀is an overfitting proposal forK and

(7) pen(K) := 2

n²

n

X

i=1

hK(., Xi), K0(., Xi)i2`(Yi)²;∀K∈ Kn.

(5)

Example. ForK_n=K_k(h_min), one should take K0(x⁰, x) = 1

h^d_min

d

Y

q=1

k

x⁰_q−xq

hmin

;∀x, x⁰ ∈R^d,

and forKn =K_B₁_,...,B_n(mmax), one should take K0(x⁰, x) =

d

Y

q=1 m_max

X

j=1

ϕ^m_j^max(xq)ϕ^m_j ^max(x⁰_q) ;∀x, x⁰∈R^d.

In the sequel, in addition to Assumption 2.1, the kernels setKn fulfills the following assumption.

Assumption 3.1. There exists a deterministic constant mK,`>0, not depending on n, such that

E

sup

K,K⁰∈Kn

hK(X1, .), sK⁰,`i²₂

6m_K,`.

The following theorem provides an oracle inequality for the adaptative estimatorbs

K,`c (n;.).

Theorem 3.2. Under Assumptions 2.1 and 3.1, if s ∈ L²(R^d) and if there exists α > 0 such that E(exp(α|`(Y1)|))<∞, then there exists a deterministic constantc3.2>0, not depending onn, such that for everyϑ∈]0,1[,

E(kbs

K,`c (n;.)−sk²₂)6(1 +ϑ) min

K∈Kn

E(kbsK,`(n;.)−sk²₂) +c3.2

ϑ

ksK₀,`−sk²₂+log(n)⁵ n

. Finally, let us discuss about Assumption 3.1. Note that ifsis bounded and

m_K:= sup{kK(x⁰, .)k²₁ ;K∈ Kn andx⁰ ∈R^d} doesn’t depend onn, thenKn fulfills Assumption 3.1. Indeed,

E sup

K,K⁰∈K_n

hK(X1, .), sK⁰,`i²2

! 6

sup

K⁰∈K_n

ksK⁰,`k²∞

E

sup

K∈K_n

kK(X1, .)k²1

6 mKsup (Z ∞

−∞

|K⁰(x⁰, x)s(x)|dx 2

;K⁰∈ Kn andx⁰∈R )

6m²Kksk²∞. In the nonparametric regression framework (see Model (1)), to assume s bounded means that bf is bounded. For instance, this condition is fulfilled by the linear regression models with Gaussian inputs.

The following examples focus on the condition onm_K. Examples:

(1) ConsiderK∈ Kk(hmin). Then, there existsh1, . . . , hd ∈ {hmin, . . . ,1}such that K(x⁰, x) =

d

Y

q=1

1 hq

k

x⁰_q−xq

hq

;∀x, x⁰ ∈R^d.

Clearly,kK(x⁰, .)k₁=kkk^d₁ for every x⁰ ∈R^d. So, forK_n =K_k(h_min),m_K6kkk^2d₁ .

(2) For Kn =K_B₁_,...,B_n(mmax), the condition on m_K seems harder to check in general. Let us show that it is satisfied for the regular histograms basis defined in Section 2. For everym1, . . . , md∈ {1, . . . , n},

d

Y

q=1 m_q

X

j=1

ψ_j^m^q(x⁰_q)ψ_j^m^q(._q) 1

6

d

Y

q=1



m_q

m_q

X

j=1

1_[(j−1)/m_q_,j/m_q_[(x⁰_q) Z j/m_q

(j−1)/mq

dx



61.

The following proposition shows thatK_B₁_,...,B_n(mmax)fulfills Assumption 3.1 for the trigonometric basis, even if the condition onmK is not satisfied.

(6)

Proposition 3.3. Consider χ₁ :=1_[0,1] and, for everyj ∈N^∗, the functions χ_2j andχ_2j+1 defined on Rby

χ2j(x) :=√

2 cos(2πjx)1_[0,1](x)andχ2j+1(x) :=√

2 sin(2πjx)1_[0,1](x) ; ∀x∈R.

Ifs∈C²(R^d)andBm={χ1, . . . , χ_m}for everym∈ {1, . . . , n}, thenKB1,...,Bn(m_max)fulfills Assumption 3.1.

4. Basic numerical experiments

Throughout this section,d= 1,`∈ {1,Id_R}andY1, . . . , Ynare defined by Model (1) withε1, . . . , εn

N(0,1). Some numerical experiments on bsK,1(n;.)(resp. bsK,Id_R(n;.)) for K ∈ Kk(hmin) have already been done in Varet et al. [13] (resp. Comte and Marie [3]). So, this section deals with basic numerical experiments on sbK,1(n;.) and sbK,Id_R(n;.) for K ∈ KB1,...,Bn(mmax)and Bm ={ψ^m₁, . . . , ψ_m^m} for every m= 1, . . . , n.

In this case,Kb =K

cm(`) where Km(x⁰, x) :=

m

X

j=1

ψ_j^m(x⁰)ψ_j^m(x) ;∀x, x⁰∈R,∀m∈ M={1, . . . , mmax},

m(`)b is a solution of the minimization problem

m∈Mmin{kbsK_m,`(n;.)−bsK_mmax,`(n;.)k²₂+pen(m)}

and

pen(m) := 2 n²

n

X

i=1

hKm(., Xi),Km_max(., Xi)i2`(Yi)² ;∀m∈ M.

For`∈ {1,Id_R}, n= 250and mmax= 30, mis selected inMfor two basic densities and two nonlinear regression functions:

• f =f1 the density ofE(5).

• f =f2 the density ofN(1/2,(1/8)²).

• b(x) =b1(x) := 10(x²−1/2)for every x∈[0,1].

• b(x) =b2(x) := cos(5πx)for every x∈[0,1].

On the one hand, on the four following figures, one can see the beam of all possible estimations off and bf (i.e. for eachm∈ M) at left, the PCO criteria for bsK,1(n;.)andbsK,Id_R(n;.)for eachm∈ Mat the middle, and the PCO estimations off andbf (i.e. form=m(1)b andm=m(Idb _R)) at right:

Figure 1. f =f₁,b=b₁,m(1) = 10b andm(Idb _R) = 10.

(7)

Figure 2. f =f₁,b=b₂,m(1) = 12b andm(Idb _R) = 20.

Figure 3. f =f2,b=b1,m(1) = 10b andm(Idb _R) = 15.

Figure 4. f =f2,b=b2,m(1) = 15b andm(Idb _R) = 6.

(8)

On the other hand, for (f, b) = (f₁, b₂) and (f, b) = (f₂, b₁), let us generate 10 datasets of n = 250 observations of(X1, Y1)and, for each of these, selectm∈ Mvia the PCO criterion introduced previously.

On the two following figures, the beam of all PCO estimations off (resp. bf) is plotted at left (resp. at right):

Figure 5. f =f1andb=b2. Figure 6. f =f2and b=b1. Appendix A. Details on kernels sets: proofs of Propositions 2.2, 2.3 and 3.3 A.1. Proof of Proposition 2.2. Consider K, K⁰ ∈ Kk(hmin). Then, there exist h, h⁰ ∈ {hmin, . . . ,1}^d such that

K(x⁰, x) =

d

Y

q=1

1 h_qk

x⁰_q−x_q h_q

andK⁰(x⁰, x) =

d

Y

q=1

1 h⁰_qk

x⁰_q−x_q h⁰_q

for everyx, x⁰∈R^d. (1) For everyx⁰∈R^d,

kK(x⁰, .)k²₂=kkk^2d₂

d

Y

q=1

1

h_q 6kkk^2d₂ n.

(2) Sinces_K,`=K∗s,ksK,`k²₂6kkk^2d₁ ksk²₂. (3) First,

sK⁰,`=kkk^2d₂ E(`(Y1)²)

d

Y

q=1

1 h⁰_q. Then,

E(hK(X1, .), K⁰(X2, .)`(Y2)i²₂) = E((K∗K⁰)(X1−X2)²`(Y2)²) 6kfk_∞kK∗K⁰k²₂E(`(Y1)²) 6kfk_∞kkk^2d₁ sK⁰,`.

(4) For everyψ∈L²(R^d),

E(hK(X1, .), ψi²₂) = E((K∗ψ)(X1)²)

6kfk_∞kK∗ψk²₂6kfk_∞kkk^2d₁ kψk²₂.

A.2. Proof of Proposition 2.3. Consider K, K⁰ ∈ K_B₁_,...,B_n(m_max). Then, there exist m, m⁰ ∈ {1, . . . , mmax}^d such that

K(x⁰, x) =

d

Y

q=1 mq

X

j=1

ϕ^m_j ^q(xq)ϕ^m_j ^q(x⁰_q)andK⁰(x⁰, x) =

d

Y

q=1 m⁰_q

X

j=1

ϕ^m

0 q

j (xq)ϕ^m

0 q

j (x⁰_q) for everyx, x⁰∈R^d.

(9)

(1) For everyx⁰∈R^d,

kK(x⁰, .)k²₂ =

d

Y

q=1 mq

X

j,j⁰=1

ϕ^m_j0^q(x⁰_q)ϕ^m_j ^q(x⁰_q) Z ∞

−∞

ϕ^m_j0^q(x)ϕ^m_j^q(x)dx

=

d

Y

q=1 m_q

X

j=1

ϕ^m_j ^q(x⁰_q)²6m^d_B

d

Y

q=1

mq 6m^d_Bn.

(2) Since

s_K,`(.) =

m1

X

j₁=1

· · ·

m_d

X

j_d=1

hs, ϕ^m_j¹

1 ⊗ · · · ⊗ϕ^m_j^d

d i₂(ϕ^m_j ¹

1 ⊗ · · · ⊗ϕ^m_j ^d

d )(.),

by Pythagore’s theorem,ksK,`k²₂6ksk²₂. (3) First,

sK⁰,`=E



`(Y1)²

d

Y

q=1 m⁰_q

X

j=1

ϕ^m

0 q

j (X1,q)²



6m^d_BE(`(Y1)²)

d

Y

q=1

m⁰_q.

On the one hand, ifB₁, . . . ,B_n satisfy Condition (5), then

E(hK(X1, .), K⁰(X2, .)`(Y2)i²2) = Z

R^d

E











d

Y

q=1 mq∧m⁰_q

X

j=1

ϕ^m

0 q j (x⁰q)ϕ^m

0 q j (X2,q)





2

`(Y2)²





f(x⁰)λd(dx⁰)

6 kfk∞E



`(Y2)²

d

Y

q=1 mq∧m⁰_q

X

j,j⁰=1

ϕ^m

0 q

j⁰ (X2,q)ϕ^m

0 q j (X2,q)

Z ∞

−∞

ϕ^m

0 q j⁰ (x⁰)ϕ^m

0 q j (x⁰)dx⁰





6kfk∞sK⁰,`.

On the other hand, ifB1, . . . ,Bn satisfy Condition (6), then

E(hK(X1, .), K⁰(X₂, .)`(Y₂)i²₂) 6E(kK(X1, .)k²₂kK⁰(X₂, .)k²₂`(Y₂)²)

=E(K(X1, X1))E(kK⁰(X2, .)k₂²`(Y2)²)6m_BsK⁰,`. (4) For everyψ∈L²(R^d),

E(hK(X1, .), ψi²₂) = E







m₁

X

j1=1

· · ·

m_d

X

jd=1

hψ, ϕ^m_j ¹

1 ⊗ · · · ⊗ϕ^m_j ^d

d i2(ϕ^m_j¹

1 ⊗ · · · ⊗ϕ^m_j^d

d )(X1)

2





6 kfk_∞

m₁

X

j1=1

· · ·

m_d

X

j_d=1

hψ, ϕ^m_j ¹

1 ⊗ · · · ⊗ϕ^m_j ^d

d i2(ϕ^m_j¹

1 ⊗ · · · ⊗ϕ^m_j^d

d )(.)

2

6kfk_∞kψk²₂.

A.3. Proof of Proposition 3.3. For the sake of readability, assume that d = 1. Consider K, K⁰ ∈ K_B₁_,...,B_n(mmax). Then, there existm, m⁰∈ {1, . . . , mmax} such that

K(x⁰, x) =

m

X

j=1

χj(x)χj(x⁰)andK⁰(x⁰, x) =

m⁰

X

j=1

χj(x)χj(x⁰) ;∀x, x⁰∈R.

(10)

First, there existm₁(m, m⁰)∈ {0, . . . , n} andc₁>0, not depending onn, K andK⁰, such that for any x⁰ ∈[0,1],

|hK(x⁰, .), sK⁰,`i2| =

m∧m⁰

X

j=1

E(`(Y1)χj(X1))χj(x⁰)

6 c1+ 2

m₁(m,m⁰)

X

j=1

E(`(Y1)(cos(2πjX1) cos(2πjx⁰) + sin(2πjX1) sin(2πjx⁰))1_[0,1](X1))

= c1+ 2

m1(m,m⁰)

X

j=1

E(`(Y1) cos(2πj(X1−x⁰))1[0,1](X1)) .

Moreover, for anyj∈ {2, . . . ,m1(m, m⁰)}, E(`(Y₁) cos(2πj(X₁−x⁰))1_[0,1](X₁)) =

Z 1 0

cos(2πj(x−x⁰))s(x)dx

= 1 j

sin(2πj(x−x⁰))

2π s(x)

1 0

+1 j²

cos(2πj(x−x⁰)) 4π² s⁰(x)

¹

0

− 1 j²

Z 1 0

cos(2πj(x−x⁰))

4π² s⁰⁰(x)dx

= s(0)−s(1)

2π · α_j(x⁰)

j +β_j(x⁰) j² whereαj(x⁰) := sin(2πjx⁰)and

βj(x⁰) := 1 4π²

(s⁰(1)−s⁰(0)) cos(2πjx⁰)− Z 1

0

cos(2πj(x−x⁰))s⁰⁰(x)dx

.

Then, there exists a deterministic constantc2>0, not depending onn,K,K⁰ andx⁰, such that

(8) hK(x⁰, .), s_K⁰_,`i²₂6c₂





1 +





m1(m,m⁰)

X

j=1

α_j(x⁰) j





2

+





m1(m,m⁰)

X

j=1

β_j(x⁰) j²





2



.

Let us show that each term of the right-hand side of Inequality (8) are uniformly bounded inx⁰,mand m⁰. On the one hand,

m₁(m,m⁰)

X

j=1

βj(x⁰) j²

6 max

j∈{1,...,n}kβjk_∞

n

X

j=1

1 j² 6 1

24(2ks⁰k_∞+ks⁰⁰k_∞).

On the other hand, for everyx∈]0, π[ such that[π/x] + 16m1(m, m⁰)(without loss of generality),

m₁(m,m⁰)

X

j=1

sin(jx) j

6

[π/x]

X

j=1

sin(jx) j

+

m₁(m,m⁰)

X

j=[π/x]+1

sin(jx) j

6 xhπ

x i

+ 2

(1 + [π/x]) sin(x/2) 6π+ 2.

(9)

Sincex7→sin(x)is continuous, odd and 2π-periodic, Inequality (9) holds true for everyx∈R. So,

m1(m,m⁰)

X

j=1

α_j(x⁰) j

6π+ 2.

(11)

Therefore, E

"

sup

K,K⁰∈KB1,...,Bn(mmax)

hK(X1, .), sK⁰,`i²₂

# 6c2

1 + (π+ 2)²+ 1

24²(2ks⁰k∞+ks⁰⁰k∞)²

.

Appendix B. Proofs of risk bounds

In this section, the proofs follow the same pattern as in Comte and Marie [2, 3].

B.1. Preliminary results. This subsection provides three lemmas used several times in the sequel.

Lemma B.1. Consider U_K,K⁰_,`(n) :=X

i6=j

hK(Xi, .)`(Y_i)−s_K,`, K⁰(X_j, .)`(Y_j)−s_K⁰_,`i2 ; ∀K, K⁰ ∈ Kn.

Under Assumption 2.1.(1,2,3), ifs∈L²(R^d)and if there exists α >0 such that E(exp(α|`(Y1)|))<∞, then there exists a deterministic constantcB.1>0, not depending onn, such that for every θ∈]0,1[,

E

sup

K,K⁰∈Kn

|UK,K⁰,`(n)|

n² −θ nsK⁰,`

6cB.1

log(n)⁵ θn . Lemma B.2. Consider

V_K,`(n) := 1 n

n

X

i=1

kK(Xi, .)`(Y_i)−s_K,`k²₂ ;∀K∈ Kn.

Under Assumption 2.1.(1,2), if s ∈ L²(R^d) and if there exists α > 0 such that E(exp(α|`(Y1)|))< ∞, then there exists a deterministic constantc_B.2>0, not depending onn, such that for every θ∈]0,1[,

E

sup

K∈Kn

1

n|VK,`(n)−sK,`| − θ nsK,`

6cB.2

log(n)³ θn . Lemma B.3. Consider

WK,K⁰,`(n) :=hbsK,`(n;.)−sK,`, sK⁰,`−si2 ; ∀K, K⁰∈ Kn.

Under Assumption 2.1.(1,2,4), ifs∈L²(R^d)and if there exists α >0 such that E(exp(α|`(Y₁)|))<∞, then there exists a deterministic constantc_B.3>0, not depending onn, such that for every θ∈]0,1[,

E

sup

K,K⁰∈Kn

{|WK,K⁰,`(n)| −θksK⁰,`−sk²₂}

6cB.3

log(n)⁴ θn . B.1.1. Proof of Lemma B.1. Considerm(n) := 8 log(n)/α. For anyK, K⁰ ∈ Kn,

UK,K⁰,`(n) =U_K,K¹ 0,`(n) +U_K,K² 0,`(n) +U_K,K³ 0,`(n) +U_K,K⁴ 0,`(n) where

U_K,K^l 0,`(n) :=X

i6=j

g^l_K,K0,`(n;Xi, Yi, Xj, Yj) ;l= 1,2,3,4

with, for every(x⁰, y),(x⁰⁰, y⁰)∈E=R^d×R,

g¹_K,K0,`(n;x⁰, y, x⁰⁰, y⁰) :=hK(x⁰, .)`(y)1_|`(y)|₆_m(n)−s_K,`⁺ (n;.), K⁰(x⁰⁰, .)`(y⁰)1_|`(y)|₆_m(n)−s⁺_K0,`(n;.)i2, g²_K,K0,`(n;x⁰, y, x⁰⁰, y⁰) :=hK(x⁰, .)`(y)1|`(y)|>m(n)−s_K,`⁻ (n;.), K⁰(x⁰⁰, .)`(y⁰)1_|`(y)|6_m(n)−s⁺_K0,`(n;.)i2, g³_K,K0,`(n;x⁰, y, x⁰⁰, y⁰) :=hK(x⁰, .)`(y)1_|`(y)|₆_m(n)−s⁺_K,`(n;.), K⁰(x⁰⁰, .)`(y⁰)1|`(y)|>m(n)−s⁻_K0,`(n;.)i2, g⁴_K,K0,`(n;x⁰, y, x⁰⁰, y⁰) :=hK(x⁰, .)`(y)1|`(y)|>m(n)−s⁻_K,`(n;.), K⁰(x⁰⁰, .)`(y⁰)1|`(y)|>m(n)−s⁻_K0,`(n;.)i₂ and, for everyk∈ Kn,

s⁺_k,`(n;.) :=E(k(X₁, .)`(Y₁)1_|`(Y₁_)|₆_m(n))ands⁻_k,`(n;.) :=E(k(X₁, .)`(Y₁)1_|`(Y₁_)|>m(n)).

(12)

On the one hand, since E(g_K,K¹ 0,`(n;x⁰, y, X₁, Y₁)) = 0 for every (x⁰, y) ∈ E, by Giné and Nickl [7], Theorem 3.4.8, there exists a universal constantm>1 such that for any λ >0, with probability larger than1−5.4e^−λ,

|U_K,K¹ 0,`(n)|

n² 6 m

n²(cK,K⁰,`(n)λ^1/2+dK,K⁰,`(n)λ+bK,K⁰,`(n)λ^3/2+aK,K⁰,`(n)λ²)

where the constants aK,K⁰,`(n), bK,K⁰,`(n), cK,K⁰,`(n) and dK,K⁰,`(n) are defined and controlled later.

First, note that

U_K,K¹ 0,`(n) = X

i6=j

(ϕK,K⁰,`(n;Xi, Yi, Xj, Yj)

−ψK,K⁰,`(n;X_i, Y_i)−ψ_K⁰_,K,`(n;X_j, Y_j) +E(ϕ_K,K⁰_,`(n;X_i, Y_i, X_j, Y_j))), (10)

where

ϕK,K⁰,`(n;x⁰, y, x⁰⁰, y⁰⁰) :=hK(x⁰, .)`(y)1_|`(y)|₆_m(n), K⁰(x⁰⁰, .)`(y⁰)1_|`(y⁰_)|₆_m(n)i2

and

ψ_k,k0,`(n;x⁰, y) :=hk(x⁰, .)`(y)1_|`(y)|6_m(n), s⁺_k0,`(n;.)i2=E(ϕ_k,k0,`(n;x⁰, y, X₁, Y₁))

for every k, k⁰ ∈ Kn and (x⁰, y),(x⁰⁰, y⁰)∈E. Let us now control aK,K⁰,`(n), bK,K⁰,`(n), cK,K⁰,`(n)and dK,K⁰,`(n):

• The constant aK,K⁰,`(n). Consider aK,K⁰,`(n) := sup

(x⁰,y),(x⁰⁰,y⁰)∈E

|g_K,K¹ 0,`(n;x⁰, y, x⁰⁰, y⁰)|.

By (10), Cauchy-Schwarz’s inequality and Assumption 2.1.(1), aK,K⁰,`(n) 6 4 sup

(x⁰,y),(x⁰⁰,y⁰)∈E

|hK(x⁰, .)`(y)1_|`(y)|6m(n), K⁰(x⁰⁰, .)`(y⁰)1_|`(y⁰_)|6m(n)i2|

6 4m(n)²

sup

x⁰∈R^d

kK(x⁰, .)k2 sup

x⁰⁰∈R^d

kK⁰(x⁰⁰, .)k2

64mK,`m(n)²n.

So,

1

n²aK,K⁰,`(n)λ²6 4

nm_K,`m(n)²λ².

• The constant bK,K⁰,`(n). Consider b_K,K⁰_,`(n)²:=n sup

(x⁰,y)∈EE(g_K,K¹ 0,`(n;x⁰, y, X₁, Y₁)²).

By (10), Jensen’s inequality, Cauchy-Schwarz’s inequality and Assumption 2.1.(1), b_K,K⁰_,`(n)² 616n sup

(x⁰,y)∈EE(hK(x⁰, .)`(y)1_|`(y)|6_m(n), K⁰(X₁, .)`(Y₁)1_|`(Y₁_)|6_m(n)i²₂) 616nm(n)² sup

x⁰∈R^d

kK(x⁰, .)k²₂E(kK⁰(X1, .)`(Y1)1_|`(Y₁_)|6m(n)k²₂)616mK,`n²m(n)²sK⁰,`. So, for anyθ∈]0,1[,

1

n²bK,K⁰,`(n)λ^3/2 6 2 3m

θ

^1/2 2

n^1/2m^1/2_K,`m(n)λ^3/2× θ

3m

^1/2 1 n^1/2s^1/2_K0,`

6 θ

3mnsK⁰,`+12mλ³

θn mK,`m(n)².

• The constant c_K,K⁰_,`(n). Consider

cK,K⁰,`(n)²:=n²E(g¹_K,K0,`(n;X1, Y1, X2, Y2)²).

By (10), Jensen’s inequality and Assumption 2.1.(3),

cK,K⁰,`(n)² 616n²E(hK(X1, .)`(Y1)1_|`(Y₁_)|6m(n), K⁰(X2, .)`(Y2)1_|`(Y₂_)|6m(n)i²₂) 616n²m(n)²E(hK(X1, .), K⁰(X2, .)`(Y2)i²₂)616mK,`n²m(n)²sK⁰,`.

(13)

So,

1

n²c_K,K⁰_,`(n)λ^1/26 θ

3mns_K⁰_,`+12mλ

θn m_K,`m(n)².

• The constant dK,K⁰,`(n). Consider dK,K⁰,`(n) := sup

(a,b)∈AE



 X

i<j

ai(Xi, Yi)bj(Xj, Yj)g¹_K,K0,`(n;Xi, Yi, Xj, Yj)



,

where

A:=





 (a, b) :

n−1

X

i=1

E(a_i(X_i, Y_i)²)61and

n

X

j=2

E(b_j(X_j, Y_j)²)61





 .

By (10), Jensen’s inequality, Cauchy-Schwarz’s inequality and Assumption 2.1.(3), dK,K⁰,`(n)6 4 sup

(a,b)∈AE





n−1

X

i=1 n

X

j=i+1

|ai(Xi, Yi)bj(Xj, Yj)ϕK,K⁰,`(n;Xi, Yi, Xj, Yj)|





6 4nm(n)E(hK(X1, .), K⁰(X2, .)`(Y2)i₂²)^1/264m^1/2_K,`nm(n)s^1/2_K0,`. So,

1

n²dK,K⁰,`(n)λ6 θ

3mnsK⁰,`+12mλ²

θn mK,`m(n)². Then, sincem>1 andλ >0, with probability larger than1−5.4e^−λ,

|U_K,K¹ 0,`(n)|

n² 6 θ

nsK⁰,`+40m²

θn mK,`m(n)²(1 +λ)³. So, with probability larger than1−5.4|Kn|e^−λ,

S_K,`(n, θ) := sup

K,K⁰∈Kn

(|U_K,K¹ 0,`(n)|

n² −θ

nsK⁰,`

)

6 40m²

θn mK,`m(n)²(1 +λ)³. For everyt∈R+, consider

λ_K,`(n, θ, t) :=−1 +

t m_K,`(n, θ)

^1/3

withm_K,`(n, θ) =40m²

θn m_K,`m(n)². Then, for anyT >0,

E(SK,`(n, θ))6 T+ Z ∞

T

P(SK,`(n, θ)>(1 +λK,`(n, θ, t))³mK,`(n, θ))dt

6 2T+ 5.4c1|Kn|mK,`(n, θ) exp

− T^1/3 2m_K,`(n, θ)^1/3

withc1= Z ∞

0

e^1−r^1/3^/2dr.

Moreover,

mK,`(n, θ)6c2

log(n)²

θn withc2=40·8²m² α² mK,`. So, by taking

T = 2³c2

log(n)⁵ θn , and since|Kn|6n,

E(S_K,`(n, θ))62⁴c₂log(n)⁵

θn + 5.4c₁m_K,`(n, θ)|K_n|

n 6(2⁴+ 5.4c₁)c₂log(n)⁵ θn .