HAL Id: hal-01298861
https://hal.archives-ouvertes.fr/hal-01298861
Preprint submitted on 6 Apr 2016
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Nested polynomial trends for the improvement of
Gaussian process-based predictors
Guillaume Perrin, Christian Soize, Josselin Garnier, Marque-Pucheu Sophie
To cite this version:
Guillaume Perrin, Christian Soize, Josselin Garnier, Marque-Pucheu Sophie. Nested polynomial trends for the improvement of Gaussian process-based predictors. 2016. �hal-01298861�
*Manuscript
d ≥ 1 L2(D d, R) Dd Rd R (·, ·) ∥·∥L2 u v L2(D d, R) (u, v)L2 := ! Dd
u(x)v(x)dx, ∥u∥2L2 := (u, u)L2.
S d x= (x1, . . . , xd) g(x) g L2(D d, R) g N " x(1), . . . , x(N )# D d g⋆ g ∀ $g ∈ L2(Dd, R), ∥g − g⋆∥2L2 ≤ ∥g − $g∥ 2 L2. g(x) g g := {g(x), x ∈ Dd} Y := {Y (x, ω), x ∈ Dd, ω ∈ Ω} (Ω, T , P) µ C Y Y ∼ (µ, C). FN σ g
Y=%y(1) = g(x(1)), . . . , y(N )= g(x(N ))&, P( · | FN) E[ · | FN] Y M L2(D d, R) f = (f1, . . . , fM) M β µ := ⟨f, β⟩ , Y | β ∼ (⟨f, β⟩ , C), ⟨·, ·⟩ RM [F ] [C] f C "x(1), . . . , x(N )# ' [F ] := [f (x(1)) · · · f(x(N ))]T, [C]ij := C(x(i), x(j)), 1 ≤ i, j ≤ N, [C] Y | β, FN ∼ (µN, CN), x, x′ D d ⎧ ⎪ ⎨ ⎪ ⎩ µN(x) := ⟨f(x), β⟩ + r(x)T[C]−1(Y − [F ]β) , CN(x, x′) := C(x, x′) − r(x)T[C]−1r(x′), r(x) :=%C(x, x(1)), . . . , C(x, x(N ))&. [F ]T[C]−1[F ] β RM Y | FN ∼ (µ , C ),
⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ µ (x) := ⟨f(x), β⋆ ⟩ + r(x)T[C]−1 (Y − [F ]β⋆) , C (x, x′) := CN(x, x′) + u(x)T([F ]T[C]−1[F ])−1u(x′), β⋆:= ([F ]T[C]−1[F ])−1[F ]T[C]−1Y, u(x) := [F ]T[C]−1r (x) − f(x). g x (Y (x) | FN) µ (x) C (x, x) µ (x) g(x) C f g C x, x′ D d C(x, x′) := σ2 d , i=1 (1 +√5hi+ 5h2i/3) exp(− √ 5hi), hi= |xi− x′i|/ℓi. C Θ= (σ, ℓ1, . . . , ℓd) FN f (Y | FN) Θ ∥g − µ ∥L2 f g f "mα, α ∈ Nd # mα(x) := xα11× · · · × xαdd, x ∈ Dd, L2(D d, R) M f M "mα, α ∈ Nd # ∥g − µ ∥L2
" mα, α ∈ Nd # " mα, α ∈ Nd # r P(r, d) := ' mα | α ∈ Nd, d -i=1 |αi| ≤ r . . P(r, d) C(r, d) r d C(r, d) = (d + r)!/(d! × r!). M ≤ C(r, d) f N g M M
g x Dd
p, q, u N∗ [a] b (u × C(q, d))
C(p, u)
Y g µ
µ(x; [a], b) :=/m(p,u)([a]m(q,d)(x)), b0, x ∈ D d, C(p, u) C(q, d) m(p,u) m(q,d) P(p, u) P(q, d) m(p,u)1 = m (q,d) 1 = 1.
µ(x; [a], b) =/m(p,u)([a]m(q,d)(x)), b0,
= -0≤|α1|+···+|αu|≤p b(α1,...,αu)× u , i=1 ⎛ ⎝ C(q,d) -k=1 [a]ikm (q,d) k (x) ⎞ ⎠ αi , = -0≤|eα1|+···+|eαd|≤p×q xαe1 1 × · · · × xe αd d 5cαe([a], b; u), u ≥ 1 x ,→ µ(x; [a], b) {P(p × q, d)} C(p, u)+u×C(q, d) ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ [a]i1= 0, C(q,d) -k=1 [a]2ik = 1, 1 ≤ i ≤ u,
2 ≤ k ≤ C(q, d) ([a]1k, . . . , [a]uk) C(q, d) − 1 [a] a [a]m(q,d)(x) = [P(q,d)(x)]a. [a] b g Y Y | a, b, Θ ∼ (µ(a, b), C(Θ)), µ(x; a, b) :=/m(p,u)([P(q,d)(x)]a), b0, x∈ Dd, Θ d+1 C a (a⋆, b⋆ , Θ⋆) (a⋆, b⋆, Θ⋆) = arg max (a,b,Θ)∈S − 1 2 ' N log(2π) + log( ([C(Θ)])) + (Y − M(a, b))T[C(Θ)]−1(Y − M(a, b)) . ,
M(a, b) :=%µ(x(1); a, b), . . . , µ(x(N ); a, b)&= [M(a)]b,
[M(a)] :=6m(p,u)([P(q,d)(x(1))]a) · · · m(p,u)([P(q,d)(x(N ))]a)7T,
S RC(q,d)−1× RC(p,u)× Rd+1
a [C(Θ)]
b,→ µ(·; ·, b) µ(x; a, b) a⋆ b⋆ µ(x; a, b) ≈89h(1)(x; a⋆, b⋆ ), h(2)(x; a⋆): , (a − a⋆, b);, h(1)(x; a⋆, b⋆ ) = [P(q,d)(x)]T[D([P(q,d)(x)]a⋆)]Tb⋆ , h(2)(x; a⋆) = m(p,u)([P(q,d)(x)]a⋆), [D(z)] := <∂m(p,u) ∂z (z) = , z ∈ Ru, < ∂m(p,u) ∂z (z) = kj := ∂m (p,u) k ∂zj (z), 1 ≤ j ≤ u, 1 ≤ k ≤ C(p, u), z ∈ R u. β := (a − a⋆, b) f := 9h(1)(·; a⋆, b⋆), h(2) (·; a⋆): a⋆ b⋆ Θ⋆ Y | β ∼ (⟨f, β⟩ , C), (Y | FN) g f g β
C(p × q, d) # (d, p, q, u = 1) # (d, p, q, u = d) C(p × q, d) # (d, p, q, u) = C(p, u)+(C(q, d)−1)−u q= p = 3 d∈ {1, 2, 5, 10, 20} u∈ {1, d} Y • d > 1 p u q q p u q g p = 1 u = d g q x g • # (d, p, q, u) = C(p, u) + (C(q, d) − 1) − u C(p × q, d) x b a ,→ µ(x; a, b)
∥g − $g∥L2 $g L2(Dd, R)
N u p q (a⋆, b⋆, Θ⋆)
L
(a, b, Θ) S
L(a, b, Θ) = log( ([C(Θ)])) + (Y − M(a, b))T[C(Θ)]−1
(Y − M(a, b)),
(a, b, Θ) S
L(a, b (a, Θ), Θ) ≤ L(a, b, Θ),
b (a, Θ) = %[M(a)]T[C(Θ)]−1[M(a)]&−1[M(a)]T[C(Θ)]Y,
[M(a)] ⎧ ⎨ ⎩ (a⋆, Θ⋆) = arg min (a,Θ)L(a, Θ),
b⋆ =%[M(a⋆)]T[C(Θ⋆)]−1[M(a⋆)]&−1[M(a⋆)]T[C(Θ⋆)]Y,
L(a, Θ) := L(a, b (a, Θ), Θ). (a, Θ) ,→ L(a, Θ)
a Θ
a Θ L(a, Θ)
ε L
L
(a⋆, Θ⋆)
L1 = 0 L2= +∞ a∗= (1, . . . , 1)/ ∥(1, . . . , 1)∥
|L2− L1| > ε
L1= L2
Θ∗ = arg maxΘL(a∗, Θ) a∗ = arg maxaL(a, Θ∗)
L2= min(L2, L(a∗, Θ∗)) a⋆≈ a∗ Θ⋆ ≈ Θ∗ L p q u $g (x) $g (x) = ⟨f(x; a⋆, Θ⋆), β⋆(a⋆, Θ⋆ )⟩ + r(x; Θ⋆)T[C(Θ⋆)]−1 (Y − [F (a⋆, Θ⋆)]β⋆(a⋆, Θ⋆)) , β⋆(a⋆, Θ⋆) := ([F (a⋆, Θ⋆)]T[C(Θ⋆)]−1[F (a⋆, Θ⋆)])−1[F (a⋆, Θ⋆)]T[C(Θ⋆)]−1Y, g(x) x Dd • a⋆ Θ⋆ [F (a⋆, Θ⋆)]T[C(Θ⋆)]−1[F (a⋆, Θ⋆)] • Y • x ,→ f(x; a⋆, Θ⋆) 9 h(1)(·; a⋆, b (a⋆, Θ⋆)), h(2) (·; a⋆):
• [F (a⋆, Θ⋆ )] := [f (x(1); a⋆, Θ⋆ ) · · · f(x(N ); a⋆, Θ⋆ )] f(·; a⋆, Θ⋆) • 1 ≤ n, m ≤ N [C(Θ⋆)]nm = C(x(n), x(m)) rn(x; Θ⋆) = C(x, x(n)) C Θ⋆ L2 ∥g − $g ∥ L2 g > >g − $g >>2 L2 ≈ ϵ 2 := 1 N N -n=1 % g(x(n)) − $g−n(x(n)) &2 , 1 ≤ n ≤ N $g−n $g N − 1 X(−n) := ⎧ ⎪ ⎨ ⎪ ⎩ " x(2), . . . , x(N )# n = 1, " x(1), . . . , x(N −1)# n = N, " x(1), . . . , x(n−1), x(n+1), . . . , x(N )# ϵ2 1 ≤ n ≤ N g(x(n)) − $g−n(x(n)) = ([ $C(a⋆, Θ⋆)]Y) n [ $C(a⋆, Θ⋆)] nn , [ $C(a⋆, Θ⋆ )] = [C(Θ⋆)]−1− [C(Θ⋆)]−1[F(a⋆, Θ⋆ )][C(Θ⋆)]−1, [F(a⋆, Θ⋆)] := [F (a⋆, Θ⋆)]([F (a⋆, Θ⋆)]T[C(Θ⋆)]−1[F (a⋆, Θ⋆)])−1[F (a⋆, Θ⋆)]T. ϵ2
ϵ2 ≈ $ϵ2 := 1 N N -n=1 $ e2 n, $e2n:= ' ([ $C(a⋆, Θ⋆)]Y) n [ $C(a⋆, Θ⋆)] nn .2 . a⋆ Θ⋆ a Θ 1 ≤ n ≤ N L(a, Θ) = L−n(a, Θ) + ([ 5C(a, Θ)]Y)2 n [ 5C(a, Θ)]nn ,
[ 5C(a, Θ)] = [C(Θ)]−1"[I] − [M(a)]([M(a)]T[C(Θ)]−1[M(a)])−1[M(a)]T[C(Θ)]−1#,
L−n(a, Θ) L(a, Θ) N − 1 X(−n) a⋆ Θ⋆ {(ai, Θi), 1 ≤ i ≤ N } N a Θ L 1 ≤ n ≤ N a⋆ Θ⋆ a⋆ −n Θ⋆ −n (a⋆ −n, Θ ⋆ −n) = arg max (a,Θ)∈{(ai,Θi), 1≤i≤N } L−n(a, Θ). 5ϵ > >g − $g >>2 L2 ≈ 5ϵ 2 := 1 N N -n=1 5e2n, 5e2n:= ' ([ $C(a⋆ −n, Θ ⋆ −n)]Y)n [ $C(a⋆ −n, Θ⋆−n)]nn .2 . p q u p q u 5ϵ2 u d
p q u g p q u g p u q q a a C(q, d) − 1 − u p u p u a⋆ a L(a, Θ) g d g $g $g g g ε2 ε2 ε2 =>>g − $g >>2L2/ ∥g∥ 2 L2, ε2 =>>g − $g >>2 L2/ ∥g∥ 2 L2. p p u q d = 1 Dd = [−1, 1] g • g(x) = P2◦ P1(x)
• g(x) = sin((x + 1)3) • g(x) = sin(20x) cos(2x) x [−1, 1] ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ P1(x) = 5 -i=1 c(1)i x i−1, c(1) = √(0, −0.03, 0.5, −0.4, −0.5) 0.032+ 0.52+ 0.42+ 0.52, P2(x) = 5 -i=1 c(2)i x i−1, c(2) = (−0.1, 0.2, 0.7, −0.2, −0.2). ε2 ε2 N g N 0 ≤ p ≤ 20, 0 ≤ p, q ≤ 10, u = 1. N g N g g • d = 2 0 ≤ p ≤ 20 0 ≤ p ≤ 6 0 ≤ q ≤ 10 1 ≤ u ≤ d g : ? [−1, 1]2 x → ,→ [−1, 1] g (x) = (1 − x2 1) cos(7x1) × (1 − x22) sin(5x2) .
5 10 15 20 10−4 10−3 10−2 10−1 100 N g(x) = P2◦ P1(x) 4 6 8 10 12 14 10−4 10−3 10−2 10−1 100 N g(x) = sin((x + 1)3 ) 10 15 20 25 30 10−4 10−3 10−2 10−1 100 N g(x) = sin(20x) cos(2x) L2 N N ε2 ε2 N
−1 −0.5 0 0.5 1 0 0.5 1 x g −1 −0.5 0 0.5 1 0 0.5 1 x g (p, u, q) = (4, 1, 4) −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 x g −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 x g (p, u, q) = (4, 1, 4) −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 x g −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 x g (p, u, q) = (6, 1, 7) g(x) = P2◦ P1(x) N = 15 g(x) = sin((x + 1)3) N = 11 g(x) = sin(20x) cos(2x) N = 20 g x g g p u q %
• d = 3 0 ≤ p ≤ 20 0 ≤ p ≤ 3 0 ≤ q ≤ 10 1 ≤ u ≤ d g : ? [−π, π]3 x= (x1, x2, x3) → ,→ R
g (x) = sin(x1) + 7 sin(x2)2+ 0.1x43sin(x1) .
• d = 6 0 ≤ p ≤ 10 0 ≤ p ≤ 3 0 ≤ q ≤ 10 1 ≤ u ≤ d g : ? [−1, 1]6 x → ,→ R g (x) = g(1)◦ g(2)(x), g(1)(z) = 0.1 cos @ 6 -i=1 zi A + 6 -i=1 z2i, z ∈ R 6,
g(2)(x) = (cos(πx1+ 1), cos(πx2+ 2), . . . , cos(πx6+ 6)) .
ε2 ε2 N L2 N u g ∥g − $g ∥L2 ε ε ∥g − $g ∥L2 $ ε ε5 N = 100 g N g $ ε ε5 ∥g − $g ∥L2
40 60 80 100 120 140 10−4 10−3 10−2 10−1 100 N g(x) = g (x) 30 40 50 60 70 80 90 100 10−8 10−6 10−4 10−2 100 N g(x) = g (x) 50 100 150 200 10−3 10−2 10−1 N g(x) = g (x) L2 N N ε2 ε2 u= 1 ε2 1 ≤ u ≤ d
5 ε ∥g − $g ∥L2 a⋆ a⋆ $ ε 5ε d
(1,1) (1,2) (1,3) (1,4) (1,5) (2,1) (2,2) (2,3) (2,4) (2,5) (3,1) (3,2) (3,3) (3,4) (3,5) (4,1) (4,2) (4,3) (4,4) (4,5) 10−6 10−5 10−4 10−3 10−2 10−1 100 101 $ ε (1,1) (1,2) (1,3) (1,4) (1,5) (2,1) (2,2) (2,3) (2,4) (2,5) (3,1) (3,2) (3,3) (3,4) (3,5) (4,1) (4,2) (4,3) (4,4) (4,5) 10−6 10−5 10−4 10−3 10−2 10−1 100 101 5 ε ∥g − $g ∥L2 $ε 5 ε N = 100 u= d 1 ≤ p ≤ 4 1 ≤ q ≤ 5 ∥g − $g ∥L2 ($e2 n, 1 ≤ n ≤ N) (5e 2 n, 1 ≤ n ≤ N)