Resampling-based confidence regions and multiple tests

(1)

Resampling-based confidence regions and multiple tests

Sylvain Arlot^1,2,3

joint work with Gilles Blanchard⁴ Etienne Roquain´ ⁵

1CNRS

2´Ecole Normale Sup´erieure de Paris

3Willow, Inria Paris-Rocquencourt

4Universit¨at Potsdam, Institut f¨ur Mathematik

5Universit´e Pierre et Marie Curie, LPMA

JSTAR 2010, Statistiques en grande dimension, Rennes October, 21st 2010

(2)

Model

Observations: Y= (Y¹, . . . ,Yⁿ) =







Y₁¹ . . . Y₁ⁿ ... ... ... ... Y_K¹ . . . Y_Kⁿ







Y¹, . . . ,Yⁿ∈R^K i.i.d. symmetric Unknown meanµ= (µk)k

Unknown covariance matrix Σ n K

Aims: Find a confidence region forµ or {ks.t.µ_k 6= 0}

(3)

Model







Y₁¹ . . . Y₁ⁿ ... ... ... ... Y_K¹ . . . Y_Kⁿ







(4)

Model







Y₁¹ . . . Y₁ⁿ ... ... ... ... Y_K¹ . . . Y_Kⁿ







(5)

Illustration (1): K = 16384 n, spatially correlated noise

20 40 60 80 100 120

20

40

60

80

100

120

b= 2

20 40 60 80 100 120

20

40

60

80

100

120

b = 12

20 40 60 80 100 120

20

40

60

80

100

120

b= 6

20 40 60 80 100 120

20

40

60

80

100

120

b= 40

(6)

Illustration (2): K = 16384 n, textured noise

(7)

Illustration (3): neuroimaging and microarrays

(neural activity) (gene expression levels)

(8)

Multiple simultaneous hypothesis testing

For everyk we test: H_0,k: “µ_k = 0” againstH_1,k: “µ_k 6= 0”.

Amultiple testing procedure rejects:

R(Y)⊂ {1, . . . ,K}.

Type I errors measured by the Family Wise Error Rate:

FWER(R) =P(∃k ∈R(Y) s.t.µk = 0).

⇒build a procedure R such that FWER(R)≤α?

strong controlof the FWER: ∀µ∈R^K power: Card(R(Y)) as large as possible

(9)

Multiple simultaneous hypothesis testing

R(Y)⊂ {1, . . . ,K}.

(10)

Multiple simultaneous hypothesis testing

R(Y)⊂ {1, . . . ,K}.

(11)

Multiple simultaneous hypothesis testing

R(Y)⊂ {1, . . . ,K}.

(12)

Multiple simultaneous hypothesis testing

R(Y)⊂ {1, . . . ,K}.

(13)

Thresholding

Reject R(Y) ={ks.t.√

n|Y_k|>t}

where

Y= _n¹Pn

i=1Yⁱ empirical mean

t =t_α(Y) threshold (independent ofk ∈ {1, . . . ,K})

FWER(R) =P(∃k s.t. µ_k = 0 and√

n|Y_k|>t)

=P(√

n sup

ks.t.µk=0

|Y_k|>t)

≤P(√ nsup

k

|Y_k|>t)

=P

kY−µk_∞>tn^−1/2

So,L^∞ confidence region ⇒ control of the FWER

(14)

Thresholding

Reject R(Y) ={ks.t.√

n|Y_k|>t}

where

Y= _n¹Pn

i=1Yⁱ empirical mean

t =t_α(Y) threshold (independent ofk ∈ {1, . . . ,K})

FWER(R) =P(∃k s.t. µ_k = 0 and√

n|Y_k|>t)

=P(√

n sup

ks.t.µk=0

|Y_k|>t)

≤P(√ nsup

k

|Y_k|>t)

=P

kY−µk_∞>tn^−1/2

So,L^∞ confidence region ⇒ control of the FWER

(15)

Bonferroni threshold, Gaussian case

Union bound:

FWER(R)≤P ∃k s.t. √ n

Y_k−µ_k >t

≤Ksup

k P(√

n|Y_k −µ_k|>t)

≤2KΦ(t/σ) ,

where Φ is the standard Gaussian upper tail function.

Bonferroni’s threshold: t_α^Bonf=σΦ⁻¹(α/(2K)).

deterministic threshold

too conservative if there are strong correlations between the coordinates Y_k

⇒how to do better ?

(16)

Bonferroni threshold, Gaussian case

Union bound:

FWER(R)≤P ∃k s.t. √ n

Y_k−µ_k >t

≤Ksup

k P(√

n|Y_k −µ_k|>t)

≤2KΦ(t/σ) ,

where Φ is the standard Gaussian upper tail function.

Bonferroni’s threshold: t_α^Bonf=σΦ⁻¹(α/(2K)).

deterministic threshold

too conservative if there are strong correlations between the coordinates Y_k

⇒how to do better ?

(17)

Goal

For everyp∈[1,+∞] , find a threshold t_α(Y) such that

∀µ∈R^K P

√nkY−µk_p>t_α(Y)

≤α .

⇒L^p confidence ball for µat level α (FWER(R)≤αif p= +∞) Non-asymptotic: ∀K,n

General correlations Assumptions:

(Gauss) Yⁱ ∼ N(µ,Σ) withσ = (Σ_k_,k)_k_=1...K known (SB)Yⁱ −µ∼µ−Yⁱ (symmetry) and∀k,

Y_kⁱ

≤M a.s.

Ideal threshold: t =q^?_α, (1−α)-quantile ofD √

nkY−µk_p q_α^? depends on D(Y) unknown⇒ estimated byresampling.

(18)

Goal

∀µ∈R^K P

≤α .

Y_kⁱ

≤M a.s.

(19)

Goal

∀µ∈R^K P

≤α .

Y_kⁱ

≤M a.s.

(20)

Classical procedures and known results

Parametric statistics: no, becauseK²nK.

Asymptotic results (e.g. [van der Vaart and Wellner 1996]):

not valid, because K n.

Holm’s multiple testing procedure: better than Bonferroni, but unefficient with strong correlations

(Multiple) tests by symmetrization:

do not work if H_0,k ={µ_k ≤0}. do not give confidence regions.

not translation-invariant ⇒less powerful if kµk∞ is large

(21)

Classical procedures and known results

not translation-invariant ⇒less powerful if kµk∞ is large

(22)

Classical procedures and known results

nottranslation-invariant ⇒less powerful if kµk∞ is large

(23)

Resampling principle [Efron 1979 ; ...]

Sample Y¹, . . . ,Yn resampling−→ (W1,Y¹), . . . ,(Wn,Yⁿ) weighted sample

“Yⁱ is keptWi times in the resample”

Weight vector: (W1, . . . ,Wn), independent ofY

Example 1: Efron’s bootstrap ⇔ n-sample with replacement

⇔(W1, . . . ,Wn)∼ M(n;n⁻¹, . . .n⁻¹)

Example 2: Rademacher weights: W_i i.i.d. ∼ ¹₂δ−1+¹₂δ₁ ⇔ subsampling, with subsample size ≈n/2

Heuristics: D(sample|true distribution)≈ D(resample|sample)

(24)

Resampling principle [Efron 1979 ; ...]

⇔(W1, . . . ,Wn)∼ M(n;n⁻¹, . . .n⁻¹)

(25)

Resampling principle [Efron 1979 ; ...]

⇔(W1, . . . ,Wn)∼ M(n;n⁻¹, . . .n⁻¹)

(26)

Resampling principle [Efron 1979 ; ...]

⇔(W1, . . . ,Wn)∼ M(n;n⁻¹, . . .n⁻¹)

(27)

Concentration method

kY−µk_p concentrates around its expectation, standard-deviation ≤ kσk_pn^−1/2

Estimate E

kY−µk_p

by resampling

⇒q_α^conc(Y) =cst×√ nE

kY_W −W Yk_p|Y

+ remainder(kσk_p, α,n)

Works well if expectations (∝p

log(K)) are larger than fluctuations (∝Φ⁻¹(α/2))

(28)

Concentration method

kY−µk_p concentrates around its expectation, standard-deviation ≤ kσk_pn^−1/2

Estimate E

kY−µk_p

by resampling

⇒q_α^conc(Y) =cst×√ nE

kY_W −W Yk_p|Y

+ remainder(kσk_p, α,n)

Works well if expectations (∝p

log(K)) are larger than fluctuations (∝Φ⁻¹(α/2))

(29)

Quantile method

Ideal threshold: q^?_α= (1−α)-quantile of D √

nkY−µk_p

⇒Resampling estimate of q^?_α: qα^quant(Y) =(1−α)-quantile ofD √

nkY_W −W Yk_p|Y

with W := 1 n

n

X

i=1

Wi

YW := 1 n

n

X

i=1

WiYⁱ Resampling empirical mean qα^quant(Y) depends only onY⇒ can be computed with real data

(30)

Quantile method

nkY−µk_p

nkY_W −W Yk_p|Y

with W := 1 n

n

X

i=1

Wi

YW := 1 n

n

X

i=1

(31)

Quantile method

nkY−µk_p

nkY_W −W Yk_p|Y

with W := 1 n

n

X

i=1

Wi

YW := 1 n

n

X

i=1

(32)

Concentration theorem

Theorem

Assume(Gauss) and W Rademacher. For everyα ∈(0,1), q_α^conc,1(Y) :=

√nE

kY_W −W Yk_p|Y

B_W +kσk_pΦ⁻¹(α/2) C_W

√nB_W + 1

satisfies

P

√nkY−µk_p >q^conc,1_α (Y)

≤α withσ :=

q

var(Y_k¹)

1≤k≤K, and

BW :=E 1 n

n

X

i=1

“

Wi−W”2!1/2

= 1− O(n^−1/2) and CW = 1

Main tool: Gaussian concentration theorem [Cirel’son, Ibragimov and Sudakov, 1976]

(33)

Remarks

Valid for quite general weights(withB_W andC_W are independent of K and easy to compute).

Similar result under assumption (SB) (with larger constants).

k·k_p can be replaced by sup_k(·)₊ ⇒ one-sided multiple tests Almost deterministicthreshold:

⇒ ifp =∞,qα^conc,2(Y)≈min

t_α^bonf,q^conc,1α (Y)

still has a FWER ≤α.

(34)

Remarks

Valid for quite general weights(withB_W andC_W are independent of K and easy to compute).

Similar result under assumption (SB) (with larger constants).

k·k_p can be replaced by sup_k(·)₊ ⇒ one-sided multiple tests Almost deterministicthreshold:

⇒ ifp =∞,qα^conc,2(Y)≈min

t_α^bonf,q^conc,1α (Y)

still has a FWER ≤α.

(35)

Practical computation

Monte-Carlo approximation with W¹, . . . ,W^B only (with an additionnal term of order B^−1/2)

Alternatively, V-fold cross-validation weights

⇒ computation time ∝V, accuracy ∝CWB_W⁻¹≈p n/V. Estimation of σ: under (Gauss), if

cσk :=

v u u t 1 n

n

X

i=1

Y_kⁱ −Y2

then for everyδ ∈(0,1) , P

kσk_p≤

Cn− 1

√nΦ⁻¹ δ

2

kbσk_p

≥1−δ , with Cn= 1− O(n⁻¹) .

(36)

Practical computation

Monte-Carlo approximation with W¹, . . . ,W^B only (with an additionnal term of order B^−1/2)

Alternatively, V-fold cross-validation weights

⇒ computation time ∝V, accuracy ∝CWB_W⁻¹≈p n/V. Estimation of σ: under (Gauss), if

cσk :=

v u u t 1 n

n

X

i=1

Y_kⁱ −Y2

then for everyδ ∈(0,1) , P

kσk_p≤

Cn− 1

√nΦ⁻¹ δ

2

kbσk_p

≥1−δ , with Cn= 1− O(n⁻¹) .

(37)

Simulations: p = +∞, n = 1000, K = 16384, σ = 1

(38)

Simulations: p = +∞, n = 1000, K = 16384, σ = 1

(39)

Quantile method

Rademacher weights only: W_i i.i.d. ∼ ¹₂δ−1+¹₂δ1

q^quantα (Y) = (1−α)-quantile of D √

nkY_W −W Yk_p Y

Heuristics ⇒ should satisfyP kY−µk_p >qα^quant(Y)

≤α

(40)

Quantile theorem: need for a supplementary term

Theorem

Y symmetric. W Rademacher. α, δ, γ∈(0,1).

If f is a non-negative threshold with level bounded byαγ: P

√nkY−µk_p >f(Y)

≤αγ Then,

q_α^quant+f(Y) =q^quantα(1−δ)(1−γ)(Y) +

r2 log(2/(δα)) n f(Y) yields a level bounded byα:

P √

nkY−µk_p>q_α^quant+f(Y)

≤α

(41)

Remarks

Uses only the symmetry of Y around its mean

k·k_p can be replaced by sup_k(·)₊ ⇒ one-sided multiple tests The supplementary threshold f only appears in a second-order term.

(Gauss) ⇒ three thresholds:

takef amongq_αγ^Bonf (if p= +∞),q^conc,1αγ andqαγ^conc,2

(SB)⇒ f =q_αγ^conc,SB

In simulation experiments,f is almost unnecessary.

q^quantα(1−δ)(1−γ)(Y) can be replaced by aMonte-Carlo estimated quantile (i.e., simulate onlyW¹, . . . ,W^B)

⇒ loose at most (B+ 1)⁻¹ in the level, nothing if α(1−δ)(1−γ)(B+ 1)∈N.

(42)

Remarks

q^quantα(1−δ)(1−γ)(Y) can be replaced by aMonte-Carlo estimated quantile (i.e., simulate onlyW¹, . . . ,W^B)

(43)

Remarks

q^quantα(1−δ)(1−γ)(Y) can be replaced by a Monte-Carlo estimated quantile (i.e., simulate only W¹, . . . ,W^B)

(44)

Simulations: n = 1000, K = 16384, σ = 1, p = +∞

(45)

Simulations: without the additive term?

(46)

Simulations: role of p (n = 1000, p = 16)

(47)

Simulations: role of p (n = 1000, p = 10)

(48)

Simulations: role of p (n = 1000, p = 2)

(49)

Simulations: role of p (n = 1000, p = 1)

(50)

Quantile approach vs. Symmetrization

Defineq^symα (Y,p) as the (1−α)-quantile of D √

nkY_Wk_p Y

Symmetrization argument: if Y symmetric andW Rademacher, then

P kY−µk_p >q_α^sym(Y−µ,p)

≤α

since

(Y−µ)_W = 1 n

n

X

i=1

W_i(Yⁱ −µ)∼ 1 n

n

X

i=1

(Yⁱ −µ) =Y−µ . q^symα (Y−µ,p) unknown⇒ replacingµbyY leads to

q^quant_α (Y,p) =q_α^sym(Y−Y,p)

(51)

Quantile approach vs. Symmetrization

nkY_Wk_p Y

≤α

since

(Y−µ)_W = 1 n

n

X

i=1

n

X

i=1

(Yⁱ −µ) =Y−µ .

q^symα (Y−µ,p) unknown⇒ replacingµbyY leads to q^quant_α (Y,p) =q_α^sym(Y−Y,p)

(52)

Quantile approach vs. Symmetrization

nkY_Wk_p Y

≤α

since

(Y−µ)_W = 1 n

n

X

i=1

n

X

i=1

(Yⁱ −µ) =Y−µ .

q^symα (Y−µ,p) unknown⇒ replacingµbyY leads to q^quant_α (Y,p) =q_α^sym(Y−Y,p)

(53)

Multiple testing with the uncentered quantile threshold

Ift≥qα^sym((Y_k)_k_s.t._µ_k₌₀,+∞) , then

FWER(R) =P(∃k s.t. µk = 0 and√

n|Y_k|>t)

=P(√

n sup

ks.t.µk=0

|Y_k|>t)≤α

by symmetry ofY. This holds in particular for

qquant. uncent.

α (Y) :=q^sym_α (Y,+∞)≥q^sym_α ((Y)_k_s.t._µ_k₌₀,+∞)

⇒can be used for multiple testing, but more conservative, especially when the signalµis strong

(54)

Simulations: n = 1000, K = 16384, σ = 1, 0 ≤ µ

_k

≤ 2.9

(55)

Step-down procedure [Holm 1979 ; Westfall and Young 1993 ; Romano and Wolf 2005]

Reorder the coordinates:

Y_σ(1)

≥ · · · ≥ Y_σ(K)

Define the thresholds t_k =t(Y_σ(k), . . . ,Y_σ(K)) for k = 1, . . . ,K

Definekb= max

ks.t.∀k⁰≤k,Y_σ(k⁰₎>tk⁰(Y) Reject H_0,k for all k ≤bk

⇒this procedure has a FWER controlled by α if each tk has (use thattK=t((Y_k)_k∈K) is a non-decreasing function ofK).

(56)

Simulations: n = 1000, K = 16384, σ = 1, 0 ≤ µ

_k

≤ 2.9

(57)

Simulations: power (0 ≤ µ

_k

≤ 2.9)

(58)

Hybrid approach, computational cost

Idea: (centered) quantile better when the signal is strong, uncentered quantile better for weak signals

First step: use q_α^quant+Bonf(Y) to reject a first set of hypotheses.

Second step: apply the step-down procedure associated with qquant. uncent.

α(1−γ) (Y) to the remaining hypotheses.

Goal: reduce the computational cost / increase the power for a given number of iterations

(59)

Hybrid approach, computational cost

(60)

Hybrid approach, computational cost

(61)

Hybrid approach: average number of iterations

3.5 4 4.5 5 5.5 6 6.5 7 7.5

5 10 15 20 25 30

Av. number of iterations

Signal range (dB) s-d. quant. uncent.

hybrid approach

(62)

With early stopping: power vs. number of iterations

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

5 10 15 20 25 30

Av. (power / max. power)

Signal range (dB) sdqu, t=1

sdqu, t=2 sdqu, t=3 hybrid, t=1 hybrid, t=2 hybrid, t=3

(63)

Conclusion

Two resampling-based procedures:

concentration (almost deterministic threshold) quantile (related to symmetrization techniques)

⇒Multiple testing + Confidence regions

FWER / level control (non-asymptotic, K may ben) very general correlation structures allowed

Simulations: efficient in presence of correlations step-down procedures are possible

Open problems:

quantile threshold withoutf? with other weights?

with non-symmetric Y?

(64)

Conclusion

Open problems:

(65)

Conclusion

Open problems: