Optimal use of auxiliary information : information geometry and empirical process.

HAL Id: hal-03276753

https://hal.archives-ouvertes.fr/hal-03276753

Preprint submitted on 2 Jul 2021

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

geometry and empirical process.

Sofiane Arradi-Alaoui

To cite this version:

Sofiane Arradi-Alaoui. Optimal use of auxiliary information : information geometry and empirical process.. 2021. �hal-03276753�

GEOMETRY AND EMPIRICAL PROCESS

A PREPRINT

Sofiane Arradi-Alaoui^∗

July 1, 2021

ABSTRACT

We incorporate into the empirical measurePnthe auxiliary information given by a finite collection of expectation in an optimal information geometry way. This allows to unify several methods exploiting a side information and to uniquely define an informed empirical measurePn^I. These methods are shown to share the same asymptotic properties. Then we study the informed em- pirical processp

n(P^In−P) subject to a true information. We establish the Glivenko-Cantelli and Donsker theorems forP^Inunder minimal assumptions and we quantify the asymptotic uniform variance reduction. Moreover, we prove that the informed empirical process is more concentrated than the empirical processp

n(Pn−P) for all largen. Finally, as an illustration of the variance reduction, we apply some of these results to the informed empirical quantiles.

Keywords:information geometry, side information, auxiliary information, empirical processes, empirical likelihood, uniform central limit theorem, variance reduction.

AMS Subject Classification:62B11 ; 62G30 ; 62G20 ; 60F17.

1 Introduction

We call auxiliary information –or side information– any information external to an observed statistical experiment that concerns the underlying distribution. For instance, in order to improve the quality of a survey analysis, it is customary to incorporate any reliable auxiliary information available at the time of the survey such as the knowledge of one or more parameters of this population determined exactly by an exhaustive census. This principle finds its origin several centuries ago, according to [8]. Indeed, around 1740 the magistrate Jean-Baptiste François de La Michodière wanted to estimate the size of the French population by assuming that the number of marriages, births and deaths is proportional to the size of the population. He then introduced the ratio estimator which was validated by Laplace [11]. This method is for instance detailed in [10]. We note that it turns out to be a special case of [17]. More recently, several authors in survey analysis and statistics have worked on the incorporation of auxiliary information after or before sampling – see [9], [12].

In this article, we focus on the auxiliary information which concerns the underlying distribution of the data.

More precisely, we assume that the auxiliary information is given by a finite collection of expectations. In the above mentioned case of a survey the side information can be given by the expectation of a random variable on the population. Rather few systematic analysis have been carried out on how to use at best such an auxiliary information in a general setting, despite the fact that in many case studies such a methodology is used. In [1], the raking-ratio method allows to incorporate the auxiliary information of the probabilities of one or many partitions of a set. This is not a perfect projection of the empirical measure on the set of constraints since it is a sequential procedure, incorporating each information after the other, not simultaneously. However the variance of large classes of estimators simultaneously decreases as the sample size tends to infinity faster than the number of successive

∗Institut de Mathématiques de Toulouse UMR 5219 ; Université Paul Sabatier, France.[email protected] toulouse.fr

partitions. The case of an independent empirical information, as for distributed data, is investigated in [3]. In [17], a general method is proposed to incorporate a general auxiliary information. This approach consists in minimizing the variance over a class of unbiased estimators, that is equivalent to find the smallest dispersion ellipsoid. In [2]

this method is applied to an auxiliary information brought by a finite collection of expectations. In particular, it is shown that this method is better than the raking-ratio [1] with respect to variance reduction. Moreover, in [14] a different method based on empirical likelihood is developed to also incorporate an auxiliary information given by a finite collection of expectations. The latter two methods will be compared and connected through our definition of an informed empirical measure. In [20] the Glivenko-Cantelli and Donsker theorems are established for the empirical likelihood method – in the special case of the class of functionsF=©

1]−∞,t],t∈Rª

. For more general classes, the Glivenko-Cantelli and Donsker theorems are established in [2] under stronger assumptions by using the strong approximation results of [7]. In addition, some works have also been done on U-statistics in the presence of auxiliary information [19] and, more recently, on informed statistical tests [4], [5].

Our contribution is to define and study an informed empirical measure supported by the sample that is optimal in the sense of information geometry. We thus intend to incorporate the auxiliary information given by expectations into the empirical measure itself by defining properly the geometrical setting in which the latter can be projected.

This leads to define two projection measures, the first of which satisfies the same optimization problem as that of the empirical likelihood [14]. Next we prove that it is possible to approximate these two projection measures by a common measure we call the informed empirical measure. This informed empirical measure is far easier to compute numerically than the true projections and turns out to coincide with the adaptive estimator of the measure with auxiliary information defined in [2]. Furthermore we show that these three measures are so close that they share the same asymptotic properties in the sense of empirical process theory – in particular the same limiting Gaussian process. This allows to unify several methods aiming to incorporate a side information, among which those mentioned above. We establish under minimal assumptions the limit theorems for the informed empirical measure indexed by a general class of functions. As a by product this extends the asymptotic result of [20]. Moreover we derive a concentration result which shows that the informed empirical process is always more concentrated than the classical empirical process when the sample size is large enough.

The paper is structured as follows. In Section 3, we introduce the geometrical framework and prove that the set of constraints associated to the auxiliary information has a submanifold structure. Then we show that an optimal method is to minimize the Kullback-Leibler divergence and its dual version on the set of constraints. The readers less familiar with geometrical notions can find reminders on information geometry in the book [6] – or could refer directly to Corollary 7. In Section 4, we study these two optimization problems. More precisely, we discuss the existence and uniqueness of their solution and we give an asymptotic theorem for the Lagrange multipliers. In Section 5, we prove that there exists a common approximation of these solutions which allows to define the informed empirical measure – see Definition 19. Then we study the informed empirical measure’s weights. In Section 6 we establish the Glivenko-Cantelli and Donsker theorems for a general class of functionsF about the informed empirical process under minimal assumptions. We also quantify the asymptotic uniform variance reduction, which justifies the use of auxiliary information. In Section 6.2 we derive a concentration result about the informed empirical process. Finally in Section 7, as an illustration of the variance reduction, we apply these results to the informed empirical quantile and prove that the informed estimator is asymptotically more efficient than the classical empirical quantile. As a special case, we find the same asymptotic result as in [22].

2 Framework

Let (X_n)_n∈N∗ be a sequence of independent and identically distributed random variables (i.i.d.) defined on a probability space (Ω,B,P) and taking values in a measurable space (X,A). The distribution ofX₁is denoted P:=P^X1. Moreover, we assume that an auxiliary informationIaboutPis available. In this article, we focus on the following particular case. We suppose thatIis the information brought by a finite collection of expectations with respect toP. More precisely, letm∈N^∗andg=¡

g₁,· · ·,g_m¢T

:X→R^mbe an integrable function with respect toP. We assume thatIis given by

P g:= Z

Xg d P:= µZ

Xg₁d P,· · ·, Z

Xg_md P

¶T

.

We shall assume at times thatP g=0 – otherwise seth=g−P g.

We denote‚n,pƒthe set of integers betweennandpwheren<pare two integers. For eachn∈N^∗, the random data set is denotedZn={X₁,· · ·,X_n} andPnthe empirical measure defined by

Pn= 1 n

n

X

i=1

δXi.

Ifn∈N^∗is fixed then we denoteZ =ZnandP(Z) the set of probability measures onZ with positive weights.

Moreover the informed empirical measure is denotedP^Inand will be defined at Section 5 in Definition 19.

Our notation for stochastic convergences are as follows. Let (Y_n)n∈N^∗a sequence of random variables with values inR^m withm∈N^∗and let (an)n∈N^∗ ⊂R^∗be a real valued sequence. We writeYn =op(an) (resp. oa.s.(an)) if (Y_n/a_n)_n∈N∗tends to 0 in probability (resp. almost surely) asn→ +∞. We writeY_n=O_p(a_n) if (Y_n/a_n)_n∈N∗is tight.

The fact that (Yn)n∈N^∗converges in distribution to a random variableY asn→ +∞is denoted byYn⇒Y.

3 A geometrical approach of auxiliary information

We intend to incorporate optimally an auxiliary information into the empirical measure. The notion of optimality may be debated but the information geometry approach seems to be a coherent and interesting answer to the problem of incorporating an auxiliary information. Assume that an auxiliary informationIaboutPis available. For Q∈P(Z), we denoteQ∼Ithe fact thatQsatisfies the auxiliary informationI. The set of probability measures on Z which satisfyIis defined by

P^I(Z)=©

Q∈P(Z), Q∼Iª .

As mentioned in Section 2, the weights ofQare positive andIis given by a finite collection of expectationsP g. However Section 3.2 remains valid for more general definitions of auxiliary informationI. We have

P^I(Z)=©

Q∈P(Z), Q g=P gª .

Assuming that the basic notions of information geometry are known – such as connection, geodesic etc – we only recall the notion of autoparallel submanifold. LetSbe a manifold,Mbe a submanifold ofSand∇a connection onS. We say thatMis∇-autoparallel if for every vector fieldsX,Y onM,∇XY is also a vector field onM. In this context, sinceP(Z) is a finite mixture model it is also a differential manifold endowed with the dually flat structure (P(Z),g_F,∇⁽¹⁾,∇⁽⁻¹⁾) whereg_F is the Fisher metric,∇⁽¹⁾is the 1-connection and∇⁽⁻¹⁾is the (−1)-connection.

Moreover, the canonical divergence associated toP(Z) is the Kullback–Leibler divergenceK L– see [13].

In order to use the auxiliary informationIlet projectPn∈P(Z) onP^I(Z) in the sense of information geometry.

To define properly this projection we first show thatP^I(Z) is a submanifold, then we recall the projection theorem in information geometry and formulate our existence result.

3.1 Submanifold structure ofP^I(Z)

The following result states thatP^I(Z) is a (n−2)-dimensional submanifold in the casem=1.

Proposition 1. Assume that there exists i 6= j such that g(X_i)6=g¡ X_j¢

and P g belongs to the convex hull of

©g(X₁),· · ·,g(X_n)ª

. Then P^I(Z)is a(n−2)-dimensional submanifold ofP(Z).

Proof. Remind that ]0, 1[ⁿis a submanifold ofRⁿas it is an open set ofRⁿ. Set S =

(

q∈]0, 1[ⁿ,

n

X

i=1

qi=1 )

.

We first show thatS is a (n−1)-dimensional submanifold of ]0, 1[ⁿ. Define the functionθ:]0, 1[ⁿ→Rbyθ(q)= Pn

i=1q_i. Observe thatS =θ⁻¹({1}). So it is enough to prove thatθis a submersion. Indeed,θis differentiable and for allq∈]0, 1[ⁿ,

Dθ(q)(h)=θ(h),h∈Rⁿ.

SoDθ(q) is surjective andθis a submersion, henceS is a (n−1)-dimensional submanifold of ]0, 1[ⁿ. Let endowS with the following global chart (S,π)

π:S →U⊂Rⁿ⁻¹ q7→(q₁,· · ·,q_n−1)

whereU=©

(q1,· · ·,qn−1)∈Rⁿ⁻¹,qi>0,i∈ ‚1,n−1ƒ, Pn−1 i=1qi<1ª

. Similarly endowP(Z) with the following global chart¡

P(Z),ϕ¢

ϕ:P(Z)→U⊂Rⁿ⁻¹ Q7→(q1,· · ·,qn−1).

Next consider the following one to one mapping

ψ:P(Z)→S Q7→q.

Notice thatψ=π⁻¹◦ϕ. Sinceπandϕare diffeomorphims and dimP(Z)=dimS =n−1, we deduce thatψis a diffeomorphim. Soψ⁻¹is also a diffeomorphism and thus an embedding. Observe thatP^I(Z)=ψ⁻¹(E) where E=©

q∈S,Pn

i=1q_ig(X_i)=P gª

is not empty becauseP gbelongs to the convex hull of {g(X₁), ...,g(X_n)}. So it is enough to prove thatE is a (n−2)-dimensional submanifold ofS. For this, it is sufficient to verify that

f :S →R q7→

n

X

i=1

qig(Xi)

is a submersion. Letγ:=f◦π⁻¹:U→Rbe defined, for allq∈U, by γ(q)=

n−1X

i=1

q_ig(X_i)+ Ã

1−

n−1X

i=1

q_i

! g(X_n).

Soγis differentiable and for allq∈U

Dγ(q)=(g(X₁)−g(X_n),· · ·,g(X_n−1)−g(X_n)).

Since there existsi6=jsuch thatX_i6=X_j, we deduce thatDγ(q) is surjective. Thereforef is a submersion andEis a (n−2)-dimensional submanifold. We conclude thatP^I(Z)=ψ⁻¹(E) is a (n−2)-dimensional submanifold, asψ⁻¹ is an embedding.

Proposition 1 can be generalised for a vector of functionsg=(g1,· · ·,gm)^T withm∈ ‚1,n−1ƒ. Assume thatP g=0.

Set

∀j∈[|1,m|],N_j(X)=(g_j(X₁)−g_j(X_n),· · ·,g_j(X_n−1)−g_j(X_n))^T,

∀j∈[|1,m|],g_j(X)=(g_j(X₁),· · ·,g_j(Xn))^T. Observe that

dim¡

V ec t(g(X₁)−g(X_n),· · ·,g(X_n−1)−g(X_n))¢

=dim¡

V ec t((N_j(X))_1≤j≤m)¢

. (1)

We are ready to state the main result of Section 3.

Proposition 2. Assume that0belongs to the convex hull of©

g(X1), ...,g(Xn)ª

and the following equality of dimensions, l:=dim¡

V ec t(g₁(X), ...,gm(X))¢

=dim¡

V ec t(g(X₁)−g(Xn),· · ·,g(Xn−1)−g(Xn))¢

≤m.

Then P^I(Z)is a(n−1−l)-dimensional submanifold ofP(Z).

Remark 3. A sufficient condition to satisfy the equality of dimensions in Proposition 2 is dimV ec t¡

(1,· · ·, 1)^T,M1(X),· · ·,Mm(X)¢

=m+1 (2)

with for all k∈[|1,m|], M_k(X)=g_k(X)=¡

g_k(X₁),· · ·,g_k(X_n)¢T

.

Proof. We keep the same notations as in the previous proof. It remains to prove thatE=©

q∈S,Pn

i=1q_ig(X_i)=0ª is a (n−1−l)-dimensional submanifold. For that, we need a technical lemma.

Lemma 4. Let x¹, ...,x^k∈Rⁿwith k∈ ‚1,n−1ƒand n∈N^∗. Define, for j∈ ‚1,kƒ,

˜ x^j=

³

x₁^j−x_n^j,· · ·,x_n−1^j −x_n^j´T

∈Rⁿ⁻¹. Then

dim³

V ec t( ˜x¹,· · ·, ˜x^k)´

≤dim³

V ec t(x¹,· · ·,x^k)´ . Moreover if l:=dim¡

V ec t( ˜x¹,· · ·, ˜x^k)¢

=dim¡

V ec t(x¹,· · ·,x^k)¢

then there exists a subset J⊂ ‚1,kƒof size l such that dim³

V ec t³

( ˜x^j)_j∈J´´

=dim³ V ec t³

(x^j)_j∈J´´

. Proof. Ifk=dim¡

V ec t(x¹,· · ·,x^k)¢

then there is nothing to prove. Assume that dim¡

V ec t(x¹,· · ·,x^k)¢

<k. So there existsj∈ ‚1,kƒsuch thatx^j=P

i6=jλixⁱ withλ1, ...,λk∈R. So for allr∈ ‚1,nƒ x_r^j−x_n^j=X

i6=j

λix_rⁱ−X

i6=j

λix_nⁱ =X

i6=j

λi(x_rⁱ−x_nⁱ).

In others words ˜x^j=P

i6=jλix˜ⁱ. We can conclude that dim³

V ec t( ˜x¹,· · ·, ˜x^k)´

≤dim³

V ec t(x¹,· · ·,x^k)´ . Now, assume thatl:=dim¡

V ec t( ˜x¹,· · ·, ˜x^k)¢

=dim¡

V ec t(x¹,· · ·,x^k)¢

. So there exists a subsetJ⊂ ‚1,kƒof sizel such that

l=dim³ V ec t³

( ˜x^j)_j∈J´´

. But if dim¡

V ec t¡

(x^j)_j∈J¢¢

<lthen there existsr∈Jsuch thatx^r=P

i∈J,i6=rλixⁱwithλ1, ...,λk∈R. By the above, we deduce that ˜x^r=P

i∈J,i6=rλix˜ⁱ. That contradicts the fact that l=dim³

V ec t³

( ˜x^j)_j∈J´´

.

We can apply Lemma 4 tog₁(X),· · ·,g_m(X). Recall that, by (1), dim¡

V ec t(g(X₁)−g(X_n),· · ·,g(X_n−1)−g(X_n))¢

=dim¡

V ec t((N_j(X))_1≤j≤m)¢ where, forj∈ ‚1,mƒ,

N_j(X)=(g_j(X₁)−g_j(X_n),· · ·,g_j(X_n−1)−g_j(X_n))^T. By the assumption of equality of dimensions we have

l:=dim¡

V ec t(g1(X),· · ·,gm(X))¢

=dim¡

V ec t(g(X1)−g(Xn),· · ·,g(Xn−1)−g(Xn))¢ . Hence by Lemma 4 there exists a subsetJ⊂ ‚1,mƒof sizelsuch that

l=dim¡ V ec t¡

g₁(X), ...,gm(X)¢¢

=dim¡ V ec t¡

(g_j(X))_j∈J¢¢

=dim¡ V ec t¡

(N_j(X))_j∈J¢¢

Denotege=(g_j)_j∈Jand notice that

©Q∈P(Z), Q g=0ª

=©

Q∈P(Z), Qge=0ª .

So, we can select only the constraintsg. For simplicity, in what follows we denotee g=(g_j)_j∈J. Define the function f :S →R^l

q7→

n

X

i=1

qig(Xi).

Let prove thatf is a submersion. Setγ:=f◦π⁻¹:U→R^ldefined for allq∈U γ(q)=

n−1X

i=1

q_ig(X_i)+ Ã

1−

n−1X

i=1

q_i

! g(X_n).

Soγis differentiable and for allq∈U

Dγ(q)=(g(X1)−g(Xn),· · ·,g(Xn−1)−g(Xn)).

Since

rk¡ Dγ(q)¢

=dim¡

V ec t(g(X₁)−g(Xn),· · ·,g(Xn−1)−g(Xn))¢

=l.

We deduce thatDγ(q) is surjective. Thusf is a submersion andEis a (n−1−l)-dimensional submanifold. We can conclude thatP^I(Z)=ψ⁻¹(E) is (n−1−l)-dimensional submanifold becauseψ⁻¹is an embedding.

3.2 Existence of a projection

Recall the projection theorem in information geometry.

Theorem 5. Consider a dually flat manifold S and denote D the canonical divergence of S. Let p∈S and M be a submanifold of S which is∇^∗-autoparallel. Then a necessary and sufficient condition for a point q∈M to satisfy

D(p||q)=min

r∈MD(p||r) (3)

is that the∇-geodesic connecting p to q is orthogonal to M in q. The point q is called∇-projection of p on M . Likewise if M is∇-autoparallel then a necessary and sufficient condition for a point q∈M to satisfy

D^∗(p||q)=min

r∈MD^∗(p||r) (4)

is that the∇^∗-geodesic connecting p to q is orthogonal to M in q and the point q is called the∇^∗-projection of p on M . Moreover, it is possible to relax the autoparallel submanifold assumption.

Proposition 6. Assume that S a dually flat manifold and denote D the canonical divergence of S. Let p∈S and M be a submanifold of S. A necessary and sufficient condition for a point q to be a stationary point of the function D(p||·) :r7→D(p||r)restraint to M (resp. D^∗(p||·) :r7→D^∗(p||r)) is that the∇-geodesic (resp.∇^∗-geodesic) connecting p to q is orthogonal to M in q.

We next apply Theorem 5 and Proposition 6 to define a projectionQ^InofPnonP^I(Z).

Corollary 7. Assume thatP^I(Z)is a submanifold ofP(Z). Then

• IfP^I(Z)is a submanifold∇⁽⁻¹⁾-autoparallel thenQ^Inis the∇⁽¹⁾-projection ofPnonP^I(Z)that is Q^In∈ar g min

Q∈P^I(Z)

K L^∗(Pn||Q)=ar g min

Q∈P^I(Z)

K L(Q||Pn) .

• IfP^I(Z)is a submanifold∇⁽¹⁾-autoparallel thenQn^I is the∇⁽⁻¹⁾-projection ofPnonP^I(Z) Qn^I ∈ar g min

Q∈P^I(Z)

K L(Pn||Q) .

• In the case whereP^I(Z)is autoparallel to neither of these two connections then Q^Inis a stationary point of one of these two maps

Q7−→K L(Pn||Q) , Q7−→K L^∗(Pn||Q) .

Remark 8. Corollary 7 does not determine a unique informed empirical measure by projection and moreover it is generally not easy to check if the submanifoldP^I(Z)is autoparallel.

4 Two measure projections

Assume thatP gbelongs to the convex hull of©

g(X₁),· · ·,g(X_n)ª

and the assumption (2) is verified then by Proposi- tion 2 P^I(Z) is a submanifold. By Theorem 7 we were able to define a projection ofPnonP^I(Z). The next step is to study these two optimization problems

ar g min

Q∈P^I(Z)

K L(Pn||Q), (5)

ar g min

Q∈P^I(Z)

K L(Q||Pn). (6)

Remark that

K L(Pn||Q)=

n

X

i=1

1 nlog

µ 1 nqi

¶

= −log(n)−1 n

n

X

i=1

logq_i, K L(Q||Pn)=

n

X

i=1

q_ilog(nq_i)=logn+

n

X

i=1

q_ilogq_i. Therefore

ar g min

Q∈P^IK L(Pn||Q)=ar g max

Q∈P^I(Z) n

X

i=1

logqi=ar g max

Q∈P^I(Z) n

Y

i=1

qi, ar g min

Q∈P^I(Z)

K L(Q||Pn)=ar g min

Q∈P^I(Z) n

X

i=1

q_ilogq_i.