xTxfollows aχ2distribution withddegrees of freedom, which has meandand variance 2d. It follows that the mean and variance of the random variabler2is
µr2=d σr22 =2d
By the central limit theorem, asd→ ∞,r2is approximately normal with meand and variance 2d, which implies that r2 is concentrated about its mean value ofd. As a consequence, the distance r of a point x to the center of the standard multivariate normal is likewise approximately concentrated around its mean√
d.
Next, to estimate the spread of the distanceraround its mean value, we need to derive the standard deviation ofr from that ofr2. Assuming thatσr is much smaller compared tor, then using the fact that dlogrdr =1r, after rearranging the terms, we have
dr
r =dlogr
=1 2dlogr2
Using the fact thatdlogrdr22 =r12, and rearranging the terms, we obtain dr
r =1 2
dr2 r2
which implies thatdr=2r1dr2. Setting the change inr2equal to the standard deviation ofr2, we havedr2=σr2=√
2d, and setting the mean radiusr=√
d, we have σr=dr= 1
2√ d
√2d= 1
√2
We conclude that for larged, the radiusr (or the distance of a pointxfrom the origin0) follows a normal distribution with mean√
d and standard deviation 1/√ 2.
Nevertheless, the density at the mean distance√
d, is exponentially smaller than that at the peak density because
f (x)
f (0)=exp
−xTx/2 =exp{−d/2}
Combined with the fact that the probability mass migrates away from the mean in high dimensions, we have another interesting observation, namely that, whereas the density of the standard multivariate normal is maximized at the center0, most of the probability mass (the points) is concentrated in a small band around the mean distance of√
dfrom the center.
6.7 APPENDIX: DERIVATION OF HYPERSPHERE VOLUME
The volume of the hypersphere can be derived via integration using spherical polar coordinates. We consider the derivation in two and three dimensions, and then for a generald.
X1 X2
bC
θ1 r
(x1, x2)
Figure 6.9. Polar coordinates in two dimensions.
Volume in Two Dimensions
As illustrated in Figure 6.9, ind =2 dimensions, the point x=(x1, x2)∈R2 can be expressed in polar coordinates as follows:
x1=rcosθ1=rc1 x2=rsinθ1=rs1
wherer= kxk, and we use the notation cosθ1=c1and sinθ1=s1for convenience.
TheJacobian matrixfor this transformation is given as
J(θ1)=
∂x1
∂r
∂x1
∂θ1
∂x2
∂r
∂x2
∂θ1
!
=
c1 −rs1 s1 rc1
The determinant of the Jacobian matrix is called theJacobian. ForJ(θ1), the Jacobian is given as
det(J(θ1))=rc12+rs12=r(c12+s12)=r (6.15) Using the Jacobian in Eq. (6.15), the volume of the hypersphere in two dimensions can be obtained by integration overrandθ1(withr >0, and 0≤θ1≤2π)
vol(S2(r))= Z
r
Z
θ1
det(J(θ1)) dr dθ1
= Zr
0
Z2π 0
r dr dθ1= Zr 0
r dr Z2π
0
dθ1
= r2 2
r
0
·θ1 2π
0 =π r2
6.7 Appendix: Derivation of Hypersphere Volume 177
X1
X2
X3
bC(x1, x2, x3)
r θ1 θ2
Figure 6.10.Polar coordinates in three dimensions.
Volume in Three Dimensions
As illustrated in Figure 6.10, ind=3 dimensions, the pointx=(x1, x2, x3)∈R3can be expressed in polar coordinates as follows:
x1=rcosθ1cosθ2=rc1c2 x2=rcosθ1sinθ2=rc1s2 x3=rsinθ1=rs1
wherer= kxk, and we used the fact that the dotted vector that lies in theX1–X2plane in Figure 6.10 has magnitudercosθ1.
The Jacobian matrix is given as
J(θ1, θ2)=
∂x1
∂r
∂x1
∂θ1
∂x1
∂θ2
∂x2
∂r
∂x2
∂θ1
∂x2
∂θ2
∂x3
∂r
∂x3
∂θ1
∂x3
∂θ2
=
c1c2 −rs1c2 −rc1s2 c1s2 −rs1s2 rc1c2
s1 rc1 0
The Jacobian is then given as
det(J(θ1, θ2))=s1(−rs1)(c1)det(J(θ2))−rc1c1c1det(J(θ2))
= −r2c1(s12+c22)= −r2c1 (6.16) In computing this determinant we made use of the fact that if a column of a matrixA is multiplied by a scalar s, then the resulting determinant issdet(A). We also relied on the fact that the(3,1)-minorof J(θ1, θ2), obtained by deleting row 3 and column 1 is actually J(θ2) with the first column multiplied by −rs1 and the second column
multiplied byc1. Likewise, the(3,2)-minor of J(θ1, θ2))isJ(θ2)with both the columns multiplied byc1.
The volume of the hypersphere ford=3 is obtained via a triple integral withr >0,
−π/2≤θ1≤π/2, and 0≤θ2≤2π
Volume indDimensions
Before deriving a general expression for the hypersphere volume ind dimensions, let us consider the Jacobian in four dimensions. Generalizing the polar coordinates from three dimensions in Figure 6.10 to four dimensions, we obtain
x1=rcosθ1cosθ2cosθ3=rc2c2c3 x2=rcosθ1cosθ2sinθ3=rc1c2s3 x3=rcosθ1sinθ2=rc1s1 x4=rsinθ1=rs1 The Jacobian matrix is given as
J(θ1, θ2, θ3)=
Utilizing the Jacobian in three dimensions [Eq. (6.16)], the Jacobian in four dimensions is given as
det(J(θ1, θ2, θ3))=s1(−rs1)(c1)(c1)det(J(θ2, θ3))−rc1(c1)(c1)(c1)det(J(θ2, θ3))
=r3s12c21c2+r3c41c2=r3c12c2(s12+c21)=r3c21c2
Jacobian indDimensions By induction, we can obtain thed-dimensional Jacobian as follows:
det(J(θ1, θ2, . . . , θd−1))=(−1)drd−1cd1−2cd2−3. . . cd−2
6.7 Appendix: Derivation of Hypersphere Volume 179 The volume of the hypersphere is given by the d-dimensional integral with r >0,
−π/2≤θi≤π/2 for alli=1, . . . , d−2, and 0≤θd−1≤2π:
Consider one of the intermediate integrals:
Zπ/2 Substituting Eq. (6.20) in Eq. (6.19), we get the new integral:
2
whereB(α, β)is thebeta function, given as B(α, β)=
Z1 0
uα−1(1−u)β−1du
and it can be expressed in terms of the gamma function [Eq. (6.6)] via the identity B(α, β)=Ŵ(α)Ŵ(β) which matches the expression in Eq. (6.4).
6.8 FURTHER READING
For an introduction to the geometry ofd-dimensional spaces see Kendall (1961) and also Scott (1992, Section 1.5). The derivation of the mean distance for the multivariate normal is from MacKay (2003, p. 130).
Kendall, M. G. (1961).A Course in the Geometry ofnDimensions. New York: Hafner.
MacKay, D. J. (2003). Information Theory, Inference and Learning Algorithms.
New York: Cambridge University Press.
Scott, D. W. (1992).Multivariate Density Estimation: Theory, Practice, and Visualiza-tion. New York: John Wiley & Sons.
6.9 EXERCISES
Q1. Given the gamma function in Eq. (6.6), show the following:
(a) Ŵ(1)=1 (b) Ŵ
1 2
=√π
(c) Ŵ(α)=(α−1)Ŵ(α−1)
Q2. Show that the asymptotic volume of the hypersphereSd(r)for any value of radiusr eventually tends to zero asdincreases.
Q3. The ball with centerc∈Rdand radiusris defined as Bd(c, r)=
x∈Rd|δ(x,c)≤r
where δ(x,c) is the distance between x and c, which can be specified using the Lp-norm:
Lp(x,c)= Xd
i=1
|xi−ci|p
!1p
where p6= 0 is any real number. The distance can also be specified using the L∞-norm:
L∞(x,c)=max
i
|xi−ci|
Answer the following questions:
(a) Ford=2, sketch the shape of the hyperball inscribed inside the unit square, using theLp-distance withp=0.5 and with centerc=(0.5,0.5)T.
(b) Withd=2 andc=(0.5,0.5)T, using theL∞-norm, sketch the shape of the ball of radiusr=0.25 inside a unit square.
(c) Compute the formula for the maximum distance between any two points in the unit hypercube in d dimensions, when using the Lp-norm. What is the maximum distance forp=0.5 whend=2? What is the maximum distance for the L∞-norm?
6.9 Exercises 181
ǫ ǫ
Figure 6.11. For Q4.
Q4. Consider the corner hypercubes of length ǫ ≤ 1 inside a unit hypercube. The 2-dimensional case is shown in Figure 6.11. Answer the following questions:
(a) Letǫ=0.1. What is the fraction of the total volume occupied by the corner cubes in two dimensions?
(b) Derive an expression for the volume occupied by all of the corner hypercubes of lengthǫ <1 as a function of the dimensiond. What happens to the fraction of the volume in the corners asd→ ∞?
(c) What is the fraction of volume occupied by the thin hypercube shell of widthǫ <1 as a fraction of the total volume of the outer (unit) hypercube, asd→ ∞? For example, in two dimensions the thin shell is the space between the outer square (solid) and inner square (dashed).
Q5. Prove Eq. (6.14), that is, limd→∞P xTx≤ −2 ln(α)
→0, for anyα∈(0,1)andx∈Rd. Q6. Consider the conceptual view of high-dimensional space shown in Figure 6.4. Derive an expression for the radius of the inscribed circle, so that the area in the spokes accurately reflects the difference between the volume of the hypercube and the inscribed hypersphere ind dimensions. For instance, if the length of a half-diagonal is fixed at 1, then the radius of the inscribed circle is √1
2in Figure 6.4a.
Q7. Consider the unit hypersphere (with radiusr=1). Inside the hypersphere inscribe a hypercube (i.e., the largest hypercube you can fit inside the hypersphere). An example in two dimensions is shown in Figure 6.12. Answer the following questions:
Figure 6.12. For Q7.
(a) Derive an expression for the volume of the inscribed hypercube for any given dimensionalityd. Derive the expression for one, two, and three dimensions, and then generalize to higher dimensions.
(b) What happens to the ratio of the volume of the inscribed hypercube to the volume of the enclosing hypersphere as d→ ∞? Again, give the ratio in one, two and three dimensions, and then generalize.
Q8. Assume that a unit hypercube is given as [0,1]d, that is, the range is [0,1] in each dimension. The main diagonal in the hypercube is defined as the vector from(0,0)= (
d−1
z }| {
0, . . . ,0,0)to(1,1)=(
d−1
z }| {
1, . . . ,1,1). For example, whend=2, the main diagonal goes from (0,0) to(1,1). On the other hand, the main anti-diagonal is defined as the vector from (1,0)=(
d−1
z }| {
1, . . . ,1,0)to(0,1)=(
d−1
z }| {
0, . . . ,0,1) For example, for d =2, the anti-diagonal is from(1,0)to(0,1).
(a) Sketch the diagonal and anti-diagonal ind=3 dimensions, and compute the angle between them.
(b) What happens to the angle between the main diagonal and anti-diagonal asd→
∞. First compute a general expression for theddimensions, and then take the limit asd→ ∞.
Q9. Draw a sketch of a hypersphere in four dimensions.