Vector Calculus
5.8 Linearization and Multivariate Taylor Series
Figure 5.12 Linear approximation of a function. The original functionf is linearized at x0=−2using a first-order Taylor series expansion.
−4 −2 0 2 4
x
−2
−1 0 1
f(x)
f(x)
f(x0) f(x0) +f0(x0)(x−x0)
Iff(x, y)is a twice (continuously) differentiable function, then
∂2f
∂x∂y = ∂2f
∂y∂x, (5.146)
i.e., the order of differentiation does not matter, and the corresponding
Hessian matrix Hessian matrix
H =
∂2f
∂x2
∂2f
∂x∂y
∂2f
∂x∂y
∂2f
∂y2
(5.147)
is symmetric. The Hessian is denoted as∇2x,yf(x, y). Generally, forx∈Rn andf : Rn → R, the Hessian is ann×nmatrix. The Hessian measures the curvature of the function locally around(x, y).
Remark(Hessian of a Vector Field). Iff :Rn → Rmis a vector field, the
Hessian is an(m×n×n)-tensor. ♦
5.8 Linearization and Multivariate Taylor Series
The gradient∇f of a functionf is often used for a locally linear approxi-mation off aroundx0:
f(x)≈f(x0) + (∇xf)(x0)(x−x0). (5.148) Here (∇xf)(x0) is the gradient off with respect to x, evaluated atx0. Figure 5.12 illustrates the linear approximation of a functionfat an input x0. The original function is approximated by a straight line. This approx-imation is locally accurate, but the farther we move away from x0 the worse the approximation gets. Equation (5.148) is a special case of a mul-tivariate Taylor series expansion of f at x0, where we consider only the first two terms. We discuss the more general case in the following, which will allow for better approximations.
Figure 5.13 Visualizing outer products. Outer products of vectors increase the dimensionality of the array by 1 per term. (a) The outer product of two vectors results in a matrix; (b) the outer product of three vectors yields a third-order tensor.
(a) Given a vectorδ∈R4, we obtain the outer productδ2:=δ⊗δ=δδ>∈ R4×4as a matrix.
(b) An outer productδ3 := δ⊗δ⊗δ ∈ R4×4×4 results in a third-order tensor (“three-dimensional matrix”), i.e., an array with three indexes.
Definition 5.7(Multivariate Taylor Series). We consider a function
f :RD →R (5.149)
x7→f(x), x∈RD, (5.150) that is smooth atx0. When we define the difference vectorδ :=x−x0, themultivariate Taylor seriesoff at(x0)is defined as
multivariate Taylor series
f(x) =
∞
X
k=0
Dkxf(x0)
k! δk, (5.151)
whereDxkf(x0)is thek-th (total) derivative off with respect tox, eval-uated atx0.
Definition 5.8(Taylor Polynomial). TheTaylor polynomialof degreenof
Taylor polynomial
f atx0contains the firstn+ 1components of the series in (5.151) and is defined as
Tn(x) =
n
X
k=0
Dkxf(x0)
k! δk. (5.152)
In (5.151) and (5.152), we used the slightly sloppy notation of δk, which is not defined for vectorsx ∈ RD, D > 1,andk > 1. Note that bothDkxf andδk are k-th order tensors, i.e., k-dimensional arrays. The
A vector can be implemented as a one-dimensional array, a matrix as a two-dimensional array.
kth-order tensorδk ∈R
ktimes
z }| {
D×D×...×D is obtained as ak-fold outer product, denoted by⊗, of the vectorδ∈RD. For example,
δ2 :=δ⊗δ=δδ>, δ2[i, j] =δ[i]δ[j] (5.153)
5.8 Linearization and Multivariate Taylor Series 167 δ3:=δ⊗δ⊗δ, δ3[i, j, k] =δ[i]δ[j]δ[k]. (5.154) Figure 5.13 visualizes two such outer products. In general, we obtain the terms in the Taylor series, whereDkxf(x0)δkcontainsk-th order polynomials.
Now that we defined the Taylor series for vector fields, let us explicitly write down the first termsDkxf(x0)δk of the Taylor series expansion for k= 0, . . . ,3andδ:=x−x0:
Example 5.15 (Taylor Series Expansion of a Function with Two Vari-ables)
Consider the function
f(x, y) =x2+ 2xy+y3. (5.161) We want to compute the Taylor series expansion off at(x0, y0) = (1,2). Before we start, let us discuss what to expect: The function in (5.161) is a polynomial of degree 3. We are looking for a Taylor series expansion, which itself is a linear combination of polynomials. Therefore, we do not expect the Taylor series expansion to contain terms of fourth or higher order to express a third-order polynomial. This means that it should be sufficient to determine the first four terms of (5.151) for an exact alterna-tive representation of (5.161).
To determine the Taylor series expansion, we start with the constant term and the first-order derivatives, which are given by
f(1,2) = 13 (5.162)
∂f
∂x = 2x+ 2y =⇒ ∂f
∂x(1,2) = 6 (5.163)
∂f
∂y = 2x+ 3y2 =⇒ ∂f
∂y(1,2) = 14. (5.164) Therefore, we obtain
D1x,yf(1,2) =∇x,yf(1,2) =h∂f∂x(1,2) ∂f∂y(1,2)i=6 14∈R1×2 (5.165) such that
D1x,yf(1,2)
1! δ=6 14 x−1
y−2
= 6(x−1) + 14(y−2). (5.166) Note thatDx,y1 f(1,2)δcontains only linear terms, i.e., first-order polyno-mials.
The second-order partial derivatives are given by
∂2f
∂x2 = 2 =⇒ ∂2f
∂x2(1,2) = 2 (5.167)
∂2f
∂y2 = 6y =⇒ ∂2f
∂y2(1,2) = 12 (5.168)
∂2f
∂y∂x = 2 =⇒ ∂2f
∂y∂x(1,2) = 2 (5.169)
∂2f
∂x∂y = 2 =⇒ ∂2f
∂x∂y(1,2) = 2. (5.170) When we collect the second-order partial derivatives, we obtain the Hes-sian
H =
"∂2f
∂x2
∂2f
∂x∂y
∂2f
∂y∂x
∂2f
∂y2
#
= 2 2
2 6y
, (5.171)
such that
H(1,2) = 2 2
2 12
∈R2×2. (5.172)
Therefore, the next term of the Taylor-series expansion is given by D2x,yf(1,2)
2! δ2 = 1
2δ>H(1,2)δ (5.173a)
= 1 2
x−1 y−2 2 2
2 12
x−1 y−2
(5.173b)
= (x−1)2+ 2(x−1)(y−2) + 6(y−2)2. (5.173c) Here,Dx,y2 f(1,2)δ2contains only quadratic terms, i.e., second-order poly-nomials.
5.8 Linearization and Multivariate Taylor Series 169 The third-order derivatives are obtained as
Dx,y3 f =h∂H∂x ∂H∂yi∈R2×2×2, (5.174)
Since most second-order partial derivatives in the Hessian in (5.171) are constant, the only nonzero third-order partial derivative is
∂3f
∂y3 = 6 =⇒ ∂3f
∂y3(1,2) = 6. (5.177) Higher-order derivatives and the mixed derivatives of degree 3 (e.g.,
∂f3
which collects all cubic terms of the Taylor series. Overall, the (exact) Taylor series expansion off at(x0, y0) = (1,2)is In this case, we obtained an exact Taylor series expansion of the polyno-mial in (5.161), i.e., the polynopolyno-mial in (5.180c) is identical to the original polynomial in (5.161). In this particular example, this result is not sur-prising since the original function was a third-order polynomial, which we expressed through a linear combination of constant terms, first-order, second-order, and third-order polynomials in (5.180c).
5.9 Further Reading
Further details of matrix differentials, along with a short review of the required linear algebra, can be found in Magnus and Neudecker (2007).
Automatic differentiation has had a long history, and we refer to Griewank and Walther (2003), Griewank and Walther (2008), and Elliott (2009) and the references therein.
In machine learning (and other disciplines), we often need to compute expectations, i.e., we need to solve integrals of the form
Ex[f(x)] = Z
f(x)p(x)dx. (5.181) Even if p(x) is in a convenient form (e.g., Gaussian), this integral gen-erally cannot be solved analytically. The Taylor series expansion of f is one way of finding an approximate solution: Assumingp(x) =N µ,Σ is Gaussian, then the first-order Taylor series expansion aroundµlocally linearizes the nonlinear functionf. For linear functions, we can compute the mean (and the covariance) exactly ifp(x)is Gaussian distributed (see Section 6.5). This property is heavily exploited by the extended Kalman
extended Kalman
filter filter (Maybeck, 1979) for online state estimation in nonlinear dynami-cal systems (also dynami-called “state-space models”). Other deterministic ways to approximate the integral in (5.181) are theunscented transform(Julier
unscented transform
and Uhlmann, 1997), which does not require any gradients, or theLaplace
Laplace
approximation approximation(MacKay, 2003; Bishop, 2006; Murphy, 2012), which uses a second-order Taylor series expansion (requiring the Hessian) for a local Gaussian approximation ofp(x)around its mode.
Exercises 5.1 Compute the derivativef0(x)for
f(x) = log(x4) sin(x3). 5.2 Compute the derivativef0(x)of the logistic sigmoid
f(x) = 1
1 + exp(−x). 5.3 Compute the derivativef0(x)of the function
f(x) = exp(−2σ12(x−µ)2), whereµ, σ∈Rare constants.
5.4 Compute the Taylor polynomials Tn,n = 0, . . . ,5off(x) = sin(x) + cos(x) atx0= 0.
5.5 Consider the following functions:
f1(x) = sin(x1) cos(x2), x∈R2 f2(x,y) =x>y, x,y∈Rn f3(x) =xx>, x∈Rn
Exercises 171 a. What are the dimensions of∂f∂xi ?
b. Compute the Jacobians.
5.6 Differentiatefwith respect totandgwith respect toX, where f(t) = sin(log(t>t)), t∈RD
g(X) =tr(AXB), A∈RD×E,X∈RE×F,B∈RF×D, where tr(·)denotes the trace.
5.7 Compute the derivativesdf /dxof the following functions by using the chain rule. Provide the dimensions of every single partial derivative. Describe your steps in detail.
a.
f(z) = log(1 +z), z=x>x, x∈RD b.
f(z) = sin(z), z=Ax+b, A∈RE×D,x∈RD,b∈RE wheresin(·)is applied to every element ofz.
5.8 Compute the derivatives df /dx of the following functions. Describe your steps in detail.
a. Use the chain rule. Provide the dimensions of every single partial deriva-tive.
f(z) = exp(−12z)
z=g(y) =y>S−1y y=h(x) =x−µ
wherex,µ∈RD,S∈RD×D. b.
f(x) =tr(xx>+σ2I), x∈RD
Here tr(A)is the trace ofA, i.e., the sum of the diagonal elementsAii. Hint: Explicitly write out the outer product.
c. Use the chain rule. Provide the dimensions of every single partial deriva-tive. You do not need to compute the product of the partial derivatives explicitly.
f= tanh(z)∈RM
z=Ax+b, x∈RN,A∈RM×N,b∈RM. Here,tanhis applied to every component ofz.
5.9 We define
g(z,ν) := logp(x,z)−logq(z,ν) z:=t(,ν)
for differentiable functionsp, q, t, andx∈RD,z∈RE,ν∈RF,∈RG. By using the chain rule, compute the gradient
d
dνg(z,ν).