part 1

Stochastic Variance-Reduced Optimization for Machine Learning

Part I

Francis Bach

INRIA - Ecole Normale Sup´erieure, Paris, France

É C O L E N O R M A L E S U P É R I E U R E

Joint tutorial with Mark Schmidt, UBC - SIOPT - 2017

Context

Machine learning for large-scale data

• Large-scale supervised machine learning: large d, large n

– d : dimension of each observation (input) or number of parameters – n : number of observations

• Examples: computer vision, advertising, bioinformatics, etc.

– Ideal running-time complexity: O(dn)

• Going back to simple methods

− Stochastic gradient methods (Robbins and Monro, 1951)

• Goal: Present recent progress

Search engines - Advertising - Marketing

Visual object recognition

Context

Machine learning for large-scale data

• Large-scale supervised machine learning: large d, large n

– d : dimension of each observation (input), or number of parameters – n : number of observations

• Examples: computer vision, advertising, bioinformatics, etc.

• Ideal running-time complexity: O(dn)

• Going back to simple methods

− Stochastic gradient methods (Robbins and Monro, 1951)

• Goal: Present recent progress

Context

Machine learning for large-scale data

• Large-scale supervised machine learning: large d, large n

– d : dimension of each observation (input), or number of parameters – n : number of observations

• Examples: computer vision, advertising, bioinformatics, etc.

• Ideal running-time complexity: O(dn)

• Going back to simple methods

– Stochastic gradient methods (Robbins and Monro, 1951)

• Goal: Present recent progress

Outline

1. Introduction/motivation: Supervised machine learning

− Optimization of finite sums

− Existing optimization methods for finite sums 2. Convex finite-sum problems

– Linearly-convergent stochastic gradient method – SAG, SAGA, SVRG, SDCA, MISO, etc.

– From lazy gradient evaluations to variance reduction 3. Non-convex problems

4. Non-independent and identically distributed 5. Non-stochastic: other types of structures 6. Non-serial: parallel and distributed settings

References

• Textbooks and tutorials

– Nesterov (2004): Introductory lectures on convex optimization

– Bubeck (2015): Convex Optimization: Algorithms and Complexity – Bertsekas (2016): Nonlinear programming

– Bottou et al. (2016): Optimization methods for large-scale machine learning

• Research papers – See end of slides

– Slides available at www.ens.fr/~fbach/

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• Motivating examples

– Linear predictions: h(x, θ) = θ^⊤Φ(x) with features Φ(x) ∈ R^d

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• Motivating examples

– Linear predictions: h(x, θ) = θ^⊤Φ(x) with features Φ(x) ∈ R^d – Neural networks: h(x, θ) = θ_m^⊤σ(θ_m^⊤₋₁σ(· · · θ₂^⊤σ(θ₁^⊤x))

x y

θ

θ

θ

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• (regularized) empirical risk minimization: find θˆ solution of min

θ∈R^d

1 n

n

X

i=1

nℓ y_i, h(x_i, θ)

+ λΩ(θ)o

= 1 n

n

X

i=1

f_i(θ)

data fitting term + regularizer

Usual losses

• Regression: y ∈ R

– Quadratic loss ℓ y, h(x, θ)

= ¹₂(y − h(x, θ))²

Usual losses

• Regression: y ∈ R

– Quadratic loss ℓ y, h(x, θ)

= ¹₂(y − h(x, θ))²

• Classification : y ∈ {−1, 1} – Logistic loss ℓ y, h(x, θ)

= log(1 + exp(−yh(x, θ)))

Usual losses

• Regression: y ∈ R

– Quadratic loss ℓ y, h(x, θ)

= ¹₂(y − h(x, θ))²

• Classification : y ∈ {−1, 1} – Logistic loss ℓ y, h(x, θ)

= log(1 + exp(−yh(x, θ)))

• Structured prediction

– Complex outputs y (k classes/labels, graphs, trees, or {0, 1}^k, etc.) – Prediction function h(x, θ) ∈ R^k

– Conditional random fields (Lafferty et al., 2001)

– Max-margin (Taskar et al., 2003; Tsochantaridis et al., 2005)

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• (regularized) empirical risk minimization: find θˆ solution of min

θ∈R^d

1 n

n

X

i=1

nℓ y_i, h(x_i, θ)

+ λΩ(θ)o

= 1 n

n

X

i=1

f_i(θ)

data fitting term + regularizer

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• (regularized) empirical risk minimization: find θˆ solution of min

θ∈R^d

1 n

n

X

i=1

nℓ y_i, h(x_i, θ)

+ λΩ(θ)o

= 1 n

n

X

i=1

f_i(θ)

data fitting term + regularizer

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• (regularized) empirical risk minimization: find θˆ solution of min

θ∈R^d

1 n

n

X

i=1

nℓ y_i, h(x_i, θ)

+ λΩ(θ)o

= 1 n

n

X

i=1

f_i(θ)

data fitting term + regularizer

• Optimization: optimization of regularized risk training cost

Parametric supervised machine learning

• Data: n observations (x_i, y_i) ∈ X × Y, i = 1, . . . , n

• Prediction function h(x, θ) ∈ R parameterized by θ ∈ R^d

• (regularized) empirical risk minimization: find θˆ solution of min

θ∈R^d

1 n

n

X

i=1

nℓ y_i, h(x_i, θ)

+ λΩ(θ)o

= 1 n

n

X

i=1

f_i(θ)

data fitting term + regularizer

• Optimization: optimization of regularized risk training cost

• Statistics: guarantees on Ep(x,y)ℓ(y, h(x, θ)) testing cost

Smoothness and (strong) convexity

• A function g : R^d → R is L-smooth if and only if it is twice differentiable and

∀θ ∈ R^d,

eigenvalues

g^′′(θ)

6 L

smooth non-smooth

Smoothness and (strong) convexity

• A function g : R^d → R is L-smooth if and only if it is twice differentiable and

∀θ ∈ R^d,

eigenvalues

g^′′(θ)

6 L

• Machine learning – with g(θ) = _n¹ Pn

i=1 ℓ(y_i, h(x_i, θ))

– Smooth prediction function θ 7→ h(x_i, θ) + smooth loss

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> 0

convex

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ

convex

strongly

convex

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ – Condition number κ = L/µ > 1

(small κ = L/µ) (large κ = L/µ)

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ

• Convexity in machine learning – With g(θ) = _n¹ Pn

i=1 ℓ(y_i, h(x_i, θ))

– Convex loss and linear predictions h(x, θ) = θ^⊤Φ(x)

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ

• Convexity in machine learning – With g(θ) = _n¹ Pn

i=1 ℓ(y_i, h(x_i, θ))

– Convex loss and linear predictions h(x, θ) = θ^⊤Φ(x)

• Relevance of convex optimization

– Easier design and analysis of algorithms

– Global minimum vs. local minimum vs. stationary points

– Gradient-based algorithms only need convexity for their analysis

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ

• Strong convexity in machine learning – With g(θ) = _n¹ Pn

i=1 ℓ(y_i, h(x_i, θ))

– Strongly convex loss and linear predictions h(x, θ) = θ^⊤Φ(x)

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ

• Strong convexity in machine learning – With g(θ) = _n¹ Pn

i=1 ℓ(y_i, h(x_i, θ))

– Strongly convex loss and linear predictions h(x, θ) = θ^⊤Φ(x) – Invertible covariance matrix _n¹ Pn

i=1 Φ(x_i)Φ(x_i)^⊤ ⇒ n > d – Even when µ > 0, µ may be arbitrarily small!

Smoothness and (strong) convexity

• A twice differentiable function g : R^d → R is µ-strongly convex if and only if

∀θ ∈ R^d, eigenvalues

g^′′(θ)

> µ

• Strong convexity in machine learning – With g(θ) = _n¹ Pn

i=1 ℓ(y_i, h(x_i, θ))

– Strongly convex loss and linear predictions h(x, θ) = θ^⊤Φ(x) – Invertible covariance matrix _n¹ Pn

i=1 Φ(x_i)Φ(x_i)^⊤ ⇒ n > d – Even when µ > 0, µ may be arbitrarily small!

• Adding regularization by ^µ₂kθk²

– creates additional bias unless µ is small, but reduces variance – Typically L/√

n > µ > L/n

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1) g(θ_t) − g(θ_∗) 6 O(1/t)

g(θ_t) − g(θ_∗) 6 O(e^−t(µ/L)) = O(e^−t/κ)

(small κ = L/µ) (large κ = L/µ)

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1) g(θ_t) − g(θ_∗) 6 O(1/t)

g(θ_t) − g(θ_∗) 6 O((1−µ/L)^t) = O(e^−t(µ/L)) if µ-strongly convex

(small κ = L/µ) (large κ = L/µ)

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions – O(e^−t/κ) linear if strongly-convex

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions – O(e^−t/κ) linear if strongly-convex

• Newton method: θ_t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t

quadratic rate

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−t/κ) linear if strongly-convex ⇔ O(κ log ¹_ε) iterations

• Newton method: θ_t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t

quadratic rate ⇔ O(log log ¹_ε) iterations

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−t/κ) linear if strongly-convex ⇔ complexity = O(nd · κ log ¹_ε)

• Newton method: θ_t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t

quadratic rate ⇔ complexity = O((nd² + d³) · log log ¹_ε)

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−t/κ) linear if strongly-convex ⇔ complexity = O(nd · κ log ¹_ε)

• Newton method: θ_t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t

quadratic rate ⇔ complexity = O((nd² + d³) · log log ¹_ε)

• Key insights for machine learning (Bottou and Bousquet, 2008) 1. No need to optimize below statistical error

2. Cost functions are averages

3. Testing error is more important than training error

Iterative methods for minimizing smooth functions

• Assumption: g convex and L-smooth on R^d

• Gradient descent: θ_t = θ_t−1 − γ_t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−t/κ) linear if strongly-convex ⇔ complexity = O(nd · κ log ¹_ε)

• Newton method: θ_t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t

quadratic rate ⇔ complexity = O((nd² + d³) · log log ¹_ε)

• Key insights for machine learning (Bottou and Bousquet, 2008) 1. No need to optimize below statistical error

2. Cost functions are averages

3. Testing error is more important than training error

Stochastic gradient descent (SGD) for finite sums

min

θ∈R^d

g(θ) = 1 n

n

X

i=1

f_i(θ)

• Iteration: θ_t = θ_t−1 − γ_tf_i(t)^′ (θ_t−1)

– Sampling with replacement: i(t) random element of {1, . . . , n} – Polyak-Ruppert averaging: θ¯_t = _t+1¹ Pt

u=0 θ_u

Stochastic gradient descent (SGD) for finite sums

min

θ∈R^d

g(θ) = 1 n

n

X

i=1

f_i(θ)

• Iteration: θ_t = θ_t−1 − γ_tf_i(t)^′ (θ_t−1)

– Sampling with replacement: i(t) random element of {1, . . . , n} – Polyak-Ruppert averaging: θ¯_t = _t+1¹ Pt

u=0 θ_u

• Convergence rate if each f_i is convex L-smooth and g µ-strongly- convex:

Eg(¯θ_t) − g(θ_∗) 6

O(1/√

t) if γ_t = 1/(L√ t) O(L/(µt)) = O(κ/t) if γ_t = 1/(µt) – No adaptivity to strong-convexity in general

– Adaptivity with self-concordance assumption (Bach, 2014) – Running-time complexity: O(d · κ/ε)

Outline

1. Introduction/motivation: Supervised machine learning – Optimization of finite sums

– Existing optimization methods for finite sums 2. Convex finite-sum problems

− Linearly-convergent stochastic gradient method

− SAG, SAGA, SVRG, SDCA, etc.

− From lazy gradient evaluations to variance reduction

3. Non-convex problems

4. Parallel and distributed settings 5. Perspectives

Stochastic vs. deterministic methods

• Minimizing g(θ) = 1 n

n

X

i=1

f_i(θ) with f_i(θ) = ℓ y_i, h(x_i, θ)

+ λΩ(θ)

Stochastic vs. deterministic methods

• Minimizing g(θ) = 1 n

n

X

i=1

f_i(θ) with f_i(θ) = ℓ y_i, h(x_i, θ)

+ λΩ(θ)

• Batch gradient descent: θ_t = θ_t−1−γ_tg^′(θ_t−1) = θ_t−1−γ_t n

n

X

i=1

f_i^′(θ_t−1) – Linear (e.g., exponential) convergence rate in O(e^−t/κ)

– Iteration complexity is linear in n

Stochastic vs. deterministic methods

• Minimizing g(θ) = 1 n

n

X

i=1

f_i(θ) with f_i(θ) = ℓ y_i, h(x_i, θ)

+ λΩ(θ)

• Batch gradient descent: θ_t = θ_t−1−γ_tg^′(θ_t−1) = θ_t−1−γ_t n

n

X

i=1

f_i^′(θ_t−1)

Stochastic vs. deterministic methods

• Minimizing g(θ) = 1 n

n

X

i=1

f_i(θ) with f_i(θ) = ℓ y_i, h(x_i, θ)

+ λΩ(θ)

• Batch gradient descent: θ_t = θ_t−1−γ_tg^′(θ_t−1) = θ_t−1−γ_t n

n

X

i=1

f_i^′(θ_t−1) – Linear (e.g., exponential) convergence rate in O(e^−t/κ)

– Iteration complexity is linear in n

• Stochastic gradient descent: θ_t = θ_t−1 − γ_tf_i(t)^′ (θ_t−1)

– Sampling with replacement: i(t) random element of {1, . . . , n} – Convergence rate in O(κ/t)

– Iteration complexity is independent of n

Stochastic vs. deterministic methods

• Minimizing g(θ) = 1 n

n

X

i=1

f_i(θ) with f_i(θ) = ℓ y_i, h(x_i, θ)

+ λΩ(θ)

• Batch gradient descent: θ_t = θ_t−1−γ_tg^′(θ_t−1) = θ_t−1−γ_t n

n

X

i=1

f_i^′(θ_t−1)

• Stochastic gradient descent: θ_t = θ_t−1 − γ_tf_i(t)^′ (θ_t−1)

Stochastic vs. deterministic methods

• Goal = best of both worlds: Linear rate with O(d) iteration cost Simple choice of step size

time

log(excess cost)

deterministic

stochastic

Stochastic vs. deterministic methods

• Goal = best of both worlds: Linear rate with O(d) iteration cost Simple choice of step size

time

log(excess cost)

deterministic stochastic

new

Accelerating gradient methods - Related work

• Generic acceleration (Nesterov, 1983, 2004)

θ_t = η_t−1 − γ_tg^′(η_t−1) and η_t = θ_t + δ_t(θ_t − θ_t−1)

Accelerating gradient methods - Related work

• Generic acceleration (Nesterov, 1983, 2004)

θ_t = η_t−1 − γ_tg^′(η_t−1) and η_t = θ_t + δ_t(θ_t − θ_t−1) – Good choice of momentum term δ_t ∈ [0, 1)

g(θ_t) − g(θ_∗) 6 O(1/t²) g(θ_t) −g(θ_∗) 6 O(e^−t√

µ/L) = O(e^−t/√κ) if µ-strongly convex – Optimal rates after t = O(d) iterations (Nesterov, 2004)

Accelerating gradient methods - Related work

• Generic acceleration (Nesterov, 1983, 2004)

θ_t = η_t−1 − γ_tg^′(η_t−1) and η_t = θ_t + δ_t(θ_t − θ_t−1) – Good choice of momentum term δ_t ∈ [0, 1)

g(θ_t) − g(θ_∗) 6 O(1/t²) g(θ_t) −g(θ_∗) 6 O(e^−t√

µ/L) = O(e^−t/√κ) if µ-strongly convex – Optimal rates after t = O(d) iterations (Nesterov, 2004)

– Still O(nd) iteration cost: complexity = O(nd · √

κ log ¹_ε)

Accelerating gradient methods - Related work

• Constant step-size stochastic gradient

– Solodov (1998); Nedic and Bertsekas (2000)

– Linear convergence, but only up to a fixed tolerance

Accelerating gradient methods - Related work

• Constant step-size stochastic gradient

– Solodov (1998); Nedic and Bertsekas (2000)

– Linear convergence, but only up to a fixed tolerance

• Stochastic methods in the dual (SDCA) – Shalev-Shwartz and Zhang (2013)

– Similar linear rate but limited choice for the f_i’s

– Extensions without duality: see Shalev-Shwartz (2016)

Accelerating gradient methods - Related work

• Constant step-size stochastic gradient

– Solodov (1998); Nedic and Bertsekas (2000)

– Linear convergence, but only up to a fixed tolerance

• Stochastic methods in the dual (SDCA) – Shalev-Shwartz and Zhang (2013)

– Similar linear rate but limited choice for the f_i’s

– Extensions without duality: see Shalev-Shwartz (2016)

• Stochastic version of accelerated batch gradient methods – Tseng (1998); Ghadimi and Lan (2010); Xiao (2010)

– Can improve constants, but still have sublinear O(1/t) rate

Stochastic average gradient

(Le Roux, Schmidt, and Bach, 2012)

• Stochastic average gradient (SAG) iteration

– Keep in memory the gradients of all functions f_i, i = 1, . . . , n – Random selection i(t) ∈ {1, . . . , n} with replacement

– Iteration: θ_t = θ_t−1 − γ_t n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise

Stochastic average gradient

(Le Roux, Schmidt, and Bach, 2012)

• Stochastic average gradient (SAG) iteration

– Keep in memory the gradients of all functions f_i, i = 1, . . . , n – Random selection i(t) ∈ {1, . . . , n} with replacement

– Iteration: θ_t = θ_t−1 − γ_t n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise

y₁^t y₂^t y₃^t y₄^t y_n−1^t y_n^t f¹ f² f³ f⁴ fⁿ⁻¹ fⁿ

1 n

Pⁿ

i=1 y_i^t g =

1 n

Pⁿ

i=1 fⁱ

functions

gradients ∈ R

Stochastic average gradient

(Le Roux, Schmidt, and Bach, 2012)

• Stochastic average gradient (SAG) iteration

– Keep in memory the gradients of all functions f_i, i = 1, . . . , n – Random selection i(t) ∈ {1, . . . , n} with replacement

– Iteration: θ_t = θ_t−1 − γ_t n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise

y₁^t y₂^t y₃^t y₄^t y_n−1^t y_n^t f¹ f² f³ f⁴ fⁿ⁻¹ fⁿ

1 n

Pⁿ

i=1 y_i^t g =

1 n

Pⁿ

i=1 fⁱ

functions

gradients ∈ R

Stochastic average gradient

(Le Roux, Schmidt, and Bach, 2012)

• Stochastic average gradient (SAG) iteration

– Keep in memory the gradients of all functions f_i, i = 1, . . . , n – Random selection i(t) ∈ {1, . . . , n} with replacement

– Iteration: θ_t = θ_t−1 − γ_t n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise

y₁^t y₂^t y₃^t y₄^t y_n−1^t y_n^t f¹ f² f³ f⁴ fⁿ⁻¹ fⁿ

1 n

Pⁿ

i=1 y_i^t g =

1 n

Pⁿ

i=1 fⁱ

functions

gradients ∈ R

Stochastic average gradient

(Le Roux, Schmidt, and Bach, 2012)

• Stochastic average gradient (SAG) iteration

– Keep in memory the gradients of all functions f_i, i = 1, . . . , n – Random selection i(t) ∈ {1, . . . , n} with replacement

– Iteration: θ_t = θ_t−1 − γ_t n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise

• Stochastic version of incremental average gradient (Blatt et al., 2008)

Stochastic average gradient

(Le Roux, Schmidt, and Bach, 2012)

• Stochastic average gradient (SAG) iteration

– Keep in memory the gradients of all functions f_i, i = 1, . . . , n – Random selection i(t) ∈ {1, . . . , n} with replacement

– Iteration: θ_t = θ_t−1 − γ_t n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise

• Stochastic version of incremental average gradient (Blatt et al., 2008)

• Extra memory requirement: n gradients in R^d in general

• Linear supervised machine learning: only n real numbers – If f_i(θ) = ℓ(y_i, Φ(x_i)^⊤θ), then f_i^′(θ) = ℓ^′(y_i, Φ(x_i)^⊤θ) Φ(x_i)

Stochastic average gradient - Convergence analysis

• Assumptions

– Each f_i is L-smooth, i = 1, . . . , n – g= _n¹ Pn

i=1 f_i is µ-strongly convex

– constant step size γ_t = 1/(16L) - no need to know µ

Stochastic average gradient - Convergence analysis

• Assumptions

– Each f_i is L-smooth, i = 1, . . . , n – g= _n¹ Pn

i=1 f_i is µ-strongly convex

– constant step size γ_t = 1/(16L) - no need to know µ

• Strongly convex case (Le Roux et al., 2012; Schmidt et al., 2016) E

g(θ_t) − g(θ_∗)

6 cst ×

1 − min n 1

8n, µ 16L

ot

– Linear (exponential) convergence rate with O(d) iteration cost – After one pass, reduction of cost by exp

− min n

1

8, _16L^nµ o – NB: in machine learning, may often restrict to µ > L/n

⇒ constant error reduction after each effective pass

Running-time comparisons (strongly-convex)

• Assumptions: g(θ) = _n¹ Pn

i=1 f_i(θ)

– Each f_i convex L-smooth and g µ-strongly convex Stochastic gradient descent d×

L

µ × ¹_ε

Gradient descent d×

n^L_µ × log ¹_ε Accelerated gradient descent d×

nq

L

µ × log ¹_ε

SAG d×

(n + ^L_µ) × log ¹_ε

– NB-1: for (accelerated) gradient descent, L = smoothness constant of g

– NB-2: with non-uniform sampling, L = average smoothness constants of all f_i’s

Running-time comparisons (strongly-convex)

• Assumptions: g(θ) = _n¹ Pn

i=1 f_i(θ)

– Each f_i convex L-smooth and g µ-strongly convex Stochastic gradient descent d×

L

µ × ¹_ε

Gradient descent d×

n^L_µ × log ¹_ε Accelerated gradient descent d×

nq

L

µ × log ¹_ε

SAG d×

(n + ^L_µ) × log ¹_ε

• Beating two lower bounds (Nemirovski and Yudin, 1983; Nesterov, 2004): with additional assumptions

(1) stochastic gradient: exponential rate for finite sums

(2) full gradient: better exponential rate using the sum structure

Running-time comparisons (non-strongly-convex)

• Assumptions: g(θ) = _n¹ Pn

i=1 f_i(θ) – Each f_i convex L-smooth

– Ill conditioned problems: g may not be strongly-convex (µ = 0) Stochastic gradient descent d×

1/ε²

Gradient descent d×

n/ε Accelerated gradient descent d×

n/√ ε

SAG d×

√n/ε

• Adaptivity to potentially hidden strong convexity

• No need to know the local/global strong-convexity constant

Stochastic average gradient

Implementation details and extensions

• Sparsity in the features

– Just-in-time updates ⇒ replace O(d) by number of non zeros – See also Leblond, Pedregosa, and Lacoste-Julien (2016)

• Mini-batches

– Reduces the memory requirement + block access to data

• Line-search

– Avoids knowing L in advance

• Non-uniform sampling

– Favors functions with large variations

• See www.cs.ubc.ca/~schmidtm/Software/SAG.html

Experimental results (logistic regression)

quantum dataset rcv1 dataset

(n = 50 000, d = 78) (n = 697 641, d = 47 236)

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶✵

✲

✶✵

✲ ✁

✶✵

✲ ✂

✶✵

✄

❊ ☎ ☎✆✝ ✞✟✠✆✡☛☞☞ ✆☞

❖✌✍✎✏✑✒✓✎✔✒✕✖✗❖✘✑✒✔✖✔

AFG

L−BFGS ASG SG

IAG

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶✵

✲

✶✵

✲ ✁

✶✵

✲✂

✶✵

✲ ✄

✶✵

☎

❊ ✆ ✆✝✞ ✟✠✡✝☛☞✌✌ ✝✌

❖✍✎✏✑✒✓✔✏✕✓✖✗✘❖✙✒✓✕✗✕

AFG L−BFGS

SG

ASG

IAG

Experimental results (logistic regression)

quantum dataset rcv1 dataset

(n = 50 000, d = 78) (n = 697 641, d = 47 236)

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶ ✵

✲ ✁

✶ ✵

✲ ✂

✶ ✵

✲✄

✶ ✵

✲☎

✶ ✵

✲✆

✶ ✵

✲ ✁

✶ ✵

✂

❊ ✝ ✝✞✟ ✠✡☛✞☞✌✍✍ ✞✍

❖✎✏✑✒✓✔✕✑✖✔✗✘✙❖✚✓✔✖✘✖

AFG

L−BFGS SG ASG

IAG

SAG−LS

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶ ✵

✲ ✁

✶ ✵

✲ ✂

✶ ✵

✲ ✁

✶ ✵

✂

❊ ✄ ✄☎✆ ✝✞✟☎✠✡☛☛ ☎☛

❖☞✌✍✎✏✑✒✍✓✑✔✕✖❖✗✏✑✓✕✓

L−BFGS AFG SG

ASG IAG

SAG−LS

Before non-uniform sampling

protein dataset sido dataset

(n = 145 751, d = 74) (n = 12 678, d = 4 932)

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶✵

✲

✶✵

✲✁

✶✵

✲ ✂

✶✵

✲ ✄

✶✵

✲☎

✶✵

✆

❊ ✝ ✝✞✟ ✠✡☛✞☞✌✍✍ ✞✍

❖✎✏✑✒✓✔✕✑✖✔✗✘✙❖✚✓✔✖✘✖

IAG AGD

L−BFGS

SG−C

ASG−C PCD−L

DCA SAG

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶✵

✲

✶✵

✲ ✁

✶✵

✲ ✂

✶✵

✲ ✄

✶✵

✲ ☎

✶✵

✲ ✆

✶✵

✝

❊ ✞ ✞✟✠ ✡☛☞✟✌✍✎✎ ✟✎

❖✏✑✒✓✔✕✖✒✗✕✘✙✚❖✛✔✕✗✙✗

IAG AGD

L−BFGS

SG−C ASG−C

PCD−L

DCA SAG

After non-uniform sampling

protein dataset sido dataset

(n = 145 751, d = 74) (n = 12 678, d = 4 932)

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶ ✵

✲ ✁

✶ ✵

✲✂ ✄

✶ ✵

✲✂ ✁

✶ ✵

✲ ✄

✶ ✵

✁

❊ ☎ ☎✆✝ ✞✟✠✆✡☛☞☞ ✆☞

❖✌✍✎✏✑✒✓✎✔✒✕✖✗❖✘✑✒✔✖✔

IAGAGD L−BFGS

SG−C

ASG−C PCD−L

DCA

SAG

SAG−LS (Lipschitz)

✵ ✶ ✵ ✷✵ ✸ ✵ ✹ ✵ ✺ ✵

✶ ✵

✲ ✁

✶ ✵

✲ ✂

✶ ✵

✲ ✁

✶ ✵

✂

❊ ✄ ✄☎✆ ✝✞✟☎✠✡☛☛ ☎☛

❖☞✌✍✎✏✑✒✍✓✑✔✕✖❖✗✏✑✓✕✓

IAG AGD

L−BFGS

SG−C ASG−CPCD−L

DCA SAG

SAG−LS (Lipschitz)

Linearly convergent stochastic gradient algorithms

• Many related algorithms

– SAG (Le Roux, Schmidt, and Bach, 2012) – SDCA (Shalev-Shwartz and Zhang, 2013)

– SVRG (Johnson and Zhang, 2013; Zhang et al., 2013) – MISO (Mairal, 2015)

– Finito (Defazio et al., 2014b)

– SAGA (Defazio, Bach, and Lacoste-Julien, 2014a) – · · ·

• Similar rates of convergence and iterations

– Different interpretations and proofs / proof lengths – Lazy gradient evaluations

– Variance reduction

Linearly convergent stochastic gradient algorithms

• Many related algorithms

– SAG (Le Roux, Schmidt, and Bach, 2012) – SDCA (Shalev-Shwartz and Zhang, 2013)

– SVRG (Johnson and Zhang, 2013; Zhang et al., 2013) – MISO (Mairal, 2015)

– Finito (Defazio et al., 2014b)

– SAGA (Defazio, Bach, and Lacoste-Julien, 2014a) – · · ·

• Similar rates of convergence and iterations

• Different interpretations and proofs / proof lengths – Lazy gradient evaluations

– Variance reduction

Variance reduction

• Principle: reducing variance of sample of X by using a sample from another random variable Y with known expectation

Z_α = α(X − Y ) + EY – EZ_α = αEX + (1 − α)EY

– var(Z_α) = α²

var(X) + var(Y ) − 2cov(X, Y )

– α = 1: no bias, α < 1: potential bias (but reduced variance) – Useful if Y positively correlated with X

Variance reduction

• Principle: reducing variance of sample of X by using a sample from another random variable Y with known expectation

Z_α = α(X − Y ) + EY – EZ_α = αEX + (1 − α)EY

– var(Z_α) = α²

var(X) + var(Y ) − 2cov(X, Y )

– α = 1: no bias, α < 1: potential bias (but reduced variance) – Useful if Y positively correlated with X

• Application to gradient estimation (Johnson and Zhang, 2013;

Zhang, Mahdavi, and Jin, 2013)

– SVRG: X = f_i(t)^′ (θ_t−1), Y = f_i(t)^′ (˜θ), α = 1, with θ˜ stored – EY = _n¹ Pn

i=1 f_i^′(˜θ) full gradient at θ,˜ X − Y = f_i(t)^′ (θ_t−1) − f_i(t)^′ (˜θ)

Stochastic variance reduced gradient (SVRG) (Johnson and Zhang, 2013; Zhang et al., 2013)

• Initialize θ˜ ∈ R^d

• For i_epoch = 1 to # of epochs

– Compute all gradients f_i^′(˜θ) ; store g^′(˜θ) = _n¹ Pn

i=1 f_i^′(˜θ) – Initialize θ₀ = ˜θ

– For t = 1 to length of epochs θ_t = θ_t−1 − γh

g^′(˜θ) + f_i(t)^′ (θ_t−1) − f_i(t)^′ (˜θ)i – Update θ˜ = θ_t

• Output: θ˜

– No need to store gradients - two gradient evaluations per inner step – Two parameters: length of epochs + step-size γ

– Same linear convergence rate as SAG, simpler proof

Stochastic variance reduced gradient (SVRG) (Johnson and Zhang, 2013; Zhang et al., 2013)

• Initialize θ˜ ∈ R^d

• For i_epoch = 1 to # of epochs

– Compute all gradients f_i^′(˜θ) ; store g^′(˜θ) = _n¹ Pn

i=1 f_i^′(˜θ) – Initialize θ₀ = ˜θ

– For t = 1 to length of epochs θ_t = θ_t−1 − γh

g^′(˜θ) + f_i(t)^′ (θ_t−1) − f_i(t)^′ (˜θ)i – Update θ˜ = θ_t

• Output: θ˜

– No need to store gradients - two gradient evaluations per inner step – Two parameters: length of epochs + step-size γ

– Same linear convergence rate as SAG, simpler proof

Interpretation of SAG as variance reduction

• SAG update: θ_t = θ_t−1 − γ n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise – Interpretation as lazy gradient evaluations

Interpretation of SAG as variance reduction

• SAG update: θ_t = θ_t−1 − γ n

n

X

i=1

y_i^t with y_i^t =

(f_i^′(θ_t−1) if i = i(t) y_i^t−1 otherwise – Interpretation as lazy gradient evaluations

• SAG update: θ_t = θ_t−1 − γh

1 n

Pn

i=1 y_i^t−1 + _n¹ f_i(t)^′ (θ_t−1) − y_i(t)^t−1i – Biased update (expectation w.r.t. to i(t) not equal to full gradient)