part 2

(1)

Non-Convex Non-IID Non-Stochastic Non-Serial

Stochastic Variance-Reduced Optimization for Machine Learning

Parts 2: Weakening the Assumptions

Presenters: Francis Bach and Mark Schmidt

2017 SIAM Conference on Optimization

May 23, 2017

(2)

Outline

1 Non-Convex

2 Non-IID

3 Non-Stochastic

4 Non-Serial

(3)

Linear of Convergence of Gradient-Based Methods

We’ve seen a variety of results of the form:

Smoothness + Strong-Convexity ⇒ Linear Convergence Error on iterationt isO(ρ^t), or we need O(log(1/)) iterations.

But even simple models are oftennot strongly-convex. Least squares, logistic regression, SVMs with bias, etc.

How much can we relax strong-convexity?

Smoothness + ???

Strong-Convexity ⇒ Linear Convergence

(4)

Linear of Convergence of Gradient-Based Methods

But even simple models are oftennot strongly-convex.

Least squares, logistic regression, SVMs with bias, etc.

Smoothness + ???

(5)

Linear of Convergence of Gradient-Based Methods

But even simple models are oftennot strongly-convex.

Least squares, logistic regression, SVMs with bias, etc.

Smoothness + ???

(6)

Polyak- Lojasiewicz (PL) Inequality

For example, in 1963 Polyak showed linear convergence of GD only assuming 1

2k∇f(x)k²≥µ(f(x)−f^∗), that gradient grows as quadratic function of sub-optimality.

Holds for SC problems, but also problems of the form f(x) =g(Ax), for strongly-convexg. Includes least squares, logistic regression (on compact set), etc. A special case of the Lojasiewicz’ inequality [1963].

We’ll call this thePolyak- Lojasiewicz (PL) inequality. Using the PL inequality we can show

Smoothness + PL Inequality

(7)

Polyak- Lojasiewicz (PL) Inequality

Holds for SC problems, but also problems of the form f(x) =g(Ax), for strongly-convexg. Includes least squares, logistic regression (on compact set), etc.

A special case of the Lojasiewicz’ inequality [1963]. We’ll call this thePolyak- Lojasiewicz (PL) inequality. Using the PL inequality we can show

Smoothness + PL Inequality

(8)

Polyak- Lojasiewicz (PL) Inequality

A special case of the Lojasiewicz’ inequality [1963].

We’ll call this thePolyak- Lojasiewicz (PL) inequality.

Using the PL inequality we can show Smoothness + PL Inequality

(9)

Polyak- Lojasiewicz (PL) Inequality

A special case of the Lojasiewicz’ inequality [1963].

We’ll call this thePolyak- Lojasiewicz (PL) inequality.

Using the PL inequality we can show Smoothness + PL Inequality

(10)

PL Inequality and Invexity

PL inequality doesn’t require uniqueness or convexity.

However, it impliesinvexity.

For smoothf, invexity↔all stationary points are global optimum. Example of invex but non-convex function satisfying PL:

f(x) =x²+ 3 sin²(x).

Gradient descent converges linearly on this non-convex problem.

(11)

PL Inequality and Invexity

For smoothf, invexity↔all stationary points are global optimum.

Example of invex but non-convex function satisfying PL: f(x) =x²+ 3 sin²(x).

(12)

PL Inequality and Invexity

For smoothf, invexity↔all stationary points are global optimum.

Example of invex but non-convex function satisfying PL:

f(x) =x²+ 3 sin²(x).

(13)

Weaker Conditions than Strong Convexity (SC)

How does PL inequality [1963] relate to more recent conditions?

EB:error bounds [Luo and Tseng, 1993]. QG:quadratic growth[Anitescu, 2000]

ESC:essential strong convexity[Liu et al., 2013]. RSI:restricted secant inequality[Zhang & Yin, 2013].

RSI plus convexity is “restricted strong-convexity”. Semi-strong convexity[Gong & Ye, 2014].

Equivalent to QG plus convexity.

Optimal strong convexity[Liu & Wright, 2015]. Equivalent to QG plus convexity.

WSC:weak strong convexity [Necoara et al., 2015]. Proofs are often more complicated under these conditions. Are they more general?

(14)

Weaker Conditions than Strong Convexity (SC)

EB:error bounds[Luo and Tseng, 1993].

QG:quadratic growth[Anitescu, 2000]

(15)

Weaker Conditions than Strong Convexity (SC)

(16)

Weaker Conditions than Strong Convexity (SC)

ESC:essential strong convexity[Liu et al., 2013].

RSI:restricted secant inequality[Zhang & Yin, 2013]. RSI plus convexity is “restricted strong-convexity”. Semi-strong convexity[Gong & Ye, 2014].

(17)

Weaker Conditions than Strong Convexity (SC)

RSI:restricted secant inequality[Zhang & Yin, 2013].

RSI plus convexity is “restricted strong-convexity”.

Semi-strong convexity[Gong & Ye, 2014]. Equivalent to QG plus convexity.

(18)

Weaker Conditions than Strong Convexity (SC)

Semi-strong convexity[Gong & Ye, 2014].

(19)

Weaker Conditions than Strong Convexity (SC)

Optimal strong convexity[Liu & Wright, 2015].

(20)

Weaker Conditions than Strong Convexity (SC)

WSC:weak strong convexity [Necoara et al., 2015].

Proofs are often more complicated under these conditions. Are they more general?

(21)

Weaker Conditions than Strong Convexity (SC)

WSC:weak strong convexity [Necoara et al., 2015].

Proofs are often more complicated under these conditions.

Are they more general?

(22)

Relationships Between Conditions

For a function f with a Lipschitz-continuous gradient, we have:

(SC) →(ESC) → (WSC)→ (RSI)→ (EB) ≡(PL)→ (QG).

If we further assume that f is convex, then

(RSI) ≡(EB) ≡(PL) ≡(QG).

QG is the weakest condition but allows non-global local minima. PL ≡ EB are most general conditionsgiving global min.

(23)

Relationships Between Conditions

(RSI) ≡(EB) ≡(PL) ≡(QG).

QG is the weakest condition but allows non-global local minima. PL ≡ EB are most general conditionsgiving global min.

(24)

Relationships Between Conditions

(RSI) ≡(EB) ≡(PL) ≡(QG).

QG is the weakest condition but allowsnon-global local minima.

PL ≡ EB are most general conditionsgiving global min.

(25)

Relationships Between Conditions

(RSI) ≡(EB) ≡(PL) ≡(QG).

QG is the weakest condition but allowsnon-global local minima.

PL ≡ EB are most general conditionsgiving global min.

(26)

Convergence of Huge-Scale Methods

For large datasets, we typically don’t use GD.

But the PL inequalitycan be used to analyze other algorithms.

It has now been used to analyze:

Classic stochastic gradientmethods [Karimi et al., 2016]: O(1/k) without strong-convexityusing basic method. Coordinate descent methods [Karimi et al, 2016]. Frank-Wolfe [Garber & Hazan, 2015].

Variance-reduced stochastic gradient(like SAGA and SVRG) [Reddi et al., 2016]. Linear convergence without strong-convexity.

(27)

Convergence of Huge-Scale Methods

Classic stochastic gradientmethods [Karimi et al., 2016]:

O(1/k) without strong-convexityusing basic method.

Coordinate descent methods [Karimi et al, 2016].

Frank-Wolfe [Garber & Hazan, 2015].

Variance-reduced stochastic gradient(like SAGA and SVRG) [Reddi et al., 2016]. Linear convergence without strong-convexity.

(28)

Convergence of Huge-Scale Methods

Classic stochastic gradientmethods [Karimi et al., 2016]:

O(1/k) without strong-convexityusing basic method.

Coordinate descent methods [Karimi et al, 2016].

Frank-Wolfe [Garber & Hazan, 2015].

Variance-reduced stochastic gradient(like SAGA and SVRG) [Reddi et al., 2016].

Linear convergence without strong-convexity.

(29)

Relevant Problems for Proximal-PL

Proximal-PLis a generalization for non-smooth composite problems.

Reddi et al. [2016] analyzeproximal-SVRGandproximal-SAGA.

Proximal-PL is satisfied when:

f is strongly-convex.

f satisfies PL andg is constant.

f =h(Ax) for strongly-convexhandg is indicator of polyhedral set.

F is convex and satisfies QG (SVM and LASSO)

Any problem satisfying KL inequality or error bounds (equivalent to these). Group L1-regularization, nuclear-norm regularization.

Another important problem class: principal component analysis(PCA) Non-convex and doesn’t satisfy PL, but we can find global optimum. But itsatisfies PL on Riemannian manifold[Zhang et al., 2016].

Newfaster method based on SVRG[Shamir, 2015, Garber & Hazan, 2016].

(30)

Relevant Problems for Proximal-PL

Any problem satisfying KL inequality or error bounds (equivalent to these).

Group L1-regularization, nuclear-norm regularization.

Another important problem class: principal component analysis(PCA) Non-convex and doesn’t satisfy PL, but we can find global optimum. But itsatisfies PL on Riemannian manifold[Zhang et al., 2016].

(31)

Relevant Problems for Proximal-PL

Another important problem class: principal component analysis (PCA) Non-convex and doesn’t satisfy PL, but we can find global optimum.

But itsatisfies PL on Riemannian manifold[Zhang et al., 2016].

(32)

Relevant Problems for Proximal-PL

Another important problem class: principal component analysis (PCA) Non-convex and doesn’t satisfy PL, but we can find global optimum.

But itsatisfies PL on Riemannian manifold[Zhang et al., 2016].

(33)

But can we say anything about general non-convexfunctions?

What if all we know is ∇f is Lipschitz and f is bounded below?

(34)

Non-Convex Rates for Gradient Descent

For strongly-convexfunctions, GD satisfies

kx_t−x∗k²=O(ρ^t).

For convex functions, for GD still satisfies

f(x^t)−f(x^∗) =O(1/t).

For non-convex and bounded below functions, GD still satisfies mink≤tk∇f(x^k)k² =O(1/t),

a convergence rate in terms of getting to a critical point[Nesterov, 2003].

(35)

Non-Convex Rates for Gradient Descent

For strongly-convexfunctions, GD satisfies

kx_t−x∗k²=O(ρ^t).

For convex functions, for GD still satisfies

f(x^t)−f(x^∗) =O(1/t).

For non-convex and bounded belowfunctions, GD still satisfies mink≤tk∇f(x^k)k² =O(1/t),

a convergence rate in terms of getting to a critical point[Nesterov, 2003].

(36)

Non-Convex Rates for Stochastic Gradient

For stochastic gradientmethods, Ghadimi & Lan [2013] show a similar result, E[k∇f(x^k)k²] =O(1√

t), for a randomly-chosen k ≤t.

For variance-reducedmethods, Reddi et al. [2016] show we get faster rate, E[k∇f(x^k)k²] =O(1/t),

for a randomly-chosen k ≤t.

(37)

Non-Convex Rates for Stochastic Gradient

For stochastic gradientmethods, Ghadimi & Lan [2013] show a similar result, E[k∇f(x^k)k²] =O(1√

t), for a randomly-chosen k ≤t.

For variance-reducedmethods, Reddi et al. [2016] show we get faster rate, E[k∇f(x^k)k²] =O(1/t),

for a randomly-chosen k ≤t.

(38)

Non-Convex Rates for Stochastic Gradient

(39)

Non-Convex Rates for Stochastic Gradient

Number of gradient evaluations to guarantee -close to critical:

Gradient descent O(n/) Stochastic gradient O(1/²)

Variance-reduced O(n+n^2/3/)

We have analogous results forvariance-reduced proximal+stochastic methods.

[Reddi et al., 2016]

We cannot show analogous results for classic proximal stochastic methods. All existing proximal+stochastic results require noise to go to zero.

Open problemthat needs to be resolved: are analogous results possible?

(40)

Non-Convex Rates for Stochastic Gradient

Gradient descent O(n/) Stochastic gradient O(1/²) Variance-reduced O(n+n^2/3/)

(41)

Non-Convex Rates for Stochastic Gradient

(42)

Non-Convex Rates for Stochastic Gradient

We cannot show analogous results for classic proximal stochastic methods.

All existing proximal+stochastic results require noise to go to zero.

(43)

Outline

1 Non-Convex

2 Non-IID

3 Non-Stochastic

4 Non-Serial

(44)

Non-IID Setting

We discussed stochastic minimizationproblems argmin

x E[f_i(x)], where we have the ability to generate IID samplesf_i(x).

Using IID samples is justified by the law of large numbers.

But it’salmost never true.

What if wecan’t get IID samples?

Classic non-IID sampling scheme [Bertsekas & Tsitsiklis, 1996]: Samples follow a Markov chainwith stationary distribution ofE[f_i(x)].

Obtain standard guarantees if Markov chain mixes fast enough [Duchi et al., 2012].

(45)

Non-IID Setting

Classic non-IID sampling scheme [Bertsekas & Tsitsiklis, 1996]: Samples follow a Markov chainwith stationary distribution ofE[f_i(x)].

(46)

Non-IID Setting

Classic non-IID sampling scheme [Bertsekas & Tsitsiklis, 1996]:

Samples follow a Markov chainwith stationary distribution ofE[f_i(x)].

(47)

General Sampling

What about general non-IID sampling schemes?

What if oursamples f_i come from an adversary? Can we say anything in this case?

Optimization error can bearbitrarily bad, but we can boundregret...

(48)

General Sampling

What if oursamples f_i come from an adversary?

Can we say anything in this case?

(49)

General Sampling

What if oursamples f_i come from an adversary?

Can we say anything in this case?

(50)

Online Convex Optimization

Consider theonline convex optimization (OCO) framework [Zinkevich, 2003]:

At timet, make a predictionx^t.

Receive nextarbitrary convex lossf_t. Pay a penalty off_t(x^t).

The regretat time t is given by

t

X

k=1

[f_k(x^k)−f_k(x^∗)],

thetotal error compared to the bestx^∗ we could have chosenfor firstt functions. The x^∗ is not the solution to the problem, it’s just the best we could have done. The x^∗ depends on t, the“solution” is changing over time.

(51)

Online Convex Optimization

At timet, make a predictionx^t. Receive nextarbitrary convex lossft.

Pay a penalty off_t(x^t). The regretat time t is given by

t

X

k=1

[f_k(x^k)−f_k(x^∗)],

(52)

Online Convex Optimization

At timet, make a predictionx^t. Receive nextarbitrary convex lossft. Pay a penalty off_t(x^t).

t

X

k=1

[f_k(x^k)−f_k(x^∗)],

(53)

Online Convex Optimization

t

X

k=1

[f_k(x^k)−f_k(x^∗)],

thetotal error compared to the bestx^∗ we could have chosenfor firstt functions.

The x^∗ is not the solution to the problem, it’s just the best we could have done. The x^∗ depends on t, the“solution” is changing over time.

(54)

Online Convex Optimization

t

X

k=1

[f_k(x^k)−f_k(x^∗)],

thetotal error compared to the bestx^∗ we could have chosenfor firstt functions.

The x^∗ is not the solution to the problem, it’s just the best we could have done.

The x^∗ depends on t, the“solution” is changing over time.

(55)

Online Convex Optimization

Assuming everything is bounded, doingnothinghas a regret ofO(t).

Consider applying stochastic gradient, treating the ft as the samples. For convex functions, has a regret ofO(√

t) [Zinkevich, 2003]. For strongly-convex, has a regret ofO(log(t)) [Hazan et al., 2006]. These areoptimal.

Key idea: x^∗ isn’t moving faster than stochastic gradient is converging.

(56)

Online Convex Optimization

Consider applying stochastic gradient, treating the ft as the samples.

For convex functions, has a regret ofO(√

t) [Zinkevich, 2003].

For strongly-convex, has a regret ofO(log(t)) [Hazan et al., 2006].

These areoptimal.

(57)

Online Convex Optimization

Consider applying stochastic gradient, treating the ft as the samples.

For convex functions, has a regret ofO(√

t) [Zinkevich, 2003].

For strongly-convex, has a regret ofO(log(t)) [Hazan et al., 2006].

These areoptimal.

(58)

Online Convex Optimization

AdaGrad is a very-popular online method [Duchi et al., 2011]:

Improves on constants in regret bounds using diagonal-scaleing x^t+1=x^t−αtD_t⁻¹∇ft(x^t), with diagonal entries (Dt)ii =δ+

q Pt

k=1∇ifk(x^k).

Adam is a generalization that is incredibly-popular for deep learning.

[Kingma & Ba, 2015]

Though trend is returning to variations on accelerated stochastic gradient. Online learning remains active area and many variations exist:

Banditmethods only receive evaluationft(x^t) rather than functionft. Main application: internet advertising and recommender systems.

(59)

Online Convex Optimization

AdaGrad is a very-popular online method [Duchi et al., 2011]:

Improves on constants in regret bounds using diagonal-scaleing x^t+1=x^t−αtD_t⁻¹∇ft(x^t), with diagonal entries (Dt)ii =δ+

q Pt

k=1∇ifk(x^k).

Adam is a generalization that is incredibly-popular for deep learning.

[Kingma & Ba, 2015]

Though trend is returning to variations on accelerated stochastic gradient.

Online learning remains active area and many variations exist:

Banditmethods only receive evaluationft(x^t) rather than functionft. Main application: internet advertising and recommender systems.

(60)

Outline

1 Non-Convex

2 Non-IID

3 Non-Stochastic

4 Non-Serial

(61)

Graph-Structured Optimization

Another structure arising in machine learning is graph-structured problems, argmin

x

X

(i,j)∈E

fij(xi,xj) +

n

X

i=1

fi(xi).

whereE is the set of edges in graph.

Includes quadratic functions,

x^TAx+b^Tx=

n

X

i=1 n

X

j=1

aijxixj +

n

X

i=1

bixi,

and other models likelabel propagation for semi-supervised learning. Thegraph is sparsity patternofA.

(62)

Graph-Structured Optimization

Another structure arising in machine learning is graph-structured problems, argmin

x

X

(i,j)∈E

fij(xi,xj) +

n

X

i=1

fi(xi).

whereE is the set of edges in graph.

Includes quadratic functions,

x^TAx+b^Tx=

n

X

i=1 n

X

j=1

aijxixj +

n

X

i=1

bixi,

and other models likelabel propagation for semi-supervised learning.

Thegraph is sparsity pattern ofA.

(63)

Coordinate Descent for Graph-Structured Optimization

Coordinate descentseems well-suited to this problem structure:

argmin

x

X

(i,j)∈E

fij(xi,xj) +

n

X

i=1

fi(xi).

To update x_i, we only need to consider f_i and the f_ij for each neighbour.

Withrandom selection of coordinates, expected iteration cost is O(|E|/n). This isn-times faster than GD iteration which costO(|E|).

(64)

Coordinate Descent for Graph-Structured Optimization

Coordinate descentseems well-suited to this problem structure:

argmin

x

X

(i,j)∈E

fij(xi,xj) +

n

X

i=1

fi(xi).

To update x_i, we only need to consider f_i and the f_ij for each neighbour.

Withrandom selection of coordinates, expected iteration cost is O(|E|/n).

This isn-times faster than GD iteration which costO(|E|).

(65)

Coordinate Descent for Graph-Structured Optimization

But for many problems randomized coordinate descent doesn’t work well...

Epochs

0 5 10 15 20 25 30

Objective

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Cyclic Random

Lipschitz

GS GSL

Graph-based label propagation

Often outperformed by the greedy Gauss-Southwell rule.

But is plotting “epochs” cheating because Gauss-Southwell is more expensive?

(66)

Coordinate Descent for Graph-Structured Optimization

But for many problems randomized coordinate descent doesn’t work well...

Epochs

0 5 10 15 20 25 30

Objective

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Cyclic Random

Lipschitz

GS GSL

Graph-based label propagation

Often outperformed by the greedy Gauss-Southwell rule.

But is plotting “epochs” cheating because Gauss-Southwell is more expensive?

(67)

Greedy Coordinate Descent

Gauss-Southwell greedy rule for picking a coordinate to update:

argmax

i

|∇_if(x)|.

x1 x2 x3

Looks expensive because computing the gradient costsO(|E|).

(68)

Greedy Coordinate Descent

argmax

i

|∇_if(x)|.

x1 x2 x3

(69)

Greedy Coordinate Descent

argmax

i

|∇_if(x)|.

x1 x2 x3

(70)

Greedy Coordinate Descent

argmax

i

|∇_if(x)|.

x1 x2 x3

Gauss-Southwell

(71)

Greedy Coordinate Descent

argmax

i

|∇_if(x)|.

x1 x2 x3

Gauss-Southwell

(72)

Cost of Greedy Coordinate Descnet

Gauss-Southwell cost depends on graph structure.

Same is true of Lipschitz sampling.

Consider problems where maximum degree and average degree are similar: Lattice graphs (max is 4, average is≈4).

Complete graphs (max and average degrees aren−1). Facebook graph (max is 7000, average is≈200).

Here we can efficientlytrack the gradient and it’s max [Meshi et al., 2012]. Updatingxi, it only changes|∇jf(x^k)|fori and its neighbours.

We can use amax-heap to track the maximum.

(73)

Cost of Greedy Coordinate Descnet

Consider problems where maximum degree and average degree are similar:

Lattice graphs (max is 4, average is≈4).

Complete graphs (max and average degrees aren−1). Facebook graph (max is 7000, average is≈200).

(74)

Cost of Greedy Coordinate Descnet

Complete graphs (max and average degrees aren−1).

Facebook graph (max is 7000, average is≈200).

(75)

Cost of Greedy Coordinate Descnet

(76)

Cost of Greedy Coordinate Descnet

Here we can efficientlytrack the gradient and it’s max [Meshi et al., 2012].

Updatingxi, it only changes|∇jf(x^k)|fori and its neighbours. We can use amax-heap to track the maximum.

(77)

Cost of Greedy Coordinate Descnet

Here we can efficientlytrack the gradient and it’s max [Meshi et al., 2012].

Updatingxi, it only changes|∇jf(x^k)|fori and its neighbours.

(78)

Convergence Rate of Greedy Coordinate Descent

But don’t random and greedy have the same rate?

Nutini et al. [2015] show that rate for Gauss-Southwell is f(x^k)−f^∗ ≤

1−µ₁ L

k

[f(x⁰)−f^∗)], where µ1 is strong-convexity constant in the 1-norm.

Constantµ1 satisfies

µ n

|{z}

random

≤ µ₁

|{z}

greedy

≤ µ

|{z}

gradient

,

so we should expect more progress under Gauss-Southwell.

(79)

Convergence Rate of Greedy Coordinate Descent

1−µ₁ L

k

[f(x⁰)−f^∗)], whereµ1 is strong-convexity constant in the 1-norm.

µ n

|{z}

random

≤ µ₁

|{z}

greedy

≤ µ

|{z}

gradient

,

(80)

Convergence Rate of Greedy Coordinate Descent

1−µ₁ L

k

[f(x⁰)−f^∗)], whereµ1 is strong-convexity constant in the 1-norm.

µ n

|{z}

random

≤ µ₁

|{z}

greedy

≤ µ

|{z}

gradient

,

(81)

Gauss-Southwell-Lipschitz Rule

Nutini et al. [2015] also give a rule with faster rate by incorporating theL_i, ik = argmax

i

|∇_if(x^k)|

√Li

, which is called the Gauss-Southwell-Lipschitzrule.

At least as fast as GS and Lipschitz samplingrules.

Intuition: if gradients are similar, more progress if L_i is small.

x1

x2

Greedy rules have lead to new methods for computing leading eigenvectors. Coordinate-wise power methods[Wei et al., 2016, Wang et al., 2017].

(82)

Gauss-Southwell-Lipschitz Rule

i

|∇_if(x^k)|

√Li

Intuition: if gradients are similar, more progress ifL_i is small.

x1

x2

(83)

Gauss-Southwell-Lipschitz Rule

i

|∇_if(x^k)|

√Li

Intuition: if gradients are similar, more progress ifL_i is small.

x1

x2 Gauss-Southwell