Acceleration in First Order Quasi-strongly
Convex Optimization by ODE Discretization
The MIT Faculty has made this article openly available.
Please share
how this access benefits you. Your story matters.
Citation
Zhang, Jingzhao et al. “Acceleration in First Order Quasi-strongly
Convex Optimization by ODE Discretization.” Paper presented in
the Proceedings of the IEEE Conference on Decision and Control,
Nice, France , December 11-13 2019, Institute of Electrical and
Electronics Engineers (IEEE) © 2019 The Author(s)
As Published
10.1109/CDC40024.2019.9030046
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Version
Original manuscript
Citable link
https://hdl.handle.net/1721.1/130426
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Acceleration in First Order Quasi-strongly Convex
Optimization by ODE Discretization
Jingzhao Zhang, Suvrit Sra and Ali Jadbabaie
Abstract— We study gradient-based optimization methods obtained by direct Runge-Kutta discretization of the ordinary differential equation (ODE) describing the movement of a heavy-ball under constant friction coefficient. When the func-tion is high order smooth and strongly convex, we show that directly simulating the ODE with known numerical integrators achieve acceleration in a nontrivial neighborhood of the optimal solution. In particular, the neighborhood can grow larger as the condition number of the function increases. Furthermore, our results also hold for nonconvex but quasi-strongly convex objectives. We provide numerical experiments that verify the theoretical rates predicted by our results.
I. INTRODUCTION
In this paper, we study accelerated first-order optimization algorithms for the problem
min
x∈Rd f (x), (1)
where f satisfies the following properties.
Assumption 1: Denote x∗ the unique optimal solution of f . Assume function f is quasi-strongly convex, i.e. ∀x,
f (x∗) ≥ f (x) + h∇f (x), x∗− xi + µ2kx − x∗k2.
Assumption 2: Function f is L−smooth, i.e. ∀x, y, k∇f (x) − ∇f (y)k ≤ Lkx − yk.
Assumption 1 is implied by the standard strong-convexity assumption and is weaker as the inequality only needs to hold when one of the two points is the optimal solution. Further, Assumption 1 does not imply convexity (see [1]). A classical method for solving (1) is gradient descent (GD), which requires T = O(Lµ log(f (x0)−f (x∗)
) iterations to achieve
accuracy (f (xT) − f (x∗) ≤ ). With the additional global
convexity assumption, it is shown in [1] that Nesterov’s accelerated gradient (NAG) method [2] matches the oracle lower bound of O(qLµlog(f (x0)−f (x∗)
) [3]. However, up to
our knowledge, no acceleration in terms of condition number dependency is achieved without the convexity assumption.
In this work, we show that with the higher order smooth-ness assumption, known Runge-Kutta discretization of the standard heavy-ball ODE (4) achieves better dependency of
L
µ compared to gradient descent in a neighborhood of the
optimal points. Unlike local results, we show that the size of the neighborhood can potentially increases with the condition number Lµ. For more details, please refer to Theorem 4.
The authors are with the Laboratory for Information and Decision Systems (LIDS), and the Institute for Data, Systems, and Society (IDSS), Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139 {jzhzhang, suvrit, jadbabai}@mit.edu
*This research was supported in part by DARPA Lagrange.
II. RELATED WORK
Acceleration of first order optimization algorithms has long been studied. Polyak proposed the heavy ball method [4] that is provably faster than gradient descent for optimizing quadratic objectives. Nesterov later designed ac-celeration methods for general smooth convex objectives [2]. It was recently proven in [1] that this algorithm also achieves optimal rate under global convex and quasi-strongly convex conditions. However, ever since its introduction, acceleration has remained somewhat mysterious, especially because Nes-terov’s original derivation relies on elegant but unintuitive algebraic arguments. This lack of understanding has spurred a variety of recent attempts to uncover the rationale behind the phenomenon of acceleration [5]–[10].
Our approach follows the sequence of work on the ODE interpretation of optimization algorithms. The ODE inter-pretation has long been studied, for example in [11]–[14]. These work analyzed the asymptotic behavior of dissipative dynamical systems. The more recent sequence of work on the ordinary ODE interpretation of the acceleration took more interests in the non-asymptotic analysis and discretized algorithms. This line of research starts with [15], who showed that the continuous limit of NAG is a second order ODE describing a physical system with vanishing friction. Later, [16]–[22] generalized this idea to study the continuous limit of other first order algorithms such as mirror descent, ADMM and rescaled gradient descent. [23]–[27] studied the discretization of ODE and aim to design accelerated algorithms with non-asymptotic convergence guarantee.
Most of the literature requires knowing Nesterov’s method beforehand to study acceleration. Our contribution is to show that acceleration can be achieved starting from the heavy-ball ODE without knowing NAG. The most relevant work to ours would be [23], [25]. We highlight two differences. First, unlike [23] which discretizes a high order ODE approxima-tion of NAG, we discretize an ODE that describes the heavy ball mechanical system. Compared to [23], our approach can produce accelerated algorithms without knowing NAG first. Second, we focus on quasi-strongly convex problems, which is different from the setting of both work.
III. PRELIMINARIES
Before presenting the main result, we will go over some tools used in our algorithms and analysis. The first one is a class of numerical discretization algorithms named Runge-Kutta integrators. The second one is a notation system that simplifies the expression for high order derivatives of autonomous dynamical systems.
A. Runge-Kutta Integration
We briefly recall explicit Runge-Kutta (RK) integrators used in our work. For a more in depth discussion please see the textbook [28].
Definition 1: Given a dynamical system ˙y = F (y), let the current point be y0 and the step size be h. An explicit
S stage Runge-Kutta method generates the next step via the following update: gi= y0+ h i−1 X j=1 aijF (gj), Φh(y0) = y0+ h S X i=1 biF (gi),
where aij and bi are suitable coefficients defined by the
integrator; Φh(y0) is the estimation of the state after time
step h, while gi (for i = 1, . . . , S) are a few neighboring
points where the gradient information F (gi) is evaluated.
By combining the gradients at several evaluation points, the integrator can achieve higher precision by matching up Taylor expansion coefficients. Let ϕh(y0) be the true solution
to the ODE with initial condition y0. Then we define the
order of an integrator below.
Definition 2: we say that an integrator Φh(y0) has order
s if its discretization error shrinks as
kΦh(y0) − ϕh(y0)k = O(hs+1), as h → 0. (2)
In general, RK methods offer a powerful class of numerical integrators, encompassing several basic schemes. The explicit Euler’s method defined by Φh(y0) = y0+ hF (y0) is an
explicit RK method of order 1, while the midpoint method Φh(y0) = y0+ hF (y0+h2F (y0)) is of order 2. Some
high-order RK methods are summarized in [29]. An high-order 4 RK method requires 4 stages, i.e., 4 gradient evaluations, while an order 9 method requires 16 stages.
B. Elementary differentials
We briefly summarize some key results on elementary differentials from [28]. For more details, please refer to chapter 3 of the book. Given a dynamical system
˙
y = F (y),
we want to find a convenient way to express and compute its higher order derivatives. To do this, let τ denote a tree structure as illustrated in Figure 1. |τ | is the number of nodes in τ . Then we can adopt the following notations as in [28]. Definition 3: For a tree τ , the elementary differential is a mapping F (τ ) : Rd→ Rd, defined recursively by F (•)(y) =
F (y) and
F (τ )(y) = ∇(m)F (y)[F (τ1)(y), ..., F (τm)(y)],
for τ = [τ1, ..., τm]. Notice thatP m
i=1|τi| = |τ | − 1.
Some examples are shown in Figure 1. With this notation, the following results from [28] Chapter 3.1 hold. The proof follows by recursively applying the product rule.
Fig. 1: An illustration from [28]. Example tree structures and corresponding function derivatives.
Lemma 1: The qth order derivative of the exact solution to ˙y = F (y) is given by dqy(t) dtq |t=tc= y (q)(t c) = F(q−1)(yc) = X |τ |=q α(τ )F (τ )(yc),
for y(tc) = yc. α(τ ) is a positive integer determined by τ
and counts the number of occurrences of the tree pattern τ . The next result is obtained by general Leibniz rule. The expression for ∂qF (gi)
∂hq can be calculated the same way as
in Lemma 1.
Lemma 2: For a Runge-Kutta method defined in definition 1, if F is qth differentiable, then dqΦ h(yc) dhq = X i≤S bi(h dqF (g i) dhq + q dq−1F (g i) dhq ), (3) where∂qF (gi)
∂hq has the same structure as F
(q)
(y) in lemma 1, except that we need to replace all F in the expression by ∂gi
∂h
and all ∇(n)F (y) by ∇(n)F (g
i).
IV. MAIN RESULTS
In this section, we will propose a family of algorithms generated via ODE discretization. We will also provide theoretical convergence guarantee and go over the proofs. A. Algorithm
In this subsection, we introduce a second-order dynamical system and use explicit RK integrators to generate iterates that converge to the optimal solution at a rate faster than gradient descent (O(Lµlog(f (x0)−f (x∗)
)). We start with the
second-oder ODE, ¨
x(t) + 2 ˙x(t) +∇f (x)
µ = 0. (4)
This ODE has very nice physical interpretation. It describes the mechanics of a heavy ball moving on the surface defined by f . Intuitively, due to friction, the heavy ball will stop
Algorithm 1: Input(f, x0, L, µ, s, N )
. Constants µ, L are the same as in Assumptions 1: Set the initial state y0= [~0; x0] ∈ R2d
2: Set step size h = C/Ns+11 . . C is determined by
µ, L, s, x0
3: xN ← Order-s-Runge-Kutta-Integrator(F, y0, N, h) . F
is defined in equation (6) 4: return xN
at the unique optimal of f . This second order ODE can be written as an autonomous dynamical system.
˙ y = ˙v ˙ x = F (y) =−2v − ∇f (x)/µ v . (5) As the stability of discretization is determined by the largest numerical error along any coordinate, we would like to balance the Lipschitz constant of x and v. This step is crucial to achieving acceleration. Particularly, we make a change of variable w = v/√Q, where Q = L
µ is the
condition number. Hence with some abuse of notations, we can rewrite the previous system in the following form,
˙ y = ˙w ˙ x = F (y) = " −2w −∇f (x)µ√ Q √ Qw # . (6)
The algorithm we propose is simply to discretize the dy-namical system in (6) with an order-s explicit Runge-Kutta integrator. It is summarized in Algorithm 1.
B. Convergence analysis
In order to utilize the order conditions of Runge-Kutta integrators, we make the following assumptions on high order smoothness. Denote kM k the operator norm of an nth
order tensor M : R n z }| { d × ... × d → R. In other word, kM k = max{M [u1, ..., un]|ui∈ Rd, kuik2= 1} Then we assume,
Assumption 3: The high order derivatives of f exist and are bounded up to order s+1. In other words, k∇(n)f (x)k ≤ L, n = 2, ..., s + 1
For this nonlinear dynamical system, we can define the following Lyapunov function to help us prove convergence in both continuous time and discrete time.
E(y) =E([w; x]) = 2(f (x) − f (x∗))/µ +Q 2kwk 2+ 1 2kx + p Qw − x∗k2. (7)
The stability of the continuous system is justified by the following Lemma.
Lemma 3: For the dynamical system defined in (6), the Lyapunov function converges exponentially along the trajec-tory with rate
˙
E(y) ≤ −1 2E(y).
Proof:[of Lemma 3] ˙ E(y) = 2 µh∇f (x), p Qwi + Qhw, −2w − ∇f (x)/(pQµ)i + hx +pQw − x∗, −pQw −∇f (x)µ i = −2Qkwk2− hx − x∗, ∇f (x)i/µ − hx +pQw − x∗,pQwi ≤ −2Qkwk2− (f (x) − f (x∗) + µ/2kx − x∗k2)/µ −1 2(kx + p Qw − x∗k2+ kpQwk2− kx − x∗k2) ≤ −E(y)/2.
The first inequality follows from Assumption 1 and the fact that ha, bi = 12(kak2+ kbk2− ka − bk2).
Then we provide the following convergence guarantee. Theorem 4: Under the Assumptions 1 3, we discretize the dynamical system (6) with an order-s Runge-Kutta integrator. By setting h ≤ min{E(y0)−γ, Q−γ} 1
2c(s)s+11
, γ = s+3 2(s+1)
for some constant c(s) determined by the order s, we have, E(yN) ≤ (1 − h/4)NE(y0).
In other word, assuming the integrator has stage S, the number of gradients evaluated to get suboptimality is O(S max{E(y0), Q}γlog(E(yµ0))).
We would like to make two remarks about this result. First, for general quasi-strongly convex functions (with the exception of quadratics), the smoothness constant L would grow with E0. Second, when the initial point is selected
such that E0 ≤ Lµ, the algorithm achieves acceleration and
approaches the theoretical lower bound as the order of the integrator s increases.
C. Proof of main theorem
The high level idea is simple. Based on the convergence result of Lemma 3, we get exponential convergence as long as the discretization error stays bounded. Then the conver-gence rate would depend on the largest stable discretization step. We postpone the technical details to the next subsection and abstract out the main proof below.
Recall that ϕh(y0) is the true solution to the ODE with
initial condition y0, while Φh(y0) is the numerical solution
generated by the integrator. We rewrite
y(h) = ϕh(y), gS(h) = Φh(y), (8)
to emphasize the explicit dependency on h. By the order condition of the Runge-Kutta integrator, we know that the first s terms in the Taylor expansions of y0(h) and gS(h)
evaluated at h = 0 are the same. By Taylor’s theorem and triangle inequality, we know that
|E(gS(h)) − E (y(h))| ≤ hs+1 max 0≤δ≤h(| ds+1 dhs+1E(gS(δ))| + | ds+1 dhs+1E(y(δ))|).
Since E (y(h)) ≤ (1 − h/2)E (y), we have E(gS(h) ≤(1 − h/2)E (y)
+hs+1 max 0≤δ≤h(| ds+1 dhs+1E(gS(δ))| + | ds+1 dhs+1E(y(δ))|) ≤(1 − h/2)E(y) +c(s)hs+1 s X i=1
Q(i+1)/2E(y)(s−i+2)/2
≤(1 − h/4)E(y0).
The second inequality follows by Lemma 10 and Lemma 14 which bound the two derivative terms respectively. The last inequality follows by the choice of h.
D. Technical lemmas
In this section, we bound the high order derivatives of the dynamical system and the discretization algorithm as a function of time. Before presenting a few key lemmas, we define some quantities and frequently used inequalities.
Lemma 5: The norm of the dynamics is bounded by the Lyapunov function, kF (y)k ≤ 5pE(y). Proof: By (6), kF (y)k2= " −2w −∇f (x)µ√ Q √ Qw # 2 ≤ 2k∇f (x) µ√Q k 2+ 8kwk2+ Qkwk2.
By Lipschitz continuity of the gradient and optimality of f (x∗), we get that k∇f (x)k2≤ L(f (x) − f (x∗)) (see [30]).
Therefore, we have k∇f (x)
µ√Qk
2= k∇f (x)k2/(µL) ≤ (f (x) − f (x∗))/µ.
Substitute this in the inequality above and we proved the lemma.
Next, we bound the high order derivatives of the system dynamics F defined in (6).
Lemma 6: k∇(n)F (y)k ≤ 4√Q, ∀n ≥ 1.
Proof: This follows by Assumption 3 and computing k∇(n)F (y)k explicitly. Next, notice ∇E(y) = Qw +√Q(x +√Qw − x∗) 2∇f (x)/µ + (x +√Qw − x∗) . From this expression, we can tell that k∇E(y)k2≤ 4QE(y).
Further, it’s easy to verify that k∇(n)E(y)k ≤ 5Q, ∀n > 2.
The above inequalities serve as the base case of the Lemmas proven below with induction and will be used repeatedly.
In the rest of this section, we let F denote the function defined in (6) and F (τ ) denotes the differentials defined in section III-B. Then the next Lemma follows. For conciseness, we write E as a shorthand for E (y).
Lemma 7: The vector norms of elementary differen-tials F (τ ) evaluated at y can be bounded by the value
of the Lyapunov function E (y). More precisely, ∀k ≥ 2,P |τ |=kkF (τ )k ≤ c(k) Pk−1 i=1 Q k−i 2 E i 2.
Proof: We already know kF (•)k = kF (y)k ≤ 5√E and k∇F (y)k ≤ 4√Q. Hence the case when k = 2 follows by Cauchy-Schwartz. We prove the Lemma by induction and assume that for any tree |τ | = k ≤ l,
X |τ |=k kF (τ )k ≤ c(k) k−1 X i=1 Qk−i2 E i 2.
By definition of tree structures, for τ0 = [τ1, ..., τm] with
|τ0| = l + 1, m ≥ 1, We have kF (τ0)k ≤ k∇(m)F k m Y i=1 kF (τi)k.
Therefore, by the inductive assumption for some absolute onstant c(l) determined by l, X |τ0|=l+1 kF (τ0)k ≤ c(l + 1) l X i=1 Q l+1−i 2 E i 2.
With the above Lemma, we are able to show that the high order derivatives of the dynamical system are bounded.
Lemma 8: Recall the definition of y(h) in (8).
∀k ≥ 2, dk dhky(h) ≤ c(k) k−1 X i=1 Q k−i 2 E i 2.
Proof: This follows directly by Lemma 1 and 7. We have already bounded the high order derivatives of the continuous trajectory. However, notice that the bounds are point-wise in time. We would like to bound the variation of the Lyapunov function for a short time span.
Lemma 9: If we set h ≤ 10√1
Q, then E (y(h)) ≤ 3E (y(0)).
Proof: By Taylor expansion, we get E(y(h)) = E(y) + h d dhE(y(h))|h=0 +h 2 2 Z h 0 d2 dh2E(y(h))dτ. Since ˙E(y(0)) ≤ −1 2E(y), we have E(y(h)) ≤ (1 − h/2)E(y) +h 2 2 Z h 0 d2 dτ2E(y(τ ))dτ. We know that d2 dτ2E(y(τ ))) = ∇
(2)E(y(τ ))[F (y(τ )), F (y(τ ))]
+∇E (y(τ ))T∇F (y(τ ))F (y(τ )). Notice also that from previous Lemmas,
k∇(2)E(y(τ ))k ≤ 5Q, kF (y(τ ))k ≤ 5pE(y(τ)),
This implies E(y(h)) ≤ (1 − h/2)E(y) + 100h2Q Z h 0 E(y(τ ))dτ ≤ E(y) + Z h 0
E(y(τ )) ≤ ehE(y) ≤ 3E(y).
The last inequality follows by h ≤ 1.
We can then bound the derivatives of the Lyapunov function. Lemma 10: ∀k ≥ 2, h ≤ 1/(2√Q), dkE(y(h)) dhk ≤ c(k) k−1 X i=1
Q(i+1)/2E(k−i+1)/2(y(0)).
Proof: By chain rule, we have dn dtnE(y(t)) = X k1,...,kn n! k1!k2!...kn! ∇(k)E(y) n Y i=1 (d iy(t) dti /i!) ki.
where the sum is taken over {k1, ..., kn ∈ Z≥0|Pni=1iki =
n}, and k =Pn
i=1ki. Then the Lemma follows by Cauchy
inequality, Lemma 8, Lemma 9, and k∇(k)Ek ≤ 5Q, ∀k ≥ 2.
We have bounded the high order derivatives of the Lya-punov function along the true solution of the ODE. Next we will bound the counterpart for the points generated by discretization. Most of the steps would be almost the same. Hence we eliminate details due to limited space.
Lemma 11: Recall that gi(y) defined in Definition 1 is
a function of the step size h. When h ≤ 10√1
Q, ∀i =
1, ..., S, we have kdhd gi(y)k ≤ cpE(y) for some constant
c determined by the integration parameters aij in Definition
1.
Proof: Notice that dhd g1(y) = a10F (y). This proves
the base case. The we prove by induction and the fact that d dhgi(y) = X j<i aijF (gj) + h X j<i aij∇F (gj) d dhgj(y). The claim follows by the inductive assumption and k∇F (gj)k ≤ 4
√
Q (by Lemma 6).
With the above lemma, we can bound the Lyapunov of the neighboring points gi(y).
Lemma 12: If we set h ≤c√1
Q, where c is determined by
the numerical integrator, then E (gi(h)) ≤ 3E (y).
Proof: The proof is almost the same as Lemma 9. Then we bound the equivalent of Lemma 10.
Lemma 13: ∀i ≤ S, ∃c determined by k and the integrator such that, k d k dhkgi(y)k ≤ c(k, S) k−1 X i=1 Qk−i2 E2i.
Proof: The proof can be carried out in the same way as in Lemma 8, except that we need to make use of Lemma 2 instead of Lemma 1.
Finally, we can bound the high order derivative of the Lyapunov function along the discretization solution.
Lemma 14: ∀k ≥ 2, h ≤ 1/(c√Q), where c is determined by the integrator and k,
dkE(g S(h)) dhk ≤ C k−1 X i=1
Q(i+1)/2E(k−i+1)/2(y(0)).
The constant C is determined by k and the integrator. Proof: The proof is exactly the same as in Lemma 10 except that instead of using Lemma 8, we use Lemma 13 to bound the high order derivatives of the trajectories.
V. NUMERICAL EXPERIMENTS
In this section, we implement our method and test its performance with two different objectives. In the first ex-ample, we run the direct discretization (DD) algorithms with different integration order s for solving the strongly convex quadratic problem
f (x) = xTDiag(λ)x. (9) λ is a vector of singular values. The condition number
λmax
λmin = 500. Initial guess x0 is all ones. We compare
our methods with GD and NAG algorithms. We use the standard parameter choice for GD and NAG by substituting in the value for smoothness constant L and strongly convex constant µ. For our discretization algorithm, we choose the step size of the format h = 10z, where z is the largest
integer such that the discretization is stable. The convergence trajectory is shown in Figure 2.
In the second example, we tested our example for regu-larized logistics regression,
f (w) = N X i=1 log(1 + e−yixTiw) +γ 2kwk 2. (10)
We generate data with a two-cluster mixture of Gaussian model. Each cluster belongs to one class. Particularly, we set the margin large enough such that the data is linearly separable. Then the function is at least γ−strongly convex. We set the parameter for NAG using µ = γ. We then scan the value for smoothness constant L = 10z, where z is the
largest integer such that the discretization is stable for both GD and NAG. The results are shown in Figure 3.
Both experiments show similar results and confirm that direct discretization can be faster than gradient descent when the integration order s > 1. We also notice that for s = 4, direct discretization can be as fast as NAG or sometimes even faster. We provide a conjecture for this. First, we use the sub-optimal NAG parameter choice where step size is L1. Second, in the analysis, our convergence rate critically relies on the trajectory of the discretization follows closely to the true solution. However, this requirement is sufficient but not necessary. In fact, by looking at the figure, we notice that the suboptimality of the sequence generated by high-order discretization techinique does not oscillate, which suggests that they are not following the solution to the ODE. Hence, our convergence guarantee is conservative.
Fig. 2: Convergence paths for GD, NAG and direct dis-cretization algorithms. The objective is quadratic and de-scribed in (9).
Fig. 3: Convergence paths for GD, NAG and direct dis-cretization algorithms. The objective is a logistic regression loss described in (10).
VI. CONCLUSION
In this work, we showed that by directly discretizating the second order heavy ball ODE, we can obtain optimization algorithms that are provably faster than gradient descent. However, the acceleration can only be shown to exist in a local region surrounding the optimal solution. This locality is different from traditional locality assumption, because as the function nonlinearity L increase, the neighborhood grows larger. We believe this phenomenon results from the fact that our proof does not rely on the convexity of the function. Whether convexity can lead to faster rate in direct discretization remains an interesting research question.
VII. ACKNOWLEDGEMENT
The authors thank Aryan Mokhtari, Cesar Uribe and Juncal Arbelaiz Mugica for helpful discussions.
REFERENCES
[1] I. Necoara, Y. Nesterov, and F. Glineur, “Linear convergence of first order methods for non-strongly convex optimization,” Mathematical Programming, pp. 1–39, 2018.
[2] Y. Nesterov, “A method of solving a convex programming problem with convergence rate o (1/k2),” in Soviet Mathematics Doklady, vol. 27, no. 2, 1983, pp. 372–376.
[3] A. Nemirovskii, D. B. Yudin, and E. R. Dawson, “Problem complexity and method efficiency in optimization,” 1983.
[4] B. T. Polyak, “Some methods of speeding up the convergence of iter-ation methods,” USSR Computiter-ational Mathematics and Mathematical Physics, vol. 4, no. 5, pp. 1–17, 1964.
[5] Z. Allen-Zhu and L. Orecchia, “Linear coupling: An ultimate unifica-tion of gradient and mirror descent,” arXiv preprint arXiv:1407.1537, 2014.
[6] S. Bubeck, Y. T. Lee, and M. Singh, “A geometric alterna-tive to nesterov’s accelerated gradient descent,” arXiv preprint arXiv:1506.08187, 2015.
[7] L. Lessard, B. Recht, and A. Packard, “Analysis and design of opti-mization algorithms via integral quadratic constraints,” SIAM Journal on Optimization, vol. 26, no. 1, pp. 57–95, 2016.
[8] B. Hu and L. Lessard, “Dissipativity theory for nesterov’s accelerated method,” arXiv preprint arXiv:1706.04381, 2017.
[9] D. Scieur, A. d’Aspremont, and F. Bach, “Regularized nonlinear acceleration,” in Advances In Neural Information Processing Systems, 2016, pp. 712–720.
[10] M. Fazlyab, A. Ribeiro, M. Morari, and V. M. Preciado, “Analysis of optimization algorithms via integral quadratic constraints: Nonstrongly convex problems,” arXiv preprint arXiv:1705.03615, 2017.
[11] F. Alvarez, “On the minimizing property of a second order dissipative system in hilbert spaces,” SIAM Journal on Control and Optimization, vol. 38, no. 4, pp. 1102–1119, 2000.
[12] H. Attouch, X. Goudou, and P. Redont, “The heavy ball with friction method, i. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system,” Communications in Contemporary Mathematics, vol. 2, no. 01, pp. 1–34, 2000.
[13] R. E. Bruck Jr, “Asymptotic convergence of nonlinear contraction semigroups in hilbert space,” Journal of Functional Analysis, vol. 18, no. 1, pp. 15–26, 1975.
[14] H. Attouch and R. Cominetti, “A dynamical approach to convex min-imization coupling approximation with the steepest descent method,” Journal of Differential Equations, vol. 128, no. 2, pp. 519–540, 1996. [15] W. Su, S. Boyd, and E. Candes, “A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights,” in Advances in Neural Information Processing Systems, 2014, pp. 2510– 2518.
[16] A. Wibisono, A. C. Wilson, and M. I. Jordan, “A variational perspec-tive on accelerated methods in optimization,” Proc. of the National Academy of Sciences, vol. 113, no. 47, pp. E7351–E7358, 2016. [17] P. Xu, T. Wang, and Q. Gu, “Accelerated stochastic mirror descent:
From continuous-time dynamics to discrete-time algorithms,” in Inter-national Conference on Artificial Intelligence and Statistics, 2018, pp. 1087–1096.
[18] W. Krichene, A. Bayen, and P. L. Bartlett, “Accelerated mirror descent in continuous and discrete time,” in Advances in neural information processing systems, 2015, pp. 2845–2853.
[19] G. Franc¸a, D. P. Robinson, and R. Vidal, “Admm and accel-erated admm as continuous dynamical systems,” arXiv preprint arXiv:1805.06579, 2018.
[20] A. Barakat and P. Bianchi, “Convergence of the adam algorithm from a dynamical system viewpoint,” arXiv preprint arXiv:1810.02263, 2018. [21] A. Wilson, L. Mackey, and A. Wibisono, “Accelerating rescaled
gradient descent,” arXiv preprint arXiv:1902.08825, 2019.
[22] C. J. Maddison, D. Paulin, Y. W. Teh, B. O’Donoghue, and A. Doucet, “Hamiltonian descent methods,” arXiv preprint arXiv:1809.05042, 2018.
[23] B. Shi, S. S. Du, W. J. Su, and M. I. Jordan, “Acceleration via symplectic discretization of high-resolution differential equations,” 2019.
[24] J. Zhang, C. A. Uribe, A. Mokhtari, and A. Jadbabaie, “Achieving acceleration in distributed optimization via direct discretization of the heavy-ball ode,” arXiv preprint:1811.02521, 2018.
[25] J. Zhang, A. Mokhtari, S. Sra, and A. Jadbabaie, “Direct Runge-Kutta discretization achieves acceleration,” arXiv :1805.00521, 2018. [26] M. Betancourt, M. I. Jordan, and A. C. Wilson, “On symplectic
optimization,” arXiv preprint arXiv:1802.03653, 2018.
[27] B. Shi, S. S. Du, M. I. Jordan, and W. J. Su, “Understanding the acceleration phenomenon via high-resolution differential equations,” arXiv preprint arXiv:1810.08907, 2018.
[28] E. Hairer, C. Lubich, and G. Wanner, Geometric numerical integration: structure-preserving algorithms for ordinary differential equations. Springer Science & Business Media, 2006, vol. 31.
[29] J. Verner, “High-order explicit runge-kutta pairs with low stage order,” Applied numerical mathematics, vol. 22, no. 1-3, pp. 345–357, 1996. [30] Y. Nesterov, Introductory lectures on convex optimization: A basic
![Fig. 1: An illustration from [28]. Example tree structures and corresponding function derivatives.](https://thumb-eu.123doks.com/thumbv2/123doknet/13820373.442556/3.918.504.806.76.342/fig-illustration-example-tree-structures-corresponding-function-derivatives.webp)
