Quadratizations of pseudo-Boolean functions

(1)

Polynomial pseudo-Boolean optimization Reductions to quadratic pseudo-Boolean optimization Linearizing quadratic problems

Quadratizations of pseudo-Boolean functions

Elisabeth Rodriguez-Heck and Yves Crama

QuantOM, HEC Management School, University of Liège Partially supported by Belspo - IAP Project COMEX

(2)

Polynomial pseudo-Boolean optimization

Reductions to quadratic pseudo-Boolean optimization

(3)

(4)

Definitions

Definition: Pseudo-Boolean functions

A pseudo-Boolean function is a mapping f : {0, 1}n→ R.

Multilinear representation

Every pseudo-Boolean function f can be represented uniquely by a multilinear polynomial (Hammer, Rosenberg, Rudeanu [7]).

Example:

(5)

Definitions

Every pseudo-Boolean function f can be represented uniquely by a multilinear polynomial (Hammer, Rosenberg, Rudeanu [7]).

Example:

(6)

Definitions

Every pseudo-Boolean function f can be represented uniquely by a multilinear polynomial (Hammer, Rosenberg, Rudeanu [7]). Example:

(7)

Applications: MAX-SAT

MAX-SAT problem

INPUT: a set of Boolean clauses Ck = ∨i ∈Akx¯i ∨ ∨j ∈Bkxj, for

k = 1, . . . , m, where xi ∈ {0, 1}, and ¯xi = 1 − xi.

OBJECTIVE: find an assignment of the variables, x∗ ∈ {0, 1}n _that

maximizes the number of satisfied clauses.

Pseudo-Boolean formulation min m X k=1   Y i ∈Ak xi     Y j ∈Bk ¯ xj  ,

C takes value 1 iff the termQ

x Q ¯

(8)

Applications: MAX-SAT

MAX-SAT problem

INPUT: a set of Boolean clauses Ck = ∨i ∈Akx¯i ∨ ∨j ∈Bkxj, for

k = 1, . . . , m, where xi ∈ {0, 1}, and ¯xi = 1 − xi.

OBJECTIVE: find an assignment of the variables, x∗ ∈ {0, 1}n _that

maximizes the number of satisfied clauses.

Pseudo-Boolean formulation min m X k=1   Y i ∈Ak xi     Y j ∈Bk ¯ xj  ,

Ck takes value 1 iff the termQi ∈Akxi

Q

(9)

Applications: Computer Vision

Image restoration problems modelled as energy minimization

E (l ) = X p∈P Dp(lp) + X S⊆P,|S|≥2 X p1,...,ps∈S Vp1,...,ps(lp1, . . . , lps), where lp∈ {0, 1} ∀p ∈ P.

(10)

Applications

Constraint Satisfaction Problem

Data mining, classification, learning theory... Graph theory

Operations research Production management ...

(11)

Pseudo-Boolean Optimization

Many problems formulated as optimization of a pseudo-Boolean function

Pseudo-Boolean Optimization

min

x ∈{0,1}nf (x)

Optimization is N P-hard, even if f is quadratic (MAX-2-SAT, MAX-CUT modelled by quadratic f ).

Much progress has been done for the quadratic case (exact algorithms, heuristics, polyhedral results...).

(12)

Pseudo-Boolean Optimization

min

x ∈{0,1}nf (x)

(13)

Pseudo-Boolean Optimization

min

x ∈{0,1}nf (x)

(14)

(15)

Quadratizations

Definition: Quadratization

Given a pseudo-Boolean function f (x) on {0, 1}n_{, we say that g (x, y ) is a}

quadratization of f if g (x, y ) is a quadratic polynomial depending on x and on m auxiliary variables y1, . . . , ym, such that

f (x) = min{g (x, y ) : y ∈ {0, 1}m} ∀x ∈ {0, 1}n.

Then, min{f (x) : x ∈ {0, 1}n} = min{g (x, y ) : x ∈ {0, 1}n_{, y ∈ {0, 1}}m_}.

Which quadratizations are “good”? Small number of auxiliary variables.

Good optimization properties: submodularity.

A quadratic pseudo-Boolean function is submodular iff all quadratic terms have non-positive coefficients.

(16)

Quadratizations

f (x) = min{g (x, y ) : y ∈ {0, 1}m} ∀x ∈ {0, 1}n.

Good optimization properties: submodularity.

A quadratic pseudo-Boolean function is submodular iff all quadratic terms have non-positive coefficients.

(17)

Quadratizations

f (x) = min{g (x, y ) : y ∈ {0, 1}m} ∀x ∈ {0, 1}n.

(18)

Quadratic pseudo-Boolean optimization methods

Quadratic optimization is N P-hard, but much work has been done:

Algorithms based on MAX-CUT (which reduces to a polynomial MIN-CUT problem when f is submodular).

Heuristics such as local search.

In computer vision: approaches based on Roof Duality (1984) ([8]). Polyhedral results: Isomorphism between boolean quadric polytope (associated to quadratic pseudo-Boolean optimization) and cut polytope (associated to MAX-CUT) (1990) ([4]), good separation algorithms...

(19)

Quadratic pseudo-Boolean optimization methods

Quadratic optimization is N P-hard, but much work has been done: Algorithms based on MAX-CUT (which reduces to a polynomial MIN-CUT problem when f is submodular).

(20)

Quadratic pseudo-Boolean optimization methods

In computer vision: approaches based on Roof Duality (1984) ([8]).

Polyhedral results: Isomorphism between boolean quadric polytope (associated to quadratic pseudo-Boolean optimization) and cut polytope (associated to MAX-CUT) (1990) ([4]), good separation algorithms...

(21)

Quadratic pseudo-Boolean optimization methods

In computer vision: approaches based on Roof Duality (1984) ([8]).

Polyhedral results: Isomorphism between boolean quadric polytope (associated to quadratic pseudo-Boolean optimization) and cut polytope (associated to MAX-CUT) (1990) ([4]), good separation algorithms...

(22)

Quadratic pseudo-Boolean optimization methods

(23)

Rosenberg

Rosenberg (1975) [11]: first quadratization method.

1 Take a product x_ix_j from a highest-degree monomial of f and

substitute it by a new variable yij.

2 Add a penalty term M(x_ix_j − 2x_iy_ij − 2x_jy_ij _{+ 3y}_ij_{) (M large enough)}

to the objective function to force yij = xixj at all optimal solutions. 3 Iterate until obtaining a quadratic function.

Advantages:

Can be applied to any pseudo-Boolean function f . The transformation is polynomial in the size of the input.

(24)

Rosenberg

Advantages:

(25)

Rosenberg

Advantages:

(26)

Termwise quadratizations

Multilinear expression of a pseudo-Boolean function:

f (x) = X S∈2[n] aS Y i ∈S xi

Idea: quadratize monomial by monomial, using different sets of auxiliary variables for each monomial.

(27)

Termwise quadratizations: negative monomials

Kolmogorov and Zabih [10], Freedman and Drineas [6].

a n Y i =1 xi = min y ∈{0,1}ay n X i =1 xi − (n − 1) ! , a < 0.

(28)

Termwise quadratizations: negative monomials

Kolmogorov and Zabih [10], Freedman and Drineas [6].

a n Y i =1 xi = min y ∈{0,1}ay n X i =1 xi − (n − 1) ! , a < 0.

(29)

Termwise quadratizations: positive monomials

Ishikawa [9] a d Y i =1 xi = a min y1,...y_nd∈{0,1} nd X i =1 yi(ci ,d(−S1+ 2i ) − 1) + aS2,

S1, S2: elementary linear and quadratic symmetric polynomials in d

variables, n_d= bd−1₂ c and c_{i ,d} = (

1, if d is odd and i = nd,

2, otherwise.

Number of variables: best known bound for positive monomials. Submodularity: d₂ positive quadratic terms, but very good computational results.

(30)

Termwise quadratizations: positive monomials

Ishikawa [9] a d Y i =1 xi = a min y1,...y_nd∈{0,1} nd X i =1 yi(ci ,d(−S1+ 2i ) − 1) + aS2,

S1, S2: elementary linear and quadratic symmetric polynomials in d

variables, n_d= bd−1₂ c and c_{i ,d} = (

1, if d is odd and i = nd,

2, otherwise.

Number of variables: best known bound for positive monomials.

Submodularity: d₂ positive quadratic terms, but very good computational results.

(31)

Number of variables

Using termwise quadratizations:

One variable per negative monomial and bd −1₂ c per positive monomial (d : degree of the monomial).

Best known upper bounds: O(nd) variables for a polynomial of fixed degree d , O(n2n_{) for an arbitrary function.}

Can we do better?

Tight upper and lower bounds independent of the quadratization procedure by Anthony, Boros, Crama and Gruber [1]

Θ(2n2) for a general pseudo-Boolean function.

Θ(nd2) for a fixed polynomial of degree d .

Note: bounds are polynomial in the size of the input:

f (x) =P

(32)

Number of variables

Can we do better?

Θ(nd2) for a fixed polynomial of degree d .

(33)

Number of variables

Can we do better?

(34)

Summary: known quadratization techniques

What has been done until now?

Specific quadratization procedures, experimental evaluations... Several quadratizations known for monomials.

Case of negative monomials well-solved (one auxiliary variable, submodular).

Improvements can perhaps be done for positive monomials.

Non-termwise quadratization techniques: reduce degree of several terms at once (Fix, Gruber, Boros, Zabih [5]).

Objective: Global perspective

(35)

Summary: known quadratization techniques

(36)

Summary: known quadratization techniques

(37)

Summary: known quadratization techniques

(38)

Summary: known quadratization techniques

(39)

Summary: known quadratization techniques

(40)

(41)

Standard linearization

Original polynomial problem

min x ∈{0,1}nf (x) = X S∈S aS Y i ∈S xi

S: set of non-constant monomials.

1. Substitute monomials min zS X S∈S aSzS s.t. zS = Y i ∈S zi, ∀S ∈ S 2. Linearize constraints min zS X S∈S aSzS (A) s.t. zS ≤ zi, ∀i ∈ S , ∀S ∈ S zS ≥ X i ∈S zi− |S| + 1, ∀S ∈ S zS ∈ {0, 1}, ∀S ∈ S

(42)

Standard linearization

1. Substitute monomials min zS X S∈S aSzS s.t. zS = Y i ∈S zi, ∀S ∈ S zS ∈ {0, 1}, ∀S ∈ S 2. Linearize constraints min zS X S∈S aSzS (A) s.t. zS ≤ zi, ∀i ∈ S , ∀S ∈ S zS ≥ X i ∈S zi− |S| + 1, ∀S ∈ S ∈ {0, 1}, ∀S ∈ S

(43)

Standard linearization

1. Substitute monomials min zS X S∈S aSzS s.t. zS = Y i ∈S zi, ∀S ∈ S 2. Linearize constraints min zS X S∈S aSzS (A) s.t. zS ≤ zi, ∀i ∈ S , ∀S ∈ S zS ≥ X zi− |S| + 1, ∀S ∈ S

(44)

Linearization of a quadratization

min x ∈{0,1}nf (x) = X S∈S a_SY i ∈S xi

S: set of non-constant monomials

1. Define quadratization for f

min x ∈{0,1}n+m X Q∈Q bQ Y i ∈Q xi

where Q is the set of non-constant monomials in the original {x1, . . . , xn}

and the auxiliary {xn+1, . . . , xn+m} variables and all Q ∈ Q have degree at

(45)

Linearization of a quadratization

min x ∈{0,1}nf (x) = X S∈S a_SY i ∈S xi

1. Define quadratization for f

min x ∈{0,1}n+m X Q∈Q bQ Y i ∈Q xi

where Q is the set of non-constant monomials in the original {x1, . . . , xn}

(46)

Linearization of a quadratization

2. Substitute monomials min wQ X Q∈Q bQwQ s.t. wQ = Y i ∈Q wi, ∀Q ∈ Q wQ ∈ {0, 1}, ∀Q ∈ Q 3. Linearize constraints min wQ X Q∈Q bQwQ (B) s.t. wQ≤ wi, ∀i ∈ Q, ∀Q ∈ Q wQ≥ X i ∈Q wi− |Q| + 1, ∀Q ∈ Q wQ∈ {0, 1}, ∀Q ∈ Q

(47)

Linearization of a quadratization

2. Substitute monomials min wQ X Q∈Q bQwQ s.t. wQ = Y i ∈Q wi, ∀Q ∈ Q wQ ∈ {0, 1}, ∀Q ∈ Q 3. Linearize constraints min wQ X Q∈Q bQwQ (B) s.t. wQ≤ wi, ∀i ∈ Q, ∀Q ∈ Q wQ≥ X i ∈Q wi− |Q| + 1, ∀Q ∈ Q wQ∈ {0, 1}, ∀Q ∈ Q

(48)

Comparing relaxations of linearizations

Relaxation of standard linearization (A) min zS X S∈S aSzS s.t. zS ≤ zi, ∀i ∈ S , ∀S ∈ S zS ≥ X i ∈S zi − |S| + 1, ∀S ∈ S 0 ≤ zS ≤ 1, ∀S ∈ S Relaxation of linearized quadratization (B) min wQ X Q∈Q bQwQ s.t. wQ ≤ wi, ∀i ∈ Q, ∀Q ∈ Q wQ ≥ X i ∈Q wi − |Q| + 1, ∀Q ∈ Q 0 ≤ wQ ≤ 1, ∀Q ∈ Q

(49)

Comparing relaxations of linearizations

Questions:

Which relaxation is tighter?

Relaxation (B) also depends on the chosen quadratization method, which quadratization gives better relaxations?

What happens if we intersect the constraints of the relaxed linearizations of all quadratizations of f ?

(50)

Comparing relaxations of linearizations

Questions:

(51)

Comparing relaxations of linearizations

Questions:

(52)

Comparing relaxations of linearizations

Questions:

(53)

Comparing polytopes at substitution step

Standard linearization (A)

min zS X S∈S aSzS s.t. zS = Y i ∈S zi, ∀S ∈ S zS ∈ {0, 1}, ∀S ∈ S Quadratization (B) min wQ X Q∈Q bQwQ s.t. wQ= Y i ∈Q wi, ∀Q ∈ Q wQ∈ {0, 1}, ∀Q ∈ Q

(54)

Comparing polytopes at substitution step

Questions:

Do we have a better knowledge of one of the convex hull of feasible solutions of one of these problems? (i.e., polyhedral description, good separation algorithms...).

(55)

Buchheim and Rinaldi’s approach

Polynomial Optimization Problem

LP-relaxation (standard lin.), polytope P

Fractional solution x∗

(56)

Buchheim and Rinaldi’s approach

Polynomial Optimization Problem

LP-relaxation (standard lin.), polytope P

Fractional solution x∗

(57)

Buchheim and Rinaldi’s approach: Polytope P

∗

?

Presented in [2], [3].

Assumption: every S ∈ S can be written as the union of two other monomials Sl, Sr ∈ S.

For a given S , there might be several pairs of subsets S_l, Sr such that

Sl ∪ Sr = S .

(58)

Buchheim and Rinaldi’s approach: Polytope P

∗

?

Assumption: every S ∈ S can be written as the union of two other

monomials Sl, Sr ∈ S.

For a given S , there might be several pairs of subsets Sl, Sr such that

Sl ∪ Sr = S .

(59)

Buchheim and Rinaldi’s approach: Polytope P

∗

?

(60)

Buchheim and Rinaldi’s approach: Polytope P

∗

?

Sl ∪ Sr = S .

(61)

Buchheim and Rinaldi’s approach: Polytope P

∗

?

Buchheim and Rinaldi’s formulation over a quadric polytope

Consider the set S∗= {{S , T } | S , T ∈ S and S ∪ T ∈ S}.

min y{S} X S∈S aSy{S} (C) s.t. y_{{S,T }}= y{S}y{T }, ∀{S , T } ∈ S∗ y{S,T }∈ {0, 1}, ∀{S, T } ∈ S∗

P∗: convex hull of feasible solutions of problem (C) Observation: P∗ is a boolean quadric polytope.

(62)

Buchheim and Rinaldi’s approach: polytope P

∗

?

Polytope P∗

Is isomorph to a Cut-Polytope (efficient separation algorithms are known).

Lives in a higher-dimensional space (introducing auxiliary variables).

P is isomorph to a face of P∗ (imposing some additional constraints to P∗): cutting on P∗ is equivalent to cutting on P.

Objective:

We are interested in understanding the quadratic problem that is used to define P∗:

(63)

Buchheim and Rinaldi’s approach: polytope P

∗

?

Polytope P∗

Lives in a higher-dimensional space (introducing auxiliary variables). P is isomorph to a face of P∗ (imposing some additional constraints

to P∗): cutting on P∗ is equivalent to cutting on P.

Objective:

(64)

Buchheim and Rinaldi’s approach: polytope P

∗

?

Polytope P∗

Lives in a higher-dimensional space (introducing auxiliary variables). P is isomorph to a face of P∗ (imposing some additional constraints to P∗): cutting on P∗ is equivalent to cutting on P.

Objective:

(65)

Buchheim and Rinaldi’s approach: polytope P

∗

?

Polytope P∗

Objective:

(66)

Buchheim and Rinaldi’s approach: polytope P

∗

?

Polytope P∗

Objective:

(67)

Buchheim and Rinaldi’s formulation and Rosenberg’s

quadratization

Theorem

Buchheim and Rinaldi’s formulation over a quadric polytope can be obtained (up to elimination of redundant constraints) by linearizing a variant of Rosenberg’s quadratization where:

the order of substituting variables is induced by the decomposition Sl, Sr of each monomial S , and

when substituting a product by a variable, we do not impose yij = xixj

with a penalty, but with a constraint.

(68)

Linearization of Rosenberg’s quadratization (B)

identical

Buchheim and Rinaldi’s formulation over a quadric polytope (C) Standard linearization (A) equivalent equivalent (P ∼= face of P∗)

(69)

Conclusions

Understand quadratization methods from a global perspective:

Properties and structure, which quadratizations are “better”?

Describe all quadratizations of f .

Linearizing quadratizations:

How does the tightness of a relaxation depend on the quadratization? How do relaxations of linearized quadratizations relate to other methods (e.g. standard linearization)?

Can we use the knowledge about the polytopes associated to linearizations? (e.g. cut polytope separation techniques...)?

(70)

Conclusions

Properties and structure, which quadratizations are “better”? Describe all quadratizations of f .

(71)

Conclusions

(72)

Conclusions

How does the tightness of a relaxation depend on the quadratization?

How do relaxations of linearized quadratizations relate to other methods (e.g. standard linearization)?

(73)

Conclusions

(74)

Conclusions

(75)

Conclusions

(76)

Some references I

M. Anthony, E. Boros, Y. Crama, and A. Gruber. Quadratic reformulations of nonlinear binary optimization problems. Working paper, 2014.

C. Buchheim and G. Rinaldi. Efficient reduction of polynomial zero-one optimization to the quadratic case. SIAM Journal on Optimization, 18(4):1398–1413, 2007.

C. Buchheim and G. Rinaldi. Terse integer linear programs for boolean optimization. Journal on Satisfiability, Boolean Modeling and Computation, 6:121–139, 2009. C. De Simone. The cut polytope and the boolean quadric polytope. Discrete Mathematics, 79(1):71–75, 1990.

A. Fix, A. Gruber, E. Boros, and R. Zabih. A hypergraph-based reduction for higher-order markov random fields. Working paper, submitted to PAMI?, 2014.

(77)

Some references II

D. Freedman and P. Drineas. Energy minimization via graph cuts: settling what is possible. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 2, pages 939–946, June 2005.

P. L. Hammer, I. Rosenberg, and S. Rudeanu. On the determination of the minima of pseudo-boolean functions. Studii si Cercetari Matematice, 14:359–364, 1963. in Romanian.

P. L. Hammer, P. Hansen, and B. Simeone. Roof duality, complementation and persistency in quadratic 0-1 optimization. Mathematical Programming, 28(2):121–155, 1984.

H. Ishikawa. Transformation of general binary mrf minimization to the first-order case. Pattern Analysis and Machine Intelligence, IEEE Transactions on,

(78)

Some references III

V. Kolmogorov and R. Zabih. What energy functions can be minimized via graph cuts? Pattern Analysis and Machine Intelligence, IEEE Transactions on,

26(2):147–159, Feb 2004.

I. G. Rosenberg. Reduction of bivalent maximization to the quadratic case. Cahiers du Centre d’Etudes de Recherche Opérationnelle, 17:71–74, 1975.

S. Živn`y, D. A. Cohen, and P. G. Jeavons. The expressive power of binary submodular functions. Discrete Applied Mathematics, 157(15):3347–3358, 2009.