Largesparselinearsystemsareinvolvedinmanymathematicalapplications Motivations

On the use of polynomial matrix approximant in the block Wiedemann algorithm

Pascal Giorgi

ECOLE NORMALE SUPERIEURE DE LYON

Laboratoire LIP, ENS Lyon, France

Symbolic Computation Group, School of Computer Science, University of Waterloo, Canada

in collaboration with C-P. Jeannerod and G. Villard ENS-Lyon (France)

June 6, CMS/CSHPM Summer 2005 Meeting

Mathematics of Computer Algebra and Analysis

Motivations

Large sparse linear systems are involved in many mathematical applications

over a field :

I

integers factorization

,

I

discrete logarithm

, over the integers :

I

number theory

,

I

group theory

,

I

integer programming

Solving sparse system over finite fields

Matrices arising in pratice have dimensions around few millions with few millions of non zero entries

The use of classic Gaussian elimination is proscribed due to fill-in

= ⇒ algorithms must preserve the sparsity of the matrix

Iteratives methods revealed successful over a finite field :

I

Krylov/Wiedemann method

conjugate gradient

,

I

Lanczos method

,

I

block Lanczos method

block Krylov/Wiedemann method

Solving sparse system over finite fields

Matrices arising in pratice have dimensions around few millions with few millions of non zero entries

The use of classic Gaussian elimination is proscribed due to fill-in

= ⇒ algorithms must preserve the sparsity of the matrix

Iteratives methods revealed successful over a finite field :

I

Krylov/Wiedemann method

conjugate gradient

,

I

Lanczos method

,

I

block Lanczos method

block Krylov/Wiedemann method

Solving sparse system over finite fields

Matrices arising in pratice have dimensions around few millions with few millions of non zero entries

The use of classic Gaussian elimination is proscribed due to fill-in

= ⇒ algorithms must preserve the sparsity of the matrix

Iteratives methods revealed successful over a finite field :

I

Krylov/Wiedemann method

I

conjugate gradient

,

I

Lanczos method

,

I

block Lanczos method

I

block Krylov/Wiedemann method

Scheme of Krylov/Wiedemann method

Let A ∈ IF

of full rank and b ∈ IF

. Then x = A

b can be

expressed as a linear combination of the Krylov subspace {b, Ab, ..., A

b}

Let Π

(λ) = c

+ c

λ + ... + λ

∈ IF[λ] be the minimal polynomial of the sequence {A

b}

A × −1

c

(c

b + c

Ab + ... + A

b)

| {z }

= b

x

Choose u ∈ IF

uniformaly and randomly and compute the minimal polynomial Π

of the scalar sequence {u

A

b}

.

with probability greater than 1 −

we have Π

= Π

.

Scheme of Krylov/Wiedemann method

Let A ∈ IF

of full rank and b ∈ IF

. Then x = A

b can be

expressed as a linear combination of the Krylov subspace {b, Ab, ..., A

b}

Let Π

(λ) = c

+ c

λ + ... + λ

∈ IF[λ] be the minimal polynomial of the sequence {A

b}

A × −1

c

(c

b + c

Ab + ... + A

b)

| {z }

= b

x

Choose u ∈ IF

uniformaly and randomly and compute the minimal polynomial Π

of the scalar sequence {u

A

b}

.

with probability greater than 1 −

we have Π

= Π

.

Scheme of Krylov/Wiedemann method

Let A ∈ IF

of full rank and b ∈ IF

. Then x = A

b can be

expressed as a linear combination of the Krylov subspace {b, Ab, ..., A

b}

Let Π

(λ) = c

+ c

λ + ... + λ

∈ IF[λ] be the minimal polynomial of the sequence {A

b}

A × −1

c

(c

b + c

Ab + ... + A

b)

| {z }

= b

x

Choose u ∈ IF

uniformaly and randomly and compute the minimal polynomial Π

of the scalar sequence {u

A

b}

.

with probability greater than 1 −

we have Π

= Π

.

Scheme of Krylov/Wiedemann method

Let A ∈ IF

of full rank and b ∈ IF

. Then x = A

b can be

expressed as a linear combination of the Krylov subspace {b, Ab, ..., A

b}

Let Π

(λ) = c

+ c

λ + ... + λ

∈ IF[λ] be the minimal polynomial of the sequence {A

b}

A × −1

c

(c

b + c

Ab + ... + A

b)

| {z }

= b

x

Choose u ∈ IF

uniformaly and randomly and compute the minimal polynomial Π

of the scalar sequence {u

A

b}

.

with probability greater than 1 −

we have Π

= Π

.

Scheme of Krylov/Wiedemann method

Let A ∈ IF

of full rank and b ∈ IF

. Then x = A

b can be

expressed as a linear combination of the Krylov subspace {b, Ab, ..., A

b}

Let Π

(λ) = c

+ c

λ + ... + λ

∈ IF[λ] be the minimal polynomial of the sequence {A

b}

A × −1

c

(c

b + c

Ab + ... + A

b)

| {z }

= b

x

Choose u ∈ IF

uniformaly and randomly and compute the minimal polynomial Π

of the scalar sequence {u

A

b}

.

with probability greater than 1 −

we have Π

= Π

.

Wiedemann algorithm

Three steps :

1. compute 2N + elements of the sequence {u

A

b}

. 2. compute the minimal polynomial of the scalar sequence.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : 2N matrix-vector products + 2N dot products

= ⇒ cost 2Nγ + O(N

) field operations

step 2 : use of Berlekamp/Massey algorithm

= ⇒ cost O(N

) field operations

step 3 : d − 1 matrix-vector products + d − 1 vectors operations

= ⇒ cost at most Nγ + O(N

) field operations

total cost of O (Nγ + N

) fied operations with O(N) additional space

Wiedemann algorithm

Three steps :

1. compute 2N + elements of the sequence {u

A

b}

. 2. compute the minimal polynomial of the scalar sequence.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : 2N matrix-vector products + 2N dot products

= ⇒ cost 2Nγ + O(N

) field operations

step 2 : use of Berlekamp/Massey algorithm

= ⇒ cost O(N

) field operations

step 3 : d − 1 matrix-vector products + d − 1 vectors operations

= ⇒ cost at most Nγ + O(N

) field operations

total cost of O (Nγ + N

) fied operations with O(N) additional space

Wiedemann algorithm

Three steps :

1. compute 2N + elements of the sequence {u

A

b}

. 2. compute the minimal polynomial of the scalar sequence.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : 2N matrix-vector products + 2N dot products

= ⇒ cost 2Nγ + O(N

) field operations

step 2 : use of Berlekamp/Massey algorithm

= ⇒ cost O(N

) field operations

step 3 : d − 1 matrix-vector products + d − 1 vectors operations

= ⇒ cost at most Nγ + O(N

) field operations

total cost of O (Nγ + N

) fied operations with O(N) additional space

Wiedemann algorithm

Three steps :

1. compute 2N + elements of the sequence {u

A

b}

. 2. compute the minimal polynomial of the scalar sequence.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : 2N matrix-vector products + 2N dot products

= ⇒ cost 2Nγ + O(N

) field operations

step 2 : use of Berlekamp/Massey algorithm

= ⇒ cost O(N

) field operations

step 3 : d − 1 matrix-vector products + d − 1 vectors operations

= ⇒ cost at most Nγ + O(N

) field operations

total cost of O (Nγ + N

) fied operations with O(N) additional space

Wiedemann algorithm

Three steps :

1. compute 2N + elements of the sequence {u

A

b}

. 2. compute the minimal polynomial of the scalar sequence.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : 2N matrix-vector products + 2N dot products

= ⇒ cost 2Nγ + O(N

) field operations

step 2 : use of Berlekamp/Massey algorithm

= ⇒ cost O(N

) field operations

step 3 : d − 1 matrix-vector products + d − 1 vectors operations

= ⇒ cost at most Nγ + O(N

) field operations

total cost of O (Nγ + N

) fied operations with O(N) additional space

Block Wiedemann method

Replace the projection vectors by blocks of vectors.

Let U ∈ IF

and V = [b V ¯ ] ∈ IF

. We now consider the matrix sequence {UA

V }

.

Main interest :

I

parallel coarse and fine grain implementation (on columns of V ),

I

better probability of success

,

I

(1 + )N matrix-vector products (sequential)

.

Difficulty :

minimal generating matrix polynomial of a matrix sequence.

Minimal generating matrix polynomial

Let {S

}

be a m × m matrix sequence.

Let P ∈ IF

[λ] be minimal with degree k s.t.

∀j > 0 :

k

X

i=0

S

P

= 0

the cost to compute P is :

I

O (m

k

) field operations

,

I

O˜(m

k log k ) field operations

,

I

O˜(m

k log k) field operations

. where ω is the exponent in the complexity of matrix multiplication

Latter complexity is based on σ-bases computation with :

• divide and conquer approach (idea from

)

• matrix product-based Gaussian elimination

Minimal generating matrix polynomial

Let {S

}

be a m × m matrix sequence.

Let P ∈ IF

[λ] be minimal with degree k s.t.

∀j > 0 :

k

X

i=0

S

P

= 0

the cost to compute P is :

I

O (m

k

) field operations

,

I

O˜(m

k log k ) field operations

,

I

O˜(m

k log k) field operations

. where ω is the exponent in the complexity of matrix multiplication

Latter complexity is based on σ-bases computation with :

• divide and conquer approach (idea from

)

• matrix product-based Gaussian elimination

Minimal approximant basis : σ-basis

Problem :

Given a matrix power series G ∈ IF

[[λ]] and an approximation order d ; find the minimal nonsingular polynomial matrix M ∈ IF

[λ] s.t.

MG = λ

R ∈ IF

[[λ]]

M is called an order d σ-bases of G.

minimality :

Let f (λ) = G(λ

)[1, λ, λ

, ..., λ

]

∈ IF[[λ]]

every v ∈ IF

[λ] such that

v(λ

)f (λ) = λ

w(λ) ∈ IF[[λ]] with r ≥ nd has a unique decomposition

v =

m

X

i=1

c

M

with deg c

+ deg M

≤ deg v

Minimal approximant basis : σ-basis

Problem :

Given a matrix power series G ∈ IF

[[λ]] and an approximation order d ; find the minimal nonsingular polynomial matrix M ∈ IF

[λ] s.t.

MG = λ

R ∈ IF

[[λ]]

M is called an order d σ-bases of G.

minimality :

Let f (λ) = G(λ

)[1, λ, λ

, ..., λ

]

∈ IF[[λ]]

every v ∈ IF

[λ] such that

v(λ

)f (λ) = λ

w(λ) ∈ IF[[λ]] with r ≥ nd has a unique decomposition

v =

m

X

i=1

c

M

with deg c

+ deg M

≤ deg v

Sketch of the reduction

divide and conquer :

[Beckermann, Labahn 1994 : theorem 6.1]

Given M

and M

two order d/2 σ-basis of respectively G and λ

M

G . The polynomial matrix M = M

M

is an order d σ-bases of G.

base case (order 1 σ-basis) :

I

compute ∆ = G mod λ,

I

compute the LSP-factorization of π∆, with π a permutation,

I

return M = DL

π where D is a diagonal matrix (with 1’s and λ’s)

cost :

• C (m, n, d) = 2C (m, n, d/2) + 2MM(m, d /2)

• C (m, n, 1) = O(MM (m))

= ⇒ reduction to polynomial matrix multiplication

Sketch of the reduction

divide and conquer :

[Beckermann, Labahn 1994 : theorem 6.1]

Given M

and M

two order d/2 σ-basis of respectively G and λ

M

G . The polynomial matrix M = M

M

is an order d σ-bases of G.

base case (order 1 σ-basis) :

I

compute ∆ = G mod λ,

I

compute the LSP-factorization of π∆, with π a permutation,

I

return M = DL

π where D is a diagonal matrix (with 1’s and λ’s)

cost :

• C (m, n, d) = 2C (m, n, d/2) + 2MM(m, d /2)

• C (m, n, 1) = O(MM (m))

= ⇒ reduction to polynomial matrix multiplication

σ-bases and minimal generating matrix polynomial

Considering the matrix power series G(λ) =

∞

X

i=0

UA

V λ

∈ IF

[[λ]]

Let P, T ∈ IF

and Q, S ∈ IF

[λ] defining the right 2N/m σ-bases G −I

P S Q T

= λ

R ∈ IF

[[λ]]

such that deg P > deg Q and P is full rank.

The reversal matrix polynomial of P according to its column degrees define a right minimal generating matrix polynomial for the matrix sequence {UA

V }

Proof :

∀k ∈ {deg P, ..., 2N/m}

degP

X

i=0

G

P

= 0

σ-bases and minimal generating matrix polynomial

Considering the matrix power series G(λ) =

∞

X

i=0

UA

V λ

∈ IF

[[λ]]

Let P, T ∈ IF

and Q, S ∈ IF

[λ] defining the right 2N/m σ-bases G −I

P S Q T

= λ

R ∈ IF

[[λ]]

such that deg P > deg Q and P is full rank.

The reversal matrix polynomial of P according to its column degrees define a right minimal generating matrix polynomial for the matrix sequence {UA

V }

Proof :

∀k ∈ {deg P, ..., 2N/m}

degP

X

i=0

G

P

= 0

Block Wiedemann algorithm

Three steps :

1. compute 2N/m + elements of the sequence {U

A

V }

. 2. compute the right minimal matrix polynomial through σ-bases.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A. step 1 : with m processors

= ⇒ cost 2Nγ/m + O(N

) field operations step 2 : with 1 processors

= ⇒ cost O˜(m

N) field operations step 3 : with m processors

= ⇒ cost at most Nγ/m + O(N

) field operations

total of O(Nγ/m) + O(N

) + O˜(n

N) fied op. with m processors

Block Wiedemann algorithm

Three steps :

1. compute 2N/m + elements of the sequence {U

A

V }

. 2. compute the right minimal matrix polynomial through σ-bases.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : with m processors

= ⇒ cost 2Nγ/m + O(N

) field operations step 2 : with 1 processors

= ⇒ cost O˜(m

N) field operations step 3 : with m processors

= ⇒ cost at most Nγ/m + O(N

) field operations

total of O(Nγ/m) + O(N

) + O˜(n

N) fied op. with m processors

Block Wiedemann algorithm

Three steps :

1. compute 2N/m + elements of the sequence {U

A

V }

. 2. compute the right minimal matrix polynomial through σ-bases.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : with m processors

= ⇒ cost 2Nγ/m + O(N

) field operations

step 2 : with 1 processors

= ⇒ cost O˜(m

N) field operations step 3 : with m processors

= ⇒ cost at most Nγ/m + O(N

) field operations

total of O(Nγ/m) + O(N

) + O˜(n

N) fied op. with m processors

Block Wiedemann algorithm

Three steps :

1. compute 2N/m + elements of the sequence {U

A

V }

. 2. compute the right minimal matrix polynomial through σ-bases.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : with m processors

= ⇒ cost 2Nγ/m + O(N

) field operations step 2 : with 1 processors

= ⇒ cost O˜(m

N) field operations

step 3 : with m processors

= ⇒ cost at most Nγ/m + O(N

) field operations

total of O(Nγ/m) + O(N

) + O˜(n

N) fied op. with m processors

Block Wiedemann algorithm

Three steps :

1. compute 2N/m + elements of the sequence {U

A

V }

. 2. compute the right minimal matrix polynomial through σ-bases.

3. compute the linear combination of {b, Ab, ..., A

b}.

Let γ be the cost of applying a vector to A.

step 1 : with m processors

= ⇒ cost 2Nγ/m + O(N

) field operations step 2 : with 1 processors

= ⇒ cost O˜(m

N) field operations step 3 : with m processors

= ⇒ cost at most Nγ/m + O(N

) field operations

total of O(Nγ/m) + O(N

) + O˜(n

N) fied op. with m processors

Implementation within LinBox library

I

LinBox project (Canada-France-USA) : www.linal.org

I

Generic implementation with respect to : finite field, blackbox.

I

σ-basis implementation :

• hybrid dense linear algebra over finite field

• polynomial matrix multiplication :

Karatsuba algorithm + BLAS-based matrix multiplication

• Karatsuba polynomial matrix middle product

Performances : minimal generating matrix polynomial

• over GF(17), matrix sparsity is 99%, block dimension is 20

2 4 8 16 32 64 128 256 512 1024

5000 10000 15000 20000 25000 30000

Time in second

Matrix size

Minimal generating matrix polynomial vs minimal polynomial PM−Basis (block)

M−Basis (block) Berlekamp/Massey (scalar)

N = 30 000 : practical block/scalar ≈ O(1)

scalar sequence computation : ≈ 12.4h

Conclusions

• O˜(n

d) algorithm for Pade approximation problem.

→ advantage for solving sparse linear system (block Wiedemann)

→ O˜(n

d) algorithm for column reduction

• still unclear : characteristic polynomial, Hermite and Frobenius form.

Into practice :

• fast polynomial matrix multiplication

[Cantor, Kaltofen 1991 ; Bostan, Schost 2004]

• compare LinBox with Magma (implementation of Thom´ e algorithm)

• Specialization for GF(2) and parallel implementation (SMP, grid)