From graph matching problem to Assignment problem : Application to Content-Based Image Retrieval.

(1)

From graph matching problem to Assignment problem :

Application to Content-Based Image Retrieval.

Romain Raveaux

LI lab (EA 6300) – Universit´e de Tours

Rouen, 1rst December, 2013

(2)

Graphs are everywhere...

Graphs in Reality

Graphs model objects and their relationships.

Also referred to as networks.

All common data

structures can be modeled as graphs.

(3)

Graphs are everywhere...

Graphs in Reality

All common data

(4)

Graphs are everywhere...

Graphs in Reality

All common data

(5)

Definition of a graph

Definition

LetL_V andL_E denote the set of node and edge labels, respectively.

A labeled graph G is a 4-tupleG = (V,E, µ, ξ) , where V is the set of nodes,

E ⊆V ×V is the set of edges

µ:V →L_V is a function assigning labels to the nodes, and ξ :E →L_E is a function assigning labels to the edges.

Let G1= (V1,E1, µ1, ξ1) be the source graph And G₂ = (V₂,E₂, µ₂, ξ₂) the target graph

WithV₁ = (u₁, ...,u_n) andV₂ = (v₁, ...,v_m) respectively

(6)

Definition and notation of a graph

Labels for both nodes and edges can be given by : Labelling functions

a set of integers L = {1, 2, 3, . . .}

a vector space L =Rⁿ

a finite set of symbolic labels L ={x,y, z, . . .}

...

Definition 1 allows us to handle arbitrarily structured graphs with unconstrained labelling functions.

(7)

Graph comparison

How similar are two graphs?

Graph similarity is the central problem for all learning tasks such as clustering and classification on graphs.

At least two ways for comparing graphs Graph Embedding

Graph Matching

(8)

Graph comparison

Graph Matching

(9)

Graph comparison

Graph Matching

(10)

Graph isomorphism

Find a mapping f :V1 →V2

i.e. x,y ∈V1 ⇒ (x,y)∈E1

f is an isomorphism iff(f(x),f(y)) is an edge of G₂.

No polynomial-time algorithm is known for graph isomorphism Neither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G₃ ofG₂ such thatG₁ andG₃ are isomorphic.

Subgraph isomorphism is NP-complete

(11)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(12)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(13)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(14)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(15)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(16)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(17)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(18)

Graph isomorphism

i.e. x,y ∈V1 ⇒ (x,y)∈E1

(19)

Graph isomorphism families

1 Exact Matching : Exact mapping between vertex and/or edge labels.

2 Inexact Matching : No exact mapping between vertex and/or edge labels can be found, but when the mapping can be associated to a cost.

In pattern recognition

Vertices and edges are labeled with measures/features which may be affected by noise.

Direct and exact comparisons are useless.

The matching problem turns from a decision one to an optimization one.

(20)

Inexact matching

Three sub-problems :

1 Substitution-Tolerant Subgraph Isomorphism

2 Inexact Subgraph Isomorphism

3 Inexact Graph Isomorphism

(21)

Substitution-Tolerant Subgraph Isomorphism

1 This formulation allows differences between labels.

2 The mapping does not affect the topology.

3 The mapping should be edge-preserving in one direction.

V’

V E E’

_f(v)

f^-‐1(v’)

f

0.5

^0.3

0.15

_0.13

Figure: Substitution-tolerant subgraph isomorphism problem

(22)

Inexact Subgraph Isomorphism

2 Topology can be different. The mapping is not edge-preserving.

V’

V E E’

f(e)

f

0.5

0.3

0.13

Δ

_e’

0.16

Unmatched Matched

Figure: Inexact Subgraph Isomorphism problem

(23)

Inexact Graph Isomorphism

1 Bijective function

3 Topology can be different.

V’

V E E’

f

0.5

0.3

0.13

Δ

_e’

0.16

Δ

_e

Δ

_v

Figure: Inexact graph isomorphism problem

(24)

Graph Matching problems

Figure: Graph isomorphism constraints

(25)

Comparison between Classical Graph-Matching Methods

Name Graph

matching problem

Models For- mulation

Optimality Complexity class

Deterministic Optimization method

References

Node Assignment

IGI Approximate Sub-optimal Polynomial Deterministic Hungarian algorithm

[Riesen 2009], [Raveaux 2010], [Jouili 2009] , [Shokoufandeh 2006]

Neural Net- work

ISGI Approximate Sub-Optimal Polynomial Non-

Deterministic

Gradient [Kuner 1988], [Suganthan 2000]

String based methods

ISGI Approximate Sub-Optimal Polynomial Deterministic Tree Search [J. Llados 2001], [Anjan Dutta 2012]

Approximated GED

IGI Exact Sub-Optimal NP-Complete Deterministic Heuristic Tree search

[Neuhaus M 2006]

Genetic algorithm

ISGI Exact Sub-Optimal Polynomial Non-

Deterministic

Genetic Algo- rithm

[A.D.J. Cross 1997]

,[Ford 1991], [Jiang 1998]

Probabilistic Relaxation

ISGI Exact Sub-Optimal Polynomial Deterministic Probabilistic Relaxation

[W. Christmas 1995], [Finch 1998], [Gold 1996]

Similarity search

MIGI Exact Sub-Optimal Polynomial Non-

Deterministic

Greedy search, Tabu Search

[Solnon 2005]

GED IGI Exact Optimal NP-Complete Deterministic Tree search [Bunke 1998],

[Bunke 1999]

GED ILP IGI Exact Optimal NP-Complete Deterministic Mathematical

Solver

[Justice D 2006]

ILP STSI Exact Optimal NP-Complete Deterministic Mathematical

solver

[LeBodic 2012]

Indexing method

ISGI Exact Optimal NP-Complete Deterministic Indexed search [Messmer 1998]

Spectral Theory

STSI Exact Optimal Polynomial Deterministic Eigen Decom-

position

[Ferrer 2005] , [Umeyama 1988].

Similarity MIGI Exact Optimal NP-Complete Deterministic Branch And [Champin ]

(26)

Some highlights

Exact graph matching is useless in many computer vision applications

Concerning graph matching under noise and distortion The matching incorporates an error model to identify the distortions which make one graph a distorted version of the other

Problems for real world applications Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes This is an enormous problem for large datasets of graphs Wanted: Polynomial-time similarity measure for graphs

(27)

Some highlights

(28)

Some highlights

(29)

Some highlights

(30)

Some highlights

(31)

Some highlights

(32)

Some highlights

(33)

Some highlights

(34)

Some highlights

(35)

Assignment problem

Jacobi (1890) exposes a polynomial time algorithm to solve the following problem:

Let a_i,j be a square matrixnxn, find a permutationσ such that P

a_i,σ(i) be maximal.

Such an algorithm was only rediscovered in 1955 par Kuhn, who called it “Hungarian method”.

It is a well-known problem in mathematical economy, under the name of “assignment problem”.

(36)

Assignment problem

Say you have three workers: Jim, Steve and Alan. You need to have one of them clean the bathroom, another sweep the floors and the third wash the windows. What’s the best (minimum-cost) way to assign the jobs? First we need a matrix of the costs of the workers doing the jobs.

Clean bathroom Sweep floors Wash windows

Jim 1e 2e 3e

Steve 3e 3e 3e

Alan 3e 3e 2e

Table: Cost matrix

Then the Hungarian algorithm, when applied to the above table would give us the minimum cost it can be done with: Jim cleans the bathroom, Steve sweeps the floors and Alan washes the windows.

(37)

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mapping local substructures

Any node(u_n) fromG₁ can be assigned to any node(v_m) ofG₂, Incurring somecostthat depends on the un-vm assignment.

It is required to map all nodes in such a way that the total cost of the assignment is minimized.

Cost matrix representation (C):

C_ij correspond to the costs of assigning the i^th node of G₁ to thej^th node of G2.

(38)

Graph comparison through assignment problem

Basic idea:

(39)

Graph comparison through assignment problem

Basic idea:

(40)

Graph comparison through assignment problem

Basic idea:

(41)

Graph comparison through assignment problem

Basic idea:

(42)

Graph comparison through assignment problem

Basic idea:

(43)

Graph comparison through assignment problem

Basic idea:

(44)

Graph comparison through assignment problem

Univalent matching Sets of equal size

Find a bijectionb :G1→G2

Given two sets of subgraphs,G1 andG2 of equal size, the problem can be expressed as a standard linear program with the objective function Eq 1 to be minimized.

X

i∈G₁

X

j∈G₂

C(i,j)xij (1)

The variablex_ij represents the assignment of a subgraph ofG₁ to a subgraph ofG₂, taking value 1 if the assignment is done and 0 otherwise.

(45)

Graph comparison through assignment problem: state of the art

Node signature Distance

[Gold 1996] Node degree+Label *

[Shokoufandeh 2006] Eigen vector L2

[Riesen 2009] (1)Node+(2)Edge Edit cost [Jouili 2009] Node degree+Weight L1 [Raveaux 2010] Sub Graph Edit Distance

*: Depends on the graph attribute type.

(46)

Graph comparison through assignment problem: an example

A subgraph (sg):

A structure gathering the edges and their

corresponding ending vertices from a root vertex.

Romain Raveaux et al. A graph matching method and a graph matching distance based on subgraph assignments.

Pattern Recognition Letters (2010)

Figure: Decomposition into

20 / 38

(47)

Graph comparison through assignment problem: an example

A subgraph (sg):

A structure gathering the edges and their

corresponding ending vertices from a root vertex.

Romain Raveaux et al. A graph matching method and a graph matching distance based on subgraph assignments.

Pattern Recognition Letters (2010)

Figure: Decomposition into

20 / 38

(48)

Graph comparison through assignment problem: an example II

A distance between graph Subgraph decomposition Optimization algorithm

21 / 38

(49)

Graph comparison through assignment problem: an example II

21 / 38

(50)

Graph comparison through assignment problem: an example II

21 / 38

(51)

Synthesis

Inexact graph matching problem Univalent matching

Polynomial algorithm : O(n³) Two main directions

Node signature based methods (vectors).

Subgraph based methods. (not a dot in a feature space.)

(52)

How Graph Matching can be used in Pattern Recognition Problems

Hypothesis :

a graph database consisting of n graphs. D=g1,g2, ...,gn a query graphq

Problem definitions :

1 Full graph search. Find all graphs gi in D s.t. gi is the same as q.

2 Subgraph search. Find all graphs gi in D containingq or contained byq.

3 Similarity search. Find all graphs gi in D s.t. gi is similar to q within a user-specified threshold based on some similarity measures.

4 Graph mining. Graph mining problem gathers similar graph or subgraph of D in order to find clusters or prototypes.

(53)

CBIR problem

Figure: Feature clustering stage

(54)

CBIR scheme

Figure: Feature clustering stage

(55)

CBIR scheme

Figure: Images to graphs

(56)

CBIR scheme

(57)

(a) (b)

Figure: Regions of Interest found by the SIFT algorithm. Processing SIFT took 407ms, 60 features were identified and processed

(58)

Figure: A segmentation result. Processing SRM took 1625ms and 26 features were identified and processed

(59)

(60)

Figure: CBIR approach categorization according to the amount of spatial information captured

(61)

Figure: CBIR taxonomy

(62)

Figure: Image Samples from the Caltech-101 Data set. The 101 object categories and the background clutter category.

(63)

Figure: Columbia University Image Library

(64)

Recognition rate

(a) Coil-100 (b) Caltech-101

Figure: Comparison between CBIR methods. Summary of results obtained with the best number of words for each method.

Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual Communication and Image Representation (2013). Elsevier.

(65)

Vocabulary size impact

(a) Coil-100 (b) Caltech-101

Figure: Comparison between the number of words in used by the methods.

Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual Communication and Image Representation (2013). Elsevier.

(66)

Time experiments

Average response time in seconds SIFT_BoW IFFS_BoW IFFS_GBR

Coil−100 28 37 1555

Table: Average response time of a query. A speed comparison of the four methods at the worst case.

(67)

Conclusion

To take the stock of :

SIFT_BoW is better on Caltech data IFFS is better on Coil-100

The Graph Based Approach gave similar recognition rate than BoW methods.

A structural approach requires a fewer number of words to reach its best performance.

Future work

To avoid visual features clustering.

To enrich the graph representation.

Combination of BoW and GbR : BoW first and then to process a re-ranking stage with the top k responses based on GbR.

(68)

R.C. Wilson A.D.J. Cross and E.R. Hancock.

Inexact Graph Matching Using Genetic Search.

Pattern Recognition, pages 953–970, 1997.

Josep Llados Anjan Dutta and Umapada Pal.

A symbol spotting approach in graphical documents by hashing serialized graphs.

Horst Bunke and Kim Shearer.

A graph distance metric based on the maximal common subgraph.

Pattern Recognition Letters, vol. 2, page 255–259, 1998.

Horst Bunke.

Error Correcting Graph Matching: On the Influence of the Underlying Cost Function.

(69)

IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 9, pages 917–922, 1999.

Pierre-antoine Champin and Christine Solnon.

Measuring the similarity of labeled graphs.

Miquel Ferrer, Francesc Serratosa and Alberto Sanfeliu.

Synthesis of Median Spectral Graph.

In IbPRIA (2), pages 139–146, 2005.

et al. Finch Wilson.

An energy function and continuous edit process for graph matching.

Neural Computat, vol. 10, 1998.

G.P. Ford and J. Zhang.

A Structural Graph Matching Approach to Image Understanding.

(70)

Intelligent Robots and Computer Vision X: Algorithms and Techniques, pages 559–569, 1991.

S. Gold and A Rangarajan.

A graduated assignment algorithm for graph matching.

Workshops SSPR and SPR, pages 377 – 388, 1996.

E. Marti J. Llados and J. J. Villanueva.

Symbol recognition by error-tolerant subgraph matching between region adjacency graphs.

Patt. Anal. Mach., pages 1137–1143, 2001.

X. Jiang and H. Bunke.

Marked subgraph isomorphism of ordered graphs.

Workshops SSPR and SPR, pages 122–131, 1998.

Salim Jouili and Salvatore Tabbone.

Graph Matching Based on Node Signatures.

In GbRPR, pages 154–163, 2009.

(71)

Hero A Justice D.

A binary linear programming formulation of the graph edit distance.

IEEE Trans Pattern Anal Mach Intell., vol. 28, page 1200–1214, 2006.

P Kuner and B Ueberreiter.

Pattern Recognition by Graph Matching: Combinatorial versus Continuous Optimization.

International Journal in Pattern Recognition and Artificial Intelligence, vol. 2, page 527–542, 1988.

P. LeBodic and H. Heroux.

An integer linear program for substitution-tolerant subgraph isomorphism and its use for symbol spotting in technical drawings.

(72)

B. Messmer and Bunke.

Error-Correcting Graph Isomorphism using Decision Trees.

Int. Journal of Pattern Recognition and Ar, vol. 12, page 721–742, 1998.

Bunke H Neuhaus M.

A convolution edit kernel for error-tolerant graph matching.

Proceedings of IEEE international conference on pattern recognition., vol. 30, pages 220–223, 2006.

Burie J. Raveaux R. and Ogier.

A graph matching method and a graph matching distance based on subgraph assignments.

Pattern Recognition Letters, page 394–406, 2010.

Kaspar Riesen and Horst Bunke.

Approximate graph edit distance computation by means of bipartite graph matching.

(73)

Image Vision Comput., vol. 27, no. 7, pages 950–959, 2009.

Bretzner L. Macrini D. Fatih Demirci M. J¨onsson C.

Shokoufandeh A. and S. Dickinson.

The representation and matching of categorical shape.

Computer Vision and Image Understanding, page 139–154, 2006.

Christine Solnon.

Reactive Tabu Search for Measuring Graph.

pages 172–182, 2005.

P. N. Suganthan.

Attributed relational graph matching by neural-gas networks.

IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing X, pages 366–374, 2000.

S. Umeyama.

(74)

An eigendecomposition approach to weighted graph matching problems.

IEEE Pattern Anal. Mach. Intel, page 695–703, 1988.

J. Kittler W. Christmas and M. Petrou.

Structural matching in computer vision using probabilistic relaxation.

IEEE Trans. PAMI,, vol. 2, page 749–764, 1995.