HABILITATION A DIRIGER DES RECHERCHES Contributions and perspectives on combinatorial optimization and machine learning for graph classiﬁcation and matching Romain Raveaux

(1)

Contributions and perspectives on combinatorial optimization and machine learning for graph classification and matching

Romain Raveaux

Polytech’Tours - Universit´e de Tours - LIFAT

26 Juin 2019

(2)

Content

1 Curriculum vitæ and pedagogical activities

2 Scientific activities

3 Research on graph matching and classification

4 Conclusions and perspectives

(3)

Curriculum vitæ and pedagogical activities

(4)

Curriculum vitæ and pedagogical activities Scientific activities Research on graph matching and classification Conclusions and perspectives

Educations, diplomas and qualifications Pedagogical activities

Education

Diplomas:

Date Degree Discipline Establishment

2004-2006 Master Electrical/computer engineering

Universit´e de Rouen 2004-2006 Master Computer science Universit´e de Rouen

Master thesis: Symbol recognition in electrical diagram images.

Supervised by S´ebastien Adam and Pierre H´eroux.

PhD thesis: Graph mining and graph classification: Application to cadastral map analysis. Supervised by Jean-Marc Ogier and Jean-Christophe Burie.

(5)

Education

Diplomas:

Date Degree Discipline Establishment

2004-2006 Master Electrical/computer engineering

Université de Rouen 2004-2006 Master Computer science Université de Rouen 2006-2010 PhD Computer science Université de La Rochelle

Master thesis: Symbol recognition in electrical diagram images.

Supervised by S´ebastien Adam and Pierre H´eroux.

PhD thesis: Graph mining and graph classification: Application to cadastral map analysis. Supervised by Jean-Marc Ogier and Jean-Christophe Burie.

(6)

Qualifications and positions

1 In 2010, qualified to be assistant professor CNU sections: 27, 61

27: Computer science

61: Automatic and signal processing

2 In 2012, assistant professor Universit´e de Tours

Teaching: Engineering school: Polytech’Tours Research: Computer science laboratory: LIFAT

(7)

Pedagogical activities

1 Teaching: around 240h (EqTD) per year in average.

Licence level:Networking(L3), Python and data sciences(L2) Master level: Mobile systems(M2), Multimedia systems(M2) 2012-2015: Lecturer in the International Master on

Computer-Aided Decision Support

2 Pedagogical responsibilities: around 25h (EqTD) per year.

In charge of the fifth year of an engineering specialty In charge of student projects (collective and final projects)

(8)

Scientific activities

(9)

Co-supervisions of students Animations and responsibilities Projects and collaborations Dissemination

Co-supervisions of PhD students

Date Name Funding Teaching Co-supervised by

2012-2016 Zeina Abu-Aisheh Minister Assistant lecturer J.Y Ramel, P. Martineau^∗ 2015-2018 Mostafa Darwiche Region Assistant lecturer D. Conte, V. T’Kindt^∗ 2015-201ˆ9 Maxime Martineau Project Assistant lecturer D. Conte, G. Venturini

∗: A collaborative work: RFAI and ROOT teams Quality of the management:

1 In numbers:

3.3 Journal papers in average by PhD

4.6 Conference, workshop papers in average by PhD

2 Without numbers:

After the PhD: they stay with us (postdoc, lecturer) Then, they find jobs (Shape.ai, CIRELT, ...)

2 PhDs as young researchers, kindness communication

(10)

Co-supervisions of PhD students

Date Name Funding Teaching Co-supervised by

2012-2016 Zeina Abu-Aisheh Minister Assistant lecturer J.Y Ramel, P. Martineau^∗ 2015-2018 Mostafa Darwiche Region Assistant lecturer D. Conte, V. T’Kindt^∗ 2015-201ˆ9 Maxime Martineau Project Assistant lecturer D. Conte, G. Venturini

∗: A collaborative work: RFAI and ROOT teams Quality of the management:

1 In numbers:

3.3 Journal papers in average by PhD

4.6 Conference, workshop papers in average by PhD

2 Without numbers:

After the PhD: they stay with us (postdoc, lecturer) Then, they find jobs (Shape.ai, CIRELT, ...)

PhD supervision: a method

(11)

Scientific expertise

1 Program Committee: Conferences

International Conference on Document Analysis and Recognition: ICDAR 2019, 2017

Graph-Based Representation for pattern recognition: GBR 2019

Graphics Recognition: GREC 2019, 2015, 2013, 2011 International Conference on Hybrid Artificial Intelligence Systems: HAIS 2010, 2011

2 Reviewer for:

Pattern Recognition Letters (PRL) (1 per year) Pattern Recognition (PR)(1 per year)

IEEE Transaction on Image Processing (TIP) (occasionally) International Journal On Document Analysis and Recognition

(12)

Scientific animation

Science administration:

Since 2016, member of the board of VALCONUM.

VALCONUM is a European public-private structure aiming to accelerate the technological transfer in the field of

dematerialization and the valorization of digital contents.

recognition.

Conference organizations:

Participation to the organization of GBR 2019, DAS 2014 and GREC 2009

(13)

Scientific animation

Science administration:

Since 2016, member of the board of VALCONUM.

VALCONUM is a European public-private structure aiming to accelerate the technological transfer in the field of

dematerialization and the valorization of digital contents.

Contest organization:

2011, participation to the organization of a contest on symbol recognition.

Conference organizations:

Participation to the organization of GBR 2019, DAS 2014 and GREC 2009

(14)

Projects and collaborations

Projects:

Name Date Scope Funding Involvement Status

e-trap2 2019 National 450K 15% Submitted

LOR 2019 National 600K 33% Submitted

VISIT 2019 Regional 150K (LIFAT) 20% Accepted. Running

ADAM-IoT 2019 European 1M – Rejected

Fibravasc 2018 Regional 100K (LIFAT) 10% Accepted. Running

Malagga 2018 National - - Rejected

ScannerLoire 2018 Regional - - Rejected

ADT 2016 Regional - - Rejected

Caramba 2016-2019 Regional 200K 33% Accepted. Running

DoD 2014-2015 Industrial 200K 33% Accepted. Done

Collaborations:

Institute Country Topic Publication

GREYC France, Caen Graph edit distance 1 journal

LITIS France, Rouen Graph prototype, Graph edit distance 2 journals

(15)

Publications

International journal IF 5 years¹ SNIP² SRJ³ #articles

Pattern Recognition 4.3 2.47 1.06 2

Expert Systems with Applications 3.7 2.4 1.27 2

Computer and Operation Research 3.2 2.094 1.9 1

Computer Vision and Image Understanding 2.7 1.8 0.71 1

Pattern Recognition Letters 2.3 1.58 0.662 6

Journal of Visual Communication and Image Representation 2.02 1.36 0.5 1 International Journal on Document Analysis and Recogni-

tion

1.26 1.4 0.387 1

Signal Image and Video Processing 1.6 NA NA 1

15

International Conference Rank⁴ #articles

International Conference on Document Analysis and Recognition (ICDAR) A 2 Structural, Syntactic, and Statistical Pattern Recognition (SSPR) A 1 International Workshop on Document Analysis Systems (DAS) B 2

European Signal Processing Conference (EUSIPCO) B 1

International Conference on Pattern Recognition (ICPR) B 3 8

1 Impact factor from editor websites (June 2018).

(16)

Data sets, codes and contest

Data sets are available online

http://www.rfai.li.univ-tours.fr/PublicData/

GDR4GED/home.html Codes are available online

http://www.rfai.lifat.univ-tours.fr/PublicData/

GraphLib/home.html

https://networkx.github.io/networkx/algorithms/

similarity

Participation to a contest ICPR 2016 https://gdc2016.greyc.fr/

(17)

Research on graph matching and classification

(18)

Problems and techniques

1 Problems: graph matching and graph classification

2 Techniques: thanks to machine learning and operations research

3 A coherent work: my PhD, PhDs of Zeina Abu-aisheh, Mostafa Darwiche and Maxime Martineau.

(19)

Problems and techniques

(20)

Problems and techniques

(21)

Toward Graph matching: A set of nodes

Each node can be a vector, an image or a word, ...

(22)

Toward Graph matching: Two sets of nodes

(23)

Toward Graph matching: Set matching

(24)

Toward Graph matching: Two graphs

(25)

Toward Graph matching (f : V

1

→ V

2

)

ij kl

(26)

Graph classification

Toxic or not

(27)

Operations Research

A problem has to be solved

1 Operations Research models and solves combinatorial problems

2 Optimization problems need to be formalized and well structured

3 The human or (a priori) knowledge about the problems is introduced through variables, constraints and objectives

=⇒ link with expert systems

(28)

Machine Learning

A problem has to be solved But: lackof formal specifications.

A possible solution:

1 Formulate a (statistical) proxy problem that relies on data

2 then use machine learning.

Machine Learning:

1 discover regularities on a given set of data (from

“unstructured” or “not formalized” information)

2 generalizes to unseen data

(29)

Machine Learning

Machine Learning:

(30)

Machine Learning

Machine Learning:

(31)

Problems and state of the art

1 Curriculum vitæ and pedagogical activities Educations, diplomas and qualifications Pedagogical activities

3 Research on graph matching and classification Overview of research topics

Problems and state of the art My contributions

(32)

Graph matching: the problem at glance

Input:

1 Two graphs: G₁,G₂

2 Similarity functions:

s_V,s_E

s_V(i,k)

sV(j,k)

s_E(ij,kl)

s_V(i,l)

(33)

Graph matching: the problem at glance

Output:

Node-to-node matching

Binary variables: Y Y_i,k = 1 if i andk are matched

Y_i,k

Yj,k

Y_i_,l

(34)

Graph matching: the problem at glance

Objective function: similarity of two graphs: Y_i,k =Y_j_,l = 1 S(G₁,G₂,Y) =s_V(i,k).Y_i,k +s_V(j,l).Y_j,l+s_E(ij,kl).Y_i,k.Y_j_,l

sV(i,k)∗Yi,k

s_E(ij,kl)∗Y_i,k∗Y_j_,l

(35)

Graph matching: the problem at glance

Graph matching problem: N P-hard [Gold and Rangarajan, 1996]

FindY such that:

1 the sum of similarities is maximized.

2 a node ofG₁ is matched to at most one node ofG₂

3 a node ofG₂ is matched to at most one node ofG₁

(36)

Graph matching: state of the art

(37)

Graph matching: state of the art

Bottom lines from the state of the art:

1 Many fast heuristics can be found in the literature

2 How accurate are these heuristics compared to exact methods?

(38)

Graph matching: pattern recognition applications

Graph matching can be involved in several pattern recognition applications.

(39)

Overview of research topics Problems and state of the art My contributions

Graph matching: pattern recognition applications

1 Graph comparison

2 Graph similarity search

Sort by similarities

3 Graph classification:

K-Nearest Neighbors(KNN)

Compare the query with all the graphs

Sort by similarities Retain the K most similar graphs

The most frequent class label

(40)

Graph matching: pattern recognition applications

1 Graph comparison

2 Graph similarity search Compare the query with all the graphs

K-Nearest Neighbors(KNN)

(41)

Graph matching: pattern recognition applications

1 Graph comparison

K-Nearest Neighbors(KNN) Compare the query with all the graphs

(42)

Graph matching: pattern recognition applications

1 Graph comparison

(43)

Graph matching: pattern recognition applications

1 Graph comparison

The most frequent class

(44)

Graph matching: KNN state of the art

Graph KNN can be sped up thanks to:

1 Fast graph matching methods [K. Riesen, 2015]

2 Reducing the number of graphs by graph prototypes [M.

Ferrer, 2011], [R. Raveaux, 2011].

1 Keep the most significant graphs

(45)

Graph matching: KNN state of the art

Graph KNN can be sped up thanks to:

1 Fast graph matching methods [K. Riesen, 2015]

2 Reducing the number of graphs by graph prototypes [M.

Ferrer, 2011], [R. Raveaux, 2011].

1 Keep the most significant graphs

How to reduce the number of comparisons without loss of information?

(46)

Graph matching: pattern recognition applications

We know how to compare graphs thank to graph matching.

Graph matching offers an elegant manner to compare graphs directly ingraph space.

Node/Edge similarity functions are crucial

Similarity functions link the graph matching problem to the final application.

BUT:

1 How to compare graphs according to a specific objective?

2 How to reach a goal that is defined by data?

3 Where machine learning can be introduced?

(47)

Graph matching: pattern recognition applications

But how to choose the similarity functions between nodes and edges?

2 How to reach a goal that is defined by data?

3 Where machine learning can be introduced?

(48)

Graph matching: pattern recognition applications

But how to choose the similarity functions between nodes and edges?

BUT:

1 How to compare graphs according to a specific objective?

(49)

Learning graph matching: the problem at glance

(50)

Learning graph matching: the state of the art

(51)

Deadlocks

1 How to design fast and accurate graph matching methods?

2 How to reduce the number of graph comparisons?

3 How to learn graph matching for classification?

(52)

My contributions

1 Curriculum vitæ and pedagogical activities Educations, diplomas and qualifications Pedagogical activities

3 Research on graph matching and classification Overview of research topics

Problems and state of the art My contributions

(53)

Deadlock 1: How to design fast and accurate graph

matching methods?

(54)

Branch and bound for graph matching

20102011 2013 20152015 201720182019

Search space is a tree A tree node is a partial matching

Depth-first search with backtracking

(55)

Branch and bound for graph matching

20102011 2013 20152015 201720182019

[Abu-Aisheh et al., 2015]

(56)

Branch and bound for graph matching

20102011 2013 20152015 201720182019

(57)

Branch and bound for graph matching

20102011 2013 20152015 201720182019

(58)

Branch and bound for graph matching

Anytime branch and bound [Abu-Aisheh et al., 2016]:

finds quickly a solution: Depth-first search [Abu-Aisheh et al., 2015]

keeps on searching for improving solutions can be easily interrupted at each tree node

can be interrupted anytime to provide the best solution found so far.

applications

(59)

Branch and bound for graph matching

Anytime branch and bound [Abu-Aisheh et al., 2016]:

finds quickly a solution: Depth-first search [Abu-Aisheh et al., 2015]

keeps on searching for improving solutions can be easily interrupted at each tree node

can be interrupted anytime to provide the best solution found so far.

Anytime algorithm: suitable when the time given to graph matching methods is uncertain. Good for pattern recognition applications

(60)

Branch and bound for graph matching

20102011 2013 20152015 201720182019

Anytime branch and bound [Abu-Aisheh et al., 2016]

Quick first solution:

Depth-first search [Abu-Aisheh et al., 2015]

Parallel branch and bound [Abu-Aisheh et al., 2018].

Work-stealing strategy to

(61)

Mathematical programming for graph matching

20102011 2013 2015 2017201820192017

ILP for graph matching (F1, F2, F3):

[Lerouge et al., 2017]

Linearization of the quadratic problem

Edge matching variables (Z)

s_V(i,k)∗Y_i,k

s_E(ij,kl)∗Y_i,k∗Y_j,l

s_V(j,l)∗Y_j,l

(62)

Mathematical programming for graph matching

20102011 2013 2015 2017201820192017

ILP for graph matching (F1, F2, F3):

[Lerouge et al., 2017]

Linearization of the quadratic problem

Edge matching variables (Z)

s_V(i,k)∗Y_i,k

s_E(ij,kl)∗Z_ij,kl

s_V(j,l)∗Y_j,l

(63)

Mathematical programming for graph matching

New constraints should be added to ensure that if 2 edges are matched (Zij,kl = 1) then related nodes are matched too.

Topological constraints:

Z_ij,kl ≤Y_i,k ∀(ij,kl)∈E₁×E₂ Zij,kl ≤Yj,l ∀(ij,kl)∈E1×E2

s_V(i,k)∗Y_i,k

s_E(ij,kl)∗Z_ij,kl

s_V(j,l)∗Y_j,l

(64)

Mathematical programming for graph matching

20102011 2013 2015 2017201820192017

1 Local Branching Heuristic: LocBra [Darwiche et al., 2018]

2 The goal is to intelligently explore the solution space By taking advantage of an ILP and a mathematical solver (Cplex)

3 It is an iterative process starting from an initial solution (x⁰)

4 x is a vector of binary variables grouping Y and Z

(65)

Mathematical programming for graph matching

Local search: Search for an improving solution inside a small region of the solution space.

Neighborhood: no more than k variables should be changed between the new solution andx⁰.

This neighborhood definition is a new constraint in the ILP

(66)

Mathematical programming for graph matching

No improved solutions found→ Diversification operator for GM:

Goal: Helps skipping a local optimum.

To change the region, change important variables

(67)

Mathematical programming for graph matching

No improved solutions found→ Diversification operator for GM:

Goal: Helps skipping a local optimum.

To change the region, change important variables

Important variables=a high impacts on the objective function value.

(68)

Deadlock 2: How to reduce the number of graph

comparisons?

(69)

Fused graph matching and KNN problems for classification

20102011 2013 2015 2017201820192017

Problem: Solve the KNN problem for graphs

Issue: Many graph comparisons and the process is slow

Answer: Merge GM and KNN in a single problem

Why: Global reasoning instead of

(70)

Fused graph matching and KNN problems for classification

(GM+KNN):

The search space is organized as atree First floor: a tree node is a graph comparison

Each graph comparison is an instance of graph matching It can be solved by a Branch and Bound method

(71)

Fused graph matching and KNN problems for classification

Use case: We are looking for the nearest neighbor

1 Compare Gq with G1: S(Gq,G1) = 4

3 From the first floor, we estimate that S(Gq,G2)≤2

4 G₁ is a better neighbor, no need to explore fully the sub-tree (Gq,G2)

5 Avoid (full) comparisons of very dissimilar graphs

(72)

Fused graph matching and KNN problems for classification

2 Start to compare G_q with G₂ and use S(G_q,G₁) = 4 as a lower bound to cut branches.

(Gq,G2)

(73)

Fused graph matching and KNN problems for classification

(74)

Fused graph matching and KNN problems for classification

(75)

Fused graph matching and KNN problems for classification

(76)

Deadlock 3: How to learn graph matching?

(77)

Learn graph matching for classification

20102011 2013 2015 2017201820192018

Parametrized graph matching

[Raveaux et al., 2017]

Learning graph matching for classification

s_V(i,k)

sE(ij,kl)

(78)

Learn graph matching for classification

20102011 2013 2015 2017201820192018

Parametrized graph matching

[Raveaux et al., 2017]

Learning graph matching for classification

s_V(i,k).β_k

sE(ij,kl).βkl

(79)

Learn graph matching for classification

20102011 2013 2015 2017201820192018

(80)

Learn graph matching for classification

20102011 2013 2015 2017201820192018

(81)

Conclusions and perspectives

(82)

Take a step back on graph matching

From the operations research side

N P-hard problem

No single method can effectively address all instances.

Computer vision and pattern recognition:

low computational time usually dominates the optimality guarantees.

Complementary of the methods There is a need to combine heuristics

(83)

Conclusions Short term perspectives Long term perspectives

Take a step back on graph matching

From the machine learning viewpoint

Similarity functions are crucial

Learning node/edge embedding, learning similarity functions To reach a specific objective (the user need).

can make the problem easier to solve.

Node embedding integrating topological information

can help to recast the quadratic problem to linear assignment problem

(84)

Take a step back on graph matching

Goodsimilarity functions

allow to easily differentiate between vertices/edges can make the problem easier to solve.

problem

(85)

Take a step back on graph matching

Goodsimilarity functions

allow to easily differentiate between vertices/edges can make the problem easier to solve.

Node embedding integrating topological information

can help to recast the quadratic problem to linear assignment

(86)

Take a step back on graph matching

Graph matching for graph classification

Question: What is the meaning of graph matching for graph classification?

GM imposes node assignment constraints

generalize on unseen data?

Are constraints useful to reduce the number of training data?

(87)

Take a step back on graph matching

GM imposes node assignment constraints GM brings constraints to the learning problem:

(88)

Take a step back on graph matching

Do constraints act like a regularization term to better generalize on unseen data?

(89)

Take a step back on graph matching

Do constraints act like a regularization term to better generalize on unseen data?

These open questions are important to legitimate graph matching for graph classification.

(90)

Graph matching what do we need?

From the operations research side:

1 Fast, scalable and accurate heuristics are always welcome for computer vision and pattern recognition communities.

2 Graph matching methods suited to execution on GPU

3 Learning problems are often solved by gradient descent so differentiable methods are wanted → link with robust optimization?

4 Avoid the storage of similarity matrices (|V₁|.|V₂| × |V₁|.|V₂|)

(91)

Graph matching what do we need?

From the machine learning viewpoint:

1 Low complexity (near linear time)

differentiable methods are wanted → link with robust optimization?

(92)

Graph matching what do we need?

optimization?

(93)

Graph matching what do we need?

(94)

Graph matching what do we need?

(95)

Short term perspectives

Future master students (low hanging fruits)

1 (KNN + GM) solved by the local branching heuristic

2 Parametrized GM as an input layer of a MLP

3 ILP for the MCS problem Future PhD students

1 Learning graph matching: hierarchical feature learning (Graph Neural Network) + a combinatorial layer (graph matching method)

2 Learning graph matching: learn to branch in a branch and bound (reinforcement learning)

(96)

Long term perspectives

To be curious:

1 Cross fertilization: OR, ML: LOR project

2 Inspired by other problems:

From computer vision: CRF MAP-inference From OR: TSP, scheduling problems Fundamental:

On the relation between graph matching and Optimal Transport (OT) (Gromov-Wasserstein distance).

Applications:

1 Graph matching for multiple object tracking in videos, table comparisons in documents, ...

(97)

Long term perspectives

To be curious:

1 Cross fertilization: OR, ML: LOR project

2 Inspired by other problems:

From computer vision: CRF MAP-inference From OR: TSP, scheduling problems Fundamental:

On the relation between graph matching and Optimal Transport (OT) (Gromov-Wasserstein distance).

Applications:

1 Graph matching for multiple object tracking in videos, table comparisons in documents, ...

2 Graph matching for unsupervised domain adaptation

(98)

Thank you

Any questions?

(99)

Pattern Recognition Letters, 84:215–224.

Abu-Aisheh, Z., Raveaux, R., and Ramel, J. (2017).

Fast nearest neighbors search in graph space based on a branch-and-bound strategy.

In [Foggia et al., 2017], pages 197–207.

Abu-Aisheh, Z., Raveaux, R., Ramel, J., and Martineau, P.

(2015).

An exact graph edit distance algorithm for solving pattern recognition problems.

In Marsico, M. D., Figueiredo, M. A. T., and Fred, A. L. N., editors,ICPRAM 2015 - Proceedings of the International

(100)

Abu-Aisheh, Z., Raveaux, R., Ramel, J., and Martineau, P.

(2018).

A parallel graph edit distance algorithm.

Expert Syst. Appl., 94:41–57.

Cho, M., Alahari, K., and Ponce, J. (2013).

Learning graphs to match.

InIEEE International Conference on Computer Vision, ICCV 2013, pages 25–32.

Cort´es, X., Serratosa, F., and Serratosa, F. (2016).

Learning graph matching substitution weights based on the ground truth node correspondence.

(101)

A local branching heuristic for solving a graph edit distance problem.

Computers and Operations Research.

Foggia, P., Liu, C., and Vento, M., editors (2017).

Graph-Based Representations in Pattern Recognition - 11th IAPR-TC-15 International Workshop, GbRPR 2017, Anacapri, Italy, May 16-18, 2017, Proceedings, volume 10310 of Lecture Notes in Computer Science.

Gold, S. and Rangarajan, A. (1996).

A graduated assignment algorithm for graph matching.

IEEE Transactions on Pattern Analysis and Machine

(102)

New binary linear programming formulation to compute the graph edit distance.

Pattern Recognition, 72:254–265.

Martineau, M., Raveaux, R., Conte, D., and Venturini, G.

(2018).

Learning error-correcting graph matching with a multiclass neural network.

Pattern Recognition Letters.

Nowak, A., Villar, S., Bandeira, A. S., and Bruna, J. (2017).

A note on learning algorithms for quadratic assignment with graph neural networks.

(103)

Learning graph matching with a graph-based perceptron in a classification context.

In [Foggia et al., 2017], pages 49–58.