HAL Id: hal-02305469
https://hal.archives-ouvertes.fr/hal-02305469v3
Submitted on 4 Dec 2019
HAL
is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire
HAL, estdestinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Removing non-signicant regions in hierarchical clustering and segmentation
Benjamin Perret, Jean Cousty, Silvio Jamil Ferzoli Guimarães, Yukiko Kenmochi, Laurent Najman
To cite this version:
Benjamin Perret, Jean Cousty, Silvio Jamil Ferzoli Guimarães, Yukiko Kenmochi, Laurent Najman.
Removing non-signicant regions in hierarchical clustering and segmentation. Pattern Recognition
Letters, Elsevier, 2019, 128, pp.433-439. �10.1016/j.patrec.2019.10.008�. �hal-02305469v3�
HAL Id: hal-02305469
https://hal.archives-ouvertes.fr/hal-02305469v3
Submitted on 4 Dec 2019
HAL
is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire
HAL, estdestinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Removing non-signicant regions in hierarchical clustering and segmentation
Benjamin Perret, Jean Cousty, Silvio Jamil Ferzoli Guimarães, Yukiko Kenmochi, Laurent Najman
To cite this version:
Benjamin Perret, Jean Cousty, Silvio Jamil Ferzoli Guimarães, Yukiko Kenmochi, Laurent Najman.
Removing non-signicant regions in hierarchical clustering and segmentation. Pattern Recognition
Letters, Elsevier, 2019, 128, pp.433-439. �10.1016/j.patrec.2019.10.008�. �hal-02305469v3�
Research Highlights (Required)
• First algorithm for removing non-relevant regions in hierarchies of partitions.
• Efficient algorithm withO(nlog(n)) time complexity.
• The regions of the simplified hierarchy are regions or union of regions of the initial hierarchies.
• Evaluation on natural image analysis and illustrations on hierarchical clustering of data points.
• The method can be used as a pre- or post-processing step to enhance the quality of hierarchical segmentation algorithms.
1
Pattern Recognition Letters
Removing non-significant regions in hierarchical clustering and segmentation
BenjaminPerreta,∗∗, JeanCoustya, Silvio Jamil FerzoliGuimarãesb, YukikoKenmochia, LaurentNajmana
aUniversité Paris-Est, LIGM, CNRS–ENPC–ESIEE Paris-UPEM
bPUC Minas–ICEI–DCC–VIPLAB
ABSTRACT
We propose an efficient algorithm that removes unimportant regions from a hierarchical partition tree, while preserving the hierarchical partition structure. Various experiments demonstrate that applying this algorithm on various classification or segmentation problems does indeed improve the results by a large margin. Code is available online athttps://github.com/higra/Higra.
1. Introduction
Many algorithms for image segmentation or data clustering contain a step that removes unimportant regions or clusters. In this paper, we are dealing with the more general problem of removing unimportant regions from a hierarchy of partitions, while still preserving the hierarchical partition structure. This is a common problem that appears in many different situations.
For example, constrained connectivity [24] solves the chaining problem well-known as one of the issues with minimum span- ning tree based approaches, but it may creates a series of small undesirable regions in situation where there is a ramp disconti- nuity (see [25] for an analysis of this particular case).
One way to achieve such a hierarchical simplification would be to extract all the possible segmentations from the hierarchy, and for each one of them, remove the non-important regions by merging these regions with one of their neighbours. One of the issues is that those merging steps have to be performed in a consistent way, so that the set of simplified segmentations is still a hierarchy. Another important issue is that such a process would be slow.
In the literature on transformations of hierarchical segmenta- tions [11, 24, 5, 4, 26], there is not guarantee that unimportant regions are removed from the hierarchy. For example, small regions (with small area) can appear at very high level in the hierarchical tree, and the methods do not remove them. Thus, there is a need for post-processing the hierarchy. To the best of
∗∗Authors are sorted by decreasing date of birth.
Corresponding author: Tel.:+33-1-4592-6709;
e-mail:[email protected](Benjamin Perret)
our knowledge, no algorithm has ever been presented for per- forming such a task.
H ΦG w
Hierarchy simplification
w0 H0 QFZ
set-orienteddomain saliencymapdomain
Fig. 1. A flowchart of the proposed method for removing non-significant regions from a given hierarchyHand obtaining a new hierarchyH0.
In order to provide such an efficient algorithm for remov- ing unimportant regions from a hierarchy of partitions, we rely on the framework proposed in [7], where the equiva- lence between various hierarchical representations (dendro- grams, saliency maps or minimum spanning trees) is demon- strated (see Section 2). As shown in Fig. 1, our algo- rithm makes use of these different representations to efficiently achieve its goal. This algorithm has been briefly introduced in the appendix of [12], but a detailed analysis and clear expla- nations were missing; they are provided in Section 3. Further- more, an empirical evaluation demonstrating its practical effec- tiveness is performed in Section 4.
2
a b c
d e f
(a)
a b c
d e f
0 1
2 0 2
2 1
(b)
P0
P1
P2
P3
a b d c f e
(c)
Fig. 2. (a) An example of hierarchy of partitions,H = (P0,P1,P2,P3) where P0 = {{a},{b},{c},{d},{e},{f}}, P1 = {{a,b},{c,f},{d},{e}}, P2 = {{a,b,d},{c,e,f}}, P3 = {{a,b,c,d,e,f}}; (b) the saliency map ofH on a graphG,ΦG(H); (c) the dendrogram of the hierarchyH.
2. Basic notions for graph-based hierarchy processing Any hierarchy can be equivalently represented by sets as se- ries of nested partitions or with a characteristic function defined on the edges of a graph and called a saliency map. The core of the hierarchy simplification method which we propose in this article and which is precisely described in Section 3 considers the saliency map representations of the hierarchies. In this sec- tion, we provide the formal definitions of the set representation of hierarchies and of the saliency maps. We also highlight how one can switch between the set and the functional representa- tions of a hierarchy.
2.1. Hierarchies of partitions
In this article, the symbolVdenotes a finite set which stands for the working space. In applications to image analysis, it can be for instance the set of all image pixels or superpixels. A partition of Vis a setPof nonempty disjoint subsets ofVwhose union isV. Each element ofPis called aregion ofP.
Ahierarchy on V is a sequenceH = (P0, . . . ,P`) of parti- tions ofV such that, for any i ∈ {1, . . . , `}, any region of the partitionPi−1is included in a region ofPi.
Figure 2 (a) and (c) illustrate an example of a hierarchy and of its dendrogram, respectively. Dendrograms are commonly used in applications. Intuitively, the dendrogram of a hierarchy embeds the inclusion relationship between the regions of this hierarchy. More precisely, it is a tree where the nodes corre- spond to the regions of the hierarchy and where each regionRis linked to the largest (non-empty) regions of the hierarchy which are proper subsets ofR, called the children ofR.
2.2. Saliency map
Any hierarchy can be represented by an edge-weighted graph [7] spanning the elements of the spaceV. We provide in this section the definition of such a representation called a saliency map. A graph (spanning V)is a pairG = (V,E) such that E is a subset of the set of all unorderd pairs of distinct elements of V, i.e., the set E is a subset of{{x,y} ⊆ V | x , y}. If G=(V,E) is a graph, each element ofVis called avertex of G, and each element ofE is called anedge of G. A subgraph of a graphG =(V,E) is a graph (V0,E0) such thatV0andE0are subsets ofVand ofE, respectively. IfXis a graph, its vertex and edge sets are denoted byV(X) andE(X), respectively.
A sequence (x0, . . . ,x`) of vertices of a graphGis called a path from x0 to x` in Gif any two successive vertices in the sequence form an edge ofG, i.e., for any i in{1, . . . , `}, the
unordered pair{xi−1,xi}is an edge ofG. Agraph is connected whenever there is a path from any of its vertices to every other one. LetGbe a graph, by extension, we say that asubset R of V is connected(forG) if the subgraph ofGinduced byRis connected,i.e., the subgraph (R,{{x,y} ∈E(G)|x∈R,y∈R}) is connected. Aconnected component of Gis a subsetRofV which is connected and maximal for this property: any proper superset ofRis not connected.
In the sequel of this article, we assume that the spaceV is structured by a graphG=(V,E). For instance, in applications to image analysis, if the setV contains the set of all pixels or superpixels of an image, the edge setEcan be obtained by any pixel or superpixel adjacency relation such as the one induced by the classical 4-, 6- or 8-adjacency relations. Furthermore, we will also assume that any hierarchy onVis connected forG meaning that any region of any considered hierarchy is con- nected for the graphG. These assumptions correspond to the situations which are the most often encountered in hierarchical image analysis. However, they can be dropped by considering that the graphGis the complete graph onV so that any subset ofVis always connected. In such case the notion of a saliency map, whose definition is recalled hereafter, corresponds exactly to the notion of ultrametric distance which is well known in classification [14].
A map w from E into the setR of real numbers is called aweight map on G. For any edgeu of E, the valuew(u) is called the weight of u, and the pair (G,w) is called an edge- weighted graph. Given a graphG =(V,E) and a hierarchyH onV, we show below how to define the saliency mapΦG(H) ofHfromEtoR, which is an equivalent representation of the hierarchyH; knowingHone can inferΦG(H) and, conversely, knowingΦG(H) one can recoverH.
Let us consider a hierarchy H = (P0, . . . ,P`) on V. The saliency map of H is the map ΦG(H) from E to L = {0, . . . , `} ⊂R, such that the weight of any edgeu={x,y}ofG forΦG(H) is the largest valueλinLsuch thatxandybelong to two distinct regions ofPλ. Figure 2 (b) shows the saliency mapΦG(H) of the hierarchyH given in Figure 2 (a).
There is a bijection between the set of all hierarchies on V and the set containing every map which is the saliency map of a hierarchy (see Theorem 1 of [7]). In the next section, we present the quasi-flat-zone transform, denoted byQF Zwhich is the inverse of ΦG and allows to recover the hierarchy H knowing only its saliency mapΦG(H). These two transforms, namelyΦG andQF Z, make it possible to treat a hierachy ei- ther in a “set-oriented domain” (left part of Figure 1) or in the
“saliency map domain” (right part of Figure 1).
An algorithm for computing the saliency map of any hierar- chyHin linear time with respect to the size of the graphG,i.e., O(|V|+|E|), is described in [7]. This algorithm can be sketched as follows:
1. preprocessHfor least common ancestors searches;
2. for each edgeu={x,y}ofGtaken in any order,
2..1 find the least common ancestorRofxand ofyin the dendrogram ofH;
2..2 set the weight ofuto the level ofRin the hierarchy.
3 2.3. Quasi-flat zone hierarchy
Quasi-flat zone transform [19, 17, 7] maps any edge- weighted graph into a hierarchy. In particular, if the departing map is the saliency map of a hierarchy, this transform allows to recover the initial hierarchy. As the hierarchy simplification method which we propose in Section 3 treats the hierarchies from their saliency maps, the quasi-flat zone transform allows us to recover the hierarchy associated with the saliency maps produced by our simplification method (see the overview dia- gram of Figure 1). Intuitively, this transform considers the se- ries of the connected component partitions induced by the suc- cessive level sets of the edge-weight map.
Given an edge-weighted graph (G,w) and a value λ ∈ R,
the λ-level set of E for w is defined by wλ(E) = {u ∈ E |
w(u) < λ}and its associated subgraph (V,wλ(E)), denoted by wλ(G), is called theλ-level graph of G for w. The set of all connected components ofwλ(G), denoted by C(wλ(G)), is a partition ofV called theλ-level partition of G for w. Given an edge-weighted graph (G,w), the quasi-flat zone hierarchy QF Z(G,w)of(G,w) is then the finite sequence of allλ-level partitions ofGforw, ordered by increasing values ofλ, namely,
QF Z(G,w)=(C(wλ(G))|λ∈R).
3. Hierarchy simplification with an attribute criterion The proposed simplification method is based on a regional attribute, such as region area (size) and contrast, which mea- sures the significance of any region. It aims at transforming an initial hierarchy into a new one such that:
• the new hierarchy does not contain any region with an at- tribute value below a given threshold;
• the regions of the new hierarchy are either regions of the initial hierarchies or regions obtained by merging adjacent regions of the initial hierarchy.
As mentioned in the introduction, in order to efficiently per- form such simplification, the hierarchies are represented by weight maps. More precisely, we consider the (spatially and functionally) minimal representation of a hierarchy introduced in [7]: it consists of a minimal weighted subgraph (in terms of inclusion relation on graphs) whose quasi-flat zone hierarchy is precisely the hierarchy that we aim to represent. Such minimal representation of a hierarchy can be obtained by considering first the graphGweighted by the saliency map of the given hi- erarchy and then restricting it to one of its minimum spanning trees (Theorem 12 in [7]), leading to a weighted tree (T,w).
The core of the method is then to produce a new weight mapw0 for this graphT, standing for the saliency map representing the resulting simplified hierarchy. In order to produce such map, the edges of the tree are considered in any order. For each edge{x,y}, the largest region of the hierarchy which containsx but noty and the largest region of the hierarchy which con- tainsybut notxare analyzed. If the attribute value of one of these two regions is below the given threshold, then the two re- gions must be merged. This is done by setting to 0 the weight
of {x,y} for w0. On the contrary, if the attributes of both re- gions are above the given threshold, then the two regions must be kept and we replicate the weight of{x,y}forwintow0. In order to efficiently implement this method, a fundamental oper- ation consists of finding the largest region of a hierarchy which contains one extremity of an edge but not the other. This can be done with the help of a data structure called a binary parti- tion tree by altitude ordering. Hence, before giving a precise presentation of the simplification algorithm in Section 3.2, we first present, in Section 3.1, binary partition trees by altitude ordering together with a simple algorithm to compute them.
3.1. Binary partition tree by altitude ordering
Binary partition trees by altitude ordering (BPTAOs) are deeply related to Kruskal’s minimum spanning tree algorithm [13].
Algorithm 1:Playing with Kruskal Data:An edge-weighted graph ((V,E),w)
Result:An arrayLMS T to store the edges of an MST ofGin non-decreasing order of weight with respect tow
Result:Its associated BPTAOB
1 eB0 ;/* Initialize the index for LMS T */
2 foreachx∈Vdo B.AddNode(x) ;
/* Assuming that V={0, . . . ,n−1} */
3 foreach{x,y} ∈E in non-decreasing order of w do
4 rxBB.FindRoot(x);
5 ryBB.FindRoot(y);
6 ifrx,rythen
7 B.CreateParent(rx,ry);
8 LMS T[e]B{x,y};
9 e+=1;
FunctionB.AddNode(x)
1 B.Parent[x]=−1;
2 B.Size+=1;
FunctionB.FindRoot(x)
1 whileB.Parent[x]≥0doxBB.Parent[x];
2 returnx
FunctionB.CreateParent(x,y)
1 iBB.Size;/* index for the new node */
2 B.AddNode(i);
3 B.Parent[x]Bi;
4 B.Parent[y]Bi;
5 B.LeftChild[i]Bx;
6 B.RightChild[i]By;
4 More precisely, the BPTAO data structure can be seen as the
(tree-based representation of a) hierarchy of partitions ofVob- tained during Kruskal’s minimum-spanning-tree algorithm. A formal definition of this structure can be found in [8] and algo- rithms to construct them are presented in [20]. In this article, for the sake of completeness, we present a simple algorithm to construct it. However, the reader interested into a more efficient construction is refered to [20].
This simple construction of a BPTAO from an edge weighted graph (G,w) is given in Algorithm 1. It corresponds to a par- ticular implementation of Kruskal’s algorithm. The auxiliary functions called in Algorithm 1, namely AddNode, FindRoot and CreateParent, are also described below Algorithm 1. In Al- gorithm 1, we initially consider a partition into singletons (Line 2) which is the first level of the BPTAO. Then, when an edge is selected by Kruskal’s algorithm, we build the next level by merging the largest regions containing the vertices of the se- lected edge{x,y}(Lines 3-9). In terms of tree, the newly cre- ated regionRis a new node of the BPTAO B, which becomes the parent of the two nodes associated with the merged regions (Line 7). There is a direct relation between the newly created regionRand the edge{x,y}that is considered for the merging which creates the regionR. In Algorithm 1, we observe that the edge{x,y}is stored at an indexe(see Line 8) of the arrayLMS T and that the index of the regionR in the tree data structureB isn+e=|V|+e(Line 7), allowing to keep track of the relation between the edge{x,y}and the regionRfor further processing.
When the algorithm terminates, we then obtain:
• a minimum spanning tree of (G,w) whose edges are stored, following a non-decreasing order of weight (called an altitude ordering), in the arrayLMS T;
• a treeB, called the BPTAO of (G,w) associated withLMS T, whose non-leaf nodes correspond to the edges the min- imum spanning tree produced by Kruskal’s algorithm (Line 8) and whose leaves correspond to the vertices of G(Line 2);
• an implicit mapping between the nodes of the BPTAOB and the vertices and edges of the minimum spanning tree.
Any node of B stored at an index between 0 andn −1 is mapped to the vertex of the graphGat the same index, whereas any node ofBwith an indexibetweennand 2n−1 is mapped to the edge of the minimum spanning tree stored inLMS T[i−n].
Figure 3 illustrates the relationship between a minimum span- ning tree ofGand its associated BPTAOB.
It should be also noticed that, if the quasi-flat zone hierarchy QF Z(G,w) is a binary hierarchy (i.e., each region is either a singleton or the result from the merging of exactly two regions), then it is equal to the BPTAO produced by Algorithm 1 [8].
Otherwise, the hierarchyQF Z(G,w) can be straightforwardly recovered fromBas shown in [8, 20].
For a more efficient implementation of Algorithm 1, readers are referred to [20]. Provided that the edges of the graphG are either already sorted or can be sorted in linear time, the efficient algorithm of [20] has a quasi-linear time complexity,
Algorithm 2:Hierarchy simplification by attribute Data:A graphG=(V,E) that is the working space Data:The saliency mapwof a hierarchyH Data:An attribute threshold valuem
Result:A (saliency) saliency mapw0defined on the edges of an MSTT of (G,w)
1 Calculate the ordered edge arrayLMS T and its associated BPTAOB(Algorithm 1);
2 Calculate the region attribute of each nodenofBand store it inA[n];
3 foreachnon-leaf node n of B/* n iterates over the set {|V|, . . . ,2|V| −1} */
4 do
5 a1BA[B.LeftChild[n]];
6 a2BA[B.RightChild[n]];
7 uBLMS T[n− |V|];
8 ifa1 ≥m and a2≥mthenw0(u)Bw(u);
9 elsew0(u)B0;
O(|E(G)| ×α(|V(G)|)), whereαis the extremely slowly growing inverse of the single-valued Ackermann function.
3.2. Hierarchy simplification algorithm
The hierarchy simplification method is precisely described, with the help of Playing with Kruskal algorithm (namely Al- gorithm 1), in Algorithm 2. In the first line of the algorithm, an MST of the saliency mapwof a given hierarchyH and its associated BPTAOBare obtained from Algorithm 1. After cal- culating the attribute for every regionRinB(Line 2), we can efficiently carry out the two main steps of the method for each edgeu∈E(G), thanks to the two structuresLMS TandB: (1) get the attribute values of the associated connected components in Lines 5 and 6, and (2) set the new edge weightw0(u) depending on the verification of the attribute criterion for the two regions merged by the edgeu(Lines 8 and 9). It can be observed that Line 7 uses the mapping between the nodes ofBand the edges of the considered MST, which was presented at the end of Sec- tion 3.1 and which is illustrated with the green arrows in Fig. 3.
It can be observed that, as presented in Figure 1, the hierarchy given to Algorithm 2 and the one resulting from its execution
n1
n2 n4
n3
4 6
3
7 11
8 5
1 2
3
4 5
Fig. 3. Given a weighted graph, its minimum spanning treeT(whose edges are thick and gray) is represented by the binary partition tree by altitude orderingB(in blue). Each leaf node corresponds to a vertex ofTwhile each non-leaf nodeniofBcorresponds to an edge ofT; the correspondences are depicted in green arrows.
5 are in the form of saliency maps denoted bywandw0respec-
tively. The tree-based representation of the simplified hierarchy resulting from Algorithm 2 can be obtained by computing the quasi-flat zone hierarchy of w0 for the MST stored in LMS T. Such computation can be done, for instance, with the algorithm presented in [20].
4. Illustrations and assessments
This section presents qualitative and quantitative assessments of the proposed method. Our tests focus on two different hi- erarchical segmentation methods: the quasi-flat zone hierar- chy (QFZ, see Section 2.3) and the watershed hierarchy by area (WS-Area). Watershed hierarchies were first proposed in [3, 21, 16] and have since been formalized in the context of minimum spanning forests [6]. Intuitively, the WS-Area hi- erarchy of an edge-weighted graph is obtained by sequentially filtering the edge weights of the graph with area closings of in- creasing sizes and then computing the sequence of watershed segmentations of these filtered edge weights.
Then, we consider two regional attributes to simplify those hierarchies:
1. the area of a region, defined as the number of vertices in the region; and
2. the frontier strength of a region, defined as the mean weight of the edges linking the region with its sibling,i.e., the edges on the frontier between the two children of the parent region.
The area attribute of each region can be computed in linear time from the BPTAO by traversing the tree from the leaves to the root, the area of the leaves being 1 and the area of a non-leaf node being the sum of the area of its two children. The frontier strength can also be computed in linear time, by traversing the edges of the graphG, finding the lowest common ancestor of the two vertices of the edge in the BPTAO (this query can be done in constant time thanks to a linear time pre-processing of the tree [2]) and accumulating the edge weights in this region.
The area attribute is used to identify non significant nodes in the QFZ hierarchy, while the frontier strength attribute is used in conjunction with the WS-Area hierarchy.
We first present illustrations of non-significant node removal on hierarchies built on point clouds and images. Then, we present extensive quantitative assessments of the benefits of our procedure for natural image analysis.
4.1. Illustrations
We first demonstrate the effectiveness of the proposed method on the hierarchical analysis of two simulated 2D point clouds (see Figure 4)1. Each point cloud is generated from three random distributions corresponding to three classes. We then consider the graph induced by the Delaunay triangulation
1All the illustrations presented in this section can be reproduced using the Python Notebooks available athttps://higra.readthedocs.io/en/
latest/notebooks.html.
of the points and we weight the edges by the Euclidean dis- tance between the points. We observe that very small regions are branching at very high levels in the dendrogram of the QFZ hierarchy. Hence, the partition containing three regions in the hierarchy fails to correctly recovers the three clusters. By re- moving non-significant nodes from the QFZ hierarchy based on an area attribute (nodes containing less than 7 points are con- sidered non-significant), we ensure that the hierarchy does not contain any small region anymore (neither at high nor at low levels). We observe that the partitions containing three regions in the simplified hierarchy correctly recovers the three clusters.
Another illustration of the effectiveness of the proposed method on hierarchical natural image analysis is demonstrated in Figure 5. In this Figure, the saliency map of a hierarchy (Sec- tion 2.2) is represented in the 2D Khalimsky grid [1, 21]: in this representation, the brightness of a contour is inversely propor- tional to the number of partitions of the hierarchy it belongs to, i.e., dark contours are the strongest ones. We can observe that in the QFZ hierarchy, most strong contours represent very small regions located on thick transitions between different re- gions of the images. When the saliency map is plotted in the 2D Khalimsky grid, this suppression of small regions looks like a sharpening, in other words, thick and blurred transitions be- come sharp. On the contrary, we can see that WS-Area hierar- chy already produces thin contours. However, it also produces a lot of non-significant contours in large homogeneous regions.
After a region removal procedure with a small contour strength from the WS-Area hierarchy, most spurious contours disappear.
4.2. Quantitative assessment
This section presents a quantitative assessment of the pro- posed method on natural image analysis. We first explain the assessment methodology, the evaluation measures, and the im- age datasets. Then, we give the results comparing the QFZ and WS-Area hierarchies to their simplified counterparts. Finally, we also compare our results with the one obtained by the trans- formation of a hierarchy into its optimal cut hierarchy for the piecewise constant Mumford-Shah energy [11].
Methodology. We mainly follow the supervised assessment framework proposed in [22]. We give an overview of the qual- ity measures and readers can refer to the provided references to get detailed descriptions. The assessment framework relies on three types of measures to encompass various aspects of hierar- chical representations:
1. precision-recall and F-measure on boundaries (FB) [1].
This measure evaluates the quality of the boundaries of each partition of a hierarchy with respect to a ground- truth segmentation. To evaluate a hierarchical method on a whole dataset, two aggregated measures are then de- fined: 1) the optimal image scale (OIS) measuring the best achievable score when taking the optimal partition in each hierarchy, and 2) the optimal data-set scale (ODS) measur- ing the best achievable score when taking partitions at the same level (the optimal scale) in every hierarchy;
2. fragmentation curves on the bidirectional-consistency er- ror (BCE) [22]. The fragmentation of a partition is defined
6
Graph QFZ QFZ clustering simplified QFZ simplified QFZ clustering
Fig. 4. Removal of non-significant nodes on the QFZ hierarchical clustering of two point clouds (first and second lines). For each graph, we show from left to right: the graph with the three ground-truth clusterings of the graph vertices (red, green, and blue), the dendrogram of the QFZ hierarchy, the clustering into 3 classes for this dendrogram, the dendrogram of the simplification of the QFZ hierarchy, and the clustering into 3 classes for this simplified dendrogram. Note that the colors used to represent clusterings are arbitrary and do not represent an explicit correspondence between two different clusterings.
Input image QFZ hierarchy simplified QFZ hierarchy WS-Area hierarchy simplified WS-Area hierarchy
Fig. 5. Removal of non-significant nodes of the QFZ hierarchy and of the WS-Area hierarchy on 4 images of the BSDS 500 dataset [1]. For each image, we show from left to right: the input image, the saliency map of the QFZ hiearchy, the saliency map of the simplified QFZ hierarchy, the saliency map of the WS-Area hierarchy, and the saliency map of the simplified WS-Area hierarchy.
as the number of regions in the partition divided by the number of regions in the ground-truth. The fragmenta- tion curve on BCE evaluates the quality of the regions of partitions of the hierarchy as the fragmentation increases, also with respect to a ground-truth segmentation. We con- sider two categories of partitions that can be extracted from a hierarchy: the partitions of the hierarchy (horizon- tal cuts), and the optimal partitions constructable from re- gions taken from any partition of the hierarchy (the opti- mal non-horizontal cuts). Two aggregated measures are defined: the area under the curve for optimal cuts (FOC) and the area under the curve for horizontal cuts (FHC);
3. object detection measure [22, 23]. This last measure is based on supervised object detection with markers (one marker for the object and one for the background) and tries to describe an object as a set of regions taken from any par- tition of the hierarchy. It quantifies how well a specific ob- ject of a scene can be retrieved with different levels of in-
formation given on its position. Markers are automatically generated from the ground-truth and corresponds to: ero- sions of the ground-truth object/background masks (Er), skeletons of the ground-truth object/background masks (Sk), and the frame of the image (Fr). Three combination of background-foreground markers are considered: Er-Er, Fr-Sk, and Sk-Sk. The quality of a detection is measure with its Jaccard index.
The precision-recall curves and fragmentation curves are evaluated on the test set of the Pascal Context dataset [18]
which consists of a pixel-wise segmentation of the last 2 498 images of the Pascal VOC’10 [10] validation set. The object detection measure is evaluated on the MS-COCO [15] dataset.
Each object of the dataset is processed independently leading to a total of 291 875 objects from the 40 504 images of the MS- COCO 2014 validation set.
7
Table 1. Comparison between the QFZ hierarchy, the simplified QFZ hierarchy at a given threshold, and the hierarchy of optimal Mumford-Shah (MS) cuts of QFZ.
FB BCE OD
Mean score
ODS OIS FOC FHC Mean Median
QFZ 0.479 0.477 0.358 0.358 0.500 0.550 0.454 QFZ+simplification 0.4% 0.525 0.580 0.510 0.464 0.552 0.613 0.541 QFZ+simplification 0.8% 0.537 0.589 0.533 0.475 0.533 0.579 0.541 QFZ+simplification 1.6% 0.517 0.602 0.550 0.483 0.505 0.524 0.530 QFZ+simplification 3.2% 0.515 0.543 0.541 0.479 0.467 0.448 0.499 QFZ+MS cuts [11] 0.525 0.523 0.368 0.368 0.503 0.551 0.473 QFZ+Simplification 0.8%+MS cuts 0.595 0.637 0.548 0.505 0.528 0.569 0.564
Table 2. Comparison between the WS-Area hierarchy, the simplified WS-Area hierarchy at a given threshold , and the hierarchy of optimal Mumford-Shah (MS) cuts of WS-Area.
FB BCE OD
Mean score
ODS OIS FOC FHC Mean Median
WS-Area 0.512 0.591 0.588 0.440 0.518 0.552 0.534 WS-Area+simplification 0.05 0.522 0.592 0.589 0.445 0.519 0.554 0.536 WS-Area+simplification 0.08 0.527 0.596 0.591 0.452 0.519 0.554 0.540 WS-Area+simplification 0.10 0.530 0.599 0.591 0.457 0.518 0.553 0.541 WS-Area+simplification 0.15 0.541 0.604 0.593 0.470 0.511 0.539 0.543 WS-Area+simplification 0.20 0.541 0.605 0.592 0.482 0.490 0.503 0.536 WS-Area+MS cuts [11] 0.535 0.585 0.615 0.514 0.531 0.576 0.559
Results. Each image of the test datasets presented in the pre- vious section was first transformed into a 4-adjacency graph.
The edge weights of the graph of an image are then defined as the mean gradient value of its two extremities, the gradient be- ing obtained with the structured edge detector [9]. In order to evaluate the benefits of the proposed method we propose two comparisons:
1. QFZ hierarchy versus a simplified QFZ hierarchy where small regions have been removed. The area threshold is expressed as a fraction of the total number of pixels in the image.
2. WS-Area hierarchy versus a simplified WS-Area hierar- chy where regions with a common weak frontier have been merged. The strength threshold assumes that gradient val- ues are normalized between 0 and 1.
Table 1 shows the results obtained with QFZ hierarchies. We can see that the removing of small regions provides signifi- cant improvements for the three measures. A threshold level of 0.4%or 0.8%(between 150 and 400 pixels on the tested images) offers a good compromise on the different measures.
Table 2 shows the results obtained with WS-Area hierarchies.
In this case, the results are more contrasted. While a suppres- sion of weak contours can provide significant improvement on precision-recall curves and fragmentation curves, the effect can be rapidly detrimental to the object detection measure. This is- sue can be due to the fact that the MS-COCO dataset contains a lot of poorly resolved objects with weak contours that can be deleted by the proposed method. However, we still see that a hi- erarchy simplification with moderate threshold values (between 0.05 and 0.1) improves all the considered quality measures.
Finally, the last lines of Tables 1 and 2 show the results ob- tained by the transformation of a hierarchy into its optimal cut hierarchy [11]. We recall that this transformation modify the level of the nodes of a hierarchy such that each partition of the transformed hierarchy is optimal for the piecewise constant Mumford-Shah energy whose regularization parameter is equal to the level of the partition. We can see that this transformation provides very good results on the WS-Area hierarchy where it can identify incorrect contours, thanks to the rich information provided by the Mumford-Shah energy, and then push them down to the bottom of the hierarchy. It is however unable to deal with the small regions present close to the top of the QFZ hierarchy as pushing them down to the bottom would require to completely collapse the hierarchy. Nevertheless, we can see that the combination of the two transformation methods, the proposed simplification strategy followed by the transformation into optimal cut hierarchy, on QFZ (last line of Table 1) gives the best result. This further support the idea that the proposed method can be used as a pre- or post-processing step to enhance the quality of hierarchical segmentation algorithms.
5. Conclusion and perspectives
In this paper, relying on the framework developed in [7], we have provided a generic solution to the common problem of removing non-significant regions from a hierarchy of par- titions. The experiments demonstrate that applying this algo- rithm does indeed improve the results in a number of situations.
Future work will combine our approach with probability func- tions (e.g., attention saliency) or some other criterion relying on deep learning techniques to achieve state-of-the-art results.
8 Acknowledgments
The authors are grateful to CNPq (Universal 421521/2016-3 and PQ 307062/2016-3), FAPEMIG (PPM-00006-16) and PUC Minas for the financial support to this work. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001 and by CAPES/COFECUB 88887.191730/2018-00.
References
[1] Arbelaez, P., Maire, M., Fowlkes, C., Malik, J., 2011. Contour detection and hierarchical image segmentation. IEEE PAMI 33, 898–916. doi:10.
1109/TPAMI.2010.161.
[2] Bender, M., Farach-Colton, M., 2000. The lca problem revisited, in:
Gonnet, G., Viola, A. (Eds.), LATIN 2000: Theoretical Informatics, Springer Berlin Heidelberg. pp. 88–94.
[3] Beucher, S., 1994. Watershed, hierarchical segmentation and waterfall algorithm, in: Serra, J., Soille, P. (Eds.), ISMM, pp. 69–76.
[4] Bosilj, P., Lefèvre, S., Kijak, E., 2013. Hierarchical image representation simplification driven by region complexity, in: International Conference on Image Analysis and Processing, Springer. pp. 562–571.
[5] Cardelino, J., Caselles, V., Bertalmio, M., Randall, G., 2013. A contrario selection of optimal partitions for image segmentation. SIAM Journal on Imaging Sciences 6, 1274–1317.
[6] Cousty, J., Najman, L., 2011. Incremental algorithm for hierarchical minimum spanning forests and saliency of watershed cuts, in: ISMM, Springer. pp. 272–283.
[7] Cousty, J., Najman, L., Kenmochi, Y., Guimarães, S., 2018. Hier- archical segmentations with graphs: Quasi-flat zones, minimum span- ning trees, and saliency maps. JMIV 60, 479–502. doi:10.1007/
s10851-017-0768-7.
[8] Cousty, J., Najman, L., Perret, B., 2013. Constructive links between some morphological hierarchies on edge-weighted graphs, in: ISMM, Springer.
pp. 86–97.
[9] Dollár, P., Zitnick, C.L., 2015. Fast edge detection using structured forests. IEEE PAMI 37, 1558–1570. doi:10.1109/TPAMI.2014.
2377715.
[10] Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A., . The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal- network.org/challenges/VOC/voc2010/workshop/index.html.
[11] Guigues, L., Cocquerez, J., Le Men, H., 2006. Scale-sets image analysis.
International Journal of Computer Vision 68, 289–317. doi:10.1007/
s11263-005-6299-0.
[12] Guimarães, S., Kenmochi, Y., Cousty, J., Patrocinio, Z., Najman, L., 2017. Hierarchizing graph-based image segmentation algorithms rely- ing on region dissimilarity: the case of the felzenszwalb-huttenlocher method. Mathematical Morphology - Theory and Applications 2, 55–75.
[13] Kruskal, J.B., 1956. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society 7, 48–50.
[14] Leclerc, B., 1981. Description combinatoire des ultramétriques. Mathé- matiques et Sciences humaines 73, 5–37.
[15] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L., 2014. Microsoft COCO: Common Objects in Context, in: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (Eds.), ECCV, pp. 740–755. doi:10.1007/978-3-319-10602-1_48.
[16] Meyer, F., 1996. The dynamics of minima and contours, in: Maragos, P., Schafer, R., Butt, M. (Eds.), ISMM, pp. 329–336.
[17] Meyer, F., Maragos, P., 1999. Morphological scale-space representa- tion with levelings, in: Nielsen, M., Johansen, P., Olsen, O., Weickert, J. (Eds.), Scale-Space Theories in Computer Vision, pp. 187–198.
[18] Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., Yuille, A., 2014. The role of context for object detection and semantic segmentation in the wild, in: IEEE CVPR.
[19] Nagao, M., Matsuyama, T., Ikeda, Y., 1979. Region extraction and shape analysis in aerial photographs. Computer Graphics and Image Processing 10, 195–223.
[20] Najman, L., Cousty, J., Perret, B., 2013. Playing with kruskal: Algo- rithms for morphological trees in edge-weighted graphs, in: Hendriks, C., Borgefors, G., Strand, R. (Eds.), ISMM, pp. 135–146.
[21] Najman, L., Schmitt, M., . Geodesic saliency of watershed contours and hierarchical segmentation. IEEE PAMI .
[22] Perret, B., Cousty, J., Guimarães, S.J., Maia, D.S., 2018. Evaluation of hierarchical watersheds. TIP 27, 1676–1688. doi:10.1109/TIP.2017.
2779604.
[23] Perret, B., Cousty, J., Rivera Ura, J.C., Guimarães, S.J.F., 2015. Eval- uation of morphological hierarchies for supervised segmentation, in:
Benediktsson, J., Chanussot, J., Najman, L., Talbot, H. (Eds.), ISMM, Springer. pp. 39–50. doi:10.1007/978-3-319-18720-4\_4. [24] Soille, P., 2008. Constrained connectivity for hierarchical image parti-
tioning and simplification. IEEE PAMI 30, 1132–1145.
[25] Soille, P., Grazzini, J., 2009. Constrained connectivity and transition re- gions, in: ISMM, Springer. pp. 59–69.
[26] Xu, Y., Carlinet, E., Géraud, T., Najman, L., 2016. Hierarchical seg- mentation using tree-based shape spaces. IEEE transactions on pattern analysis and machine intelligence 39, 457–469.
