• Aucun résultat trouvé

Shape-based analysis on component-graphs for multivalued image processing

N/A
N/A
Protected

Academic year: 2021

Partager "Shape-based analysis on component-graphs for multivalued image processing"

Copied!
21
0
0

Texte intégral

(1)

HAL Id: hal-01695384

https://hal.univ-reims.fr/hal-01695384v3

Submitted on 17 Sep 2018

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Shape-based analysis on component-graphs for

multivalued image processing

Eloïse Grossiord, Benoît Naegel, Hugues Talbot, Laurent Najman, Nicolas

Passat

To cite this version:

Eloïse Grossiord, Benoît Naegel, Hugues Talbot, Laurent Najman, Nicolas Passat. Shape-based

anal-ysis on component-graphs for multivalued image processing. Mathematical Morphology - Theory and

Applications, De Gruyter 2019, 3 (1), pp.45-70. �10.1515/mathm-2019-0003�. �hal-01695384v3�

(2)

(will be inserted by the editor)

Shape-Based Analysis on Component-Graphs for Multivalued

Image Processing

´

Elo¨ıse Grossiord · Benoˆıt Naegel · Hugues Talbot · Laurent Najman ·

Nicolas Passat

the date of receipt and acceptance should be inserted later

Abstract Connected operators based on hierarchical image models have been increasingly considered for the design of efficient image segmentation and filtering tools in various application fields. Among hierarchical im-age models, component-trees represent the structure of grey-level images by considering their nested binary level-sets obtained from successive thresholds. Recently, a new notion of component-graph was introduced to extend the component-tree to any grey-level or multi-valued images. The notion of shaping was also intro-duced as a way to improve the anti-extensive filtering by considering a two-layer component-tree for grey-level image processing. In this article, we study how compo-nent-graphs (that extend the component-tree from a spectral point of view) and shapings (that extend the component-tree from a conceptual point of view) can be associated for the effective processing of multival-ued images. We provide structural and algorithmic de-velopments. Although the contributions of this article are theoretical and methodological, we also provide two illustration examples that qualitatively emphasize the potential use and usefulness of the proposed paradigms for image analysis purposes.

´

Elo¨ıse Grossiord

Institut Universitaire du Cancer (IUCT) – Oncople, Toulouse, Francer

E-mail: eloise.grossiord@gmail.com Benoˆıt Naegel

ICube, Universit´e de Strasbourg, CNRS, France

E-mail: b.naegel@unistra.fr

Hugues Talbot and Laurent Najman

LIGM, ESIEE, Universit´e Paris-Est, CNRS, France

E-mail: {hugues.talbot,laurent.najman}@esiee.fr Nicolas Passat

CReSTIC, Universit´e de Reims Champagne-Ardenne, France

E-mail: nicolas.passat@univ-reims.fr

Keywords Mathematical morphology · connected operators · component-tree · component-graph · shaping · morphological hierarchies · anti-extensive filtering · multivalued images

AMS mathematics subject classification 06 – Order, lattices, ordered algebraic structures 68 – Computer science

1 Introduction

Mathematical morphology is a well known non-linear theory of image processing [1, 2]. It was first defined on binary images, and then extended to the grey-level case [3]. Its extension to multivalued (colour, multis-pectral, label) images is an ongoing, important task, motivated by potential applications in multiple areas, such as medical imaging, remote sensing, astronomy or computer vision. Indeed, with the evolution of imaging technology, an increasing number of image modalities have become available. For instance, in medical ima-ging, distinct image modalities are used in combination to provide complementary characteristics in the body; in remote sensing, sensors are used to generate a num-ber of multispectral bands [4, 5]. There are many others. In the framework of mathematical morphology, the basic algebraic structure is the complete lattice [6], i.e. a empty set of ordered elements, whose every non-empty subset admits an infimum and a supremum. This means that the definition of morphological operators requires the definition of an ordering relation between elements to be processed. In the case of grey-level (resp. binary) images, the complete lattice is the partially ordered set of functions on R or Z (resp. on {0, 1}), equipped with the point-wise partial ordering induced

(3)

from the canonical total order on the reals or integers. In the case for multivalued images, values are not ca-nonically equipped with a total ordering due to their vectorial nature. While this does not prevent the defin-ition of a mathematically correct complete lattice struc-ture, in applications this can create difficulties [7].

Several contributions have been devoted to this spe-cific problem. A recent review can be found in [8]. Ex-cept in a small number of works (for instance in the case of label spaces [9, 10]), most contributions intend to define relevant total orderings on multivalued spaces as originally described in [11]. In particular, two main ways were explored: splitting the value space into sev-eral totally ordered ones (marginal processing), or de-fining ad hoc total order relations [7], often guided by semantic considerations (vectorial processing). This is particularly considered for handling colour images [12, 13, 14] and less frequently multi- / hyperspectral im-ages [15]: marginal ordering, conditional ordering (C-ordering, widely studied in the framework of colour morphology [7], including lexicographic ordering [16, 14]), reduced ordering (R-ordering) [4], partial order-ing (P-orderorder-ing), and more recently a combination of several of these orderings [12].

These approaches present the advantage of embed-ding multivalued images into models that simplify pro-cessing, similarly to grey-level images, in particular re-ducing algorithmic complexity. However, these simpli-fications of multivalued spaces induce some possible loss and bias of the information intrinsically carried by these, more complex but richer, partially ordered value sets.

In this article —which is an improved version of the conference paper [17]— we propose a new way to effi-ciently handle multivalued images in the framework of connected filtering. Our approach relies on the paradigm of connectivity, which models the spatial / structural links between some elementary patterns in an image. Intuitively, the notion of connectivity on a set Γ allows us to decide whether it is possible to move from a point or a subset of Γ to another, while always remaining in Γ . If this property is verified, we say that Γ is connec-ted. Several —similar, and sometimes equivalent [18]— ways can be considered to define connectivity: from the standard notions of algebraic topology; from the no-tions of paths in digital / discrete spaces [19, 20, 21]; or by using morphological definitions of connectivity [6, 22, 23, 24, 25].

In mathematical morphology, the notion of connecti-vity led to define many hierarchical data structures, de-signed to model simultaneously the spatial and spectral information of an image. These data structures induced various algorithmic approaches for image processing,

ly-ing in the family of connected operators. A brief state of the art on the notion of component-tree (the most popular morphological hierarchy data structure) and previous works about the hierarchical handling of mul-tivalued images, are proposed in Sec. 2, in order to provide the context of our contribution. The formal-ism of component-tree and its extension to multivalued imaging, namely component-graphs, is then recalled in Sec. 3.

Our main contribution is proposed in Secs. 4 and 5. We describe how the notion of shaping, which con-sists of composing several component-trees at differ-ent semantic levels, can be extended to handle both component-trees and component-graphs, thus allowing the processing of multivalued images via a hierarchical approach. In particular, we describe in Sec. 4 the ex-tension of the classical anti-extensive filtering paradigm based on component-trees; and we provide a complete algorithmic description of the way to actually imple-ment this filtering in the case of images taking their values in multiband spaces. Two application examples, provided in Sec. 6, qualitatively illustrate the method-ological interest of our hierarchical approach, for hand-ling multivalued images and for fusing multiple binary segmentations, via component-graphs and shapings.

2 Related works 2.1 Component-trees

The component-tree is a compact, lossless, hierarchi-cal representation of grey-level images. Induced by the inclusion relation between the binary components of successive level-sets, this structure models image char-acteristics in a mixed spectral-spatial space. By con-struction, the component-tree is well adapted for the development of image processing and analysis meth-ods, based on topology properties (connectivity), and aiming at extracting structures of interest with specific intensities (global / local extremal values).

From a methodological point of view, the efficiency of the component-tree relies on its low computation cost. Several efficient algorithms have been proposed for building the component-tree in quasi-linear time, in sequential [26, 27, 28] and in distributed ways [29]. A recent survey on the different computation algorithms can be found in [30]. More generally, a recent survey on partition hierarchies, including the component-trees, can be found in [31].

As basic operations on tree nodes can be interpreted in terms of processing on the image, component-trees have been involved in several applications. Practically, the proposed techniques have been designed to detect

(4)

structures of interest using information computed on the nodes. In particular, filtering and segmentation [26, 32, 25] can be easily carried out by simply selecting nodes, leading to connected operators. The versatility of the component-tree structure has also led to many other applications, such as image retrieval [33], classi-fication [34], interactive visualisation [35], or document binarisation [36].

The success of these methods relies on the develop-ment of efficient algorithmic processes for node selec-tion. Two main approaches were developed to cope with filtering and segmentation issues. The first approach consists of minimizing an energy globally defined over the tree nodes, leading to compute an optimal cut [37], interpreted as a segmentation of the image. This of-ten formulated as an optimization problem, where the space of solutions is composed of partitions from the hierarchy (in this context, a notion of braids of parti-tions [39] was introduced as a general framework for the optimization of segmentation based on hierarchical partitions). This is the basis for interactive segmenta-tion [38]. The second approach consists of determining locally the nodes that should be preserved or discarded, based on attribute values [40] stored at each node of the tree. The computed attributes are chosen according to the application context. This approach is formalized as an anti-extensive filtering framework [26, 32], recalled in Sec. 4, and constitutes the methodological basis of the present work. The subtree obtained by pruning the component-tree of the image, with respect to these at-tributes, can then be used to reconstruct a binary or grey-level result.

The main two limitations of the component-tree are (1) structural: it is heavily constrained by the topo-logical structure of the image; and (2) spectral: it is limited to grey-level (i.e. totally ordered) value images. Structural extensions of the component-tree have been proposed in [41] to deal with ordered families of con-nectivities, leading to component-hypertrees, and in [42] to handle images defined as valued directed graphs, leading to directed acyclic graphs (DAGs) structured over a tree. Spectral extensions were first considered by exploring marginal approaches for colour image hand-ling [43]. Then, actual extensions of component-trees to partially-ordered value images were pioneered in [44] and further formalized in [45]. Except in specific cases where the values are themselves hierarchically organ-ized [46], the induced data structure, namely a compo-nent-graph, is no longer a tree, but a DAG. The anti-extensive framework proposed for component-tree fil-tering remains valid in theory, but algorithmic issues have to be dealt with, both for node selection and im-age reconstruction [47, 48].

2.2 Hierarchical approaches for multivalued image processing

Connected operators are effective image processing tools in the framework of mathematical morphology. They were intensively studied for the last twenty years [1, 49, 26, 50]. In this context, operators based on hierarch-ical image models, i.e. trees, have been the object of several structural, algorithmic and methodological de-velopments [51] in order to tackle issues associated to specific application fields.

When dealing with the extension of connected oper-ators based on hierarchical image models to multivalued images, two major approaches are generally considered: hierarchies of segmentations and morphological trees.

The first rely on hierarchical clustering. Indeed, hier-archical segmentation trees aim at either growing and merging regions in a bottom-up fashion, or splitting regions in a top-down fashion. (see [52] for a recent sur-vey on hierarchical segmentations in the graph frame-work, as used in image processing.) The advantage of these methods when dealing with multivalued images is that they do not directly consider image levels but a simplifying metric during their construction process. In other words, they rely on a distance function / norm between image values: e.g., a saliency measure for hier-archical watersheds [53]; a merging order (region adja-cency graph merging using techniques such as the ir-regular pyramid [54], constrained connectivity [55]) for partition trees (binary partition trees [56], α-trees [57], quadtrees [58]); or via hyperconnections [59]. The use of these intermediate functions hides the complexity of the space, but necessarily induces a bias in the con-structed data structure.

Alternatively, morphological trees focus on the in-clusion relationship between components. These com-ponents result from thresholding operations on the im-age at every levels. A total order on the imim-age values ensures the inclusion of the level sets. Such total or-dering is then required to compute these trees. Unfor-tunately, and contrary to grey-level images, the spaces in which multivalued images take their values are not canonically equipped with total orders. Then, an order-ing on intensities that has to be decided upon. Hence, morphological trees consist of simplifying the multival-ued space of images a priori, to retrieve tractable totally ordered values, e.g. by marginal or vectorial policies [5]. This strategy makes it possible to reuse morphological trees specifically designed for grey-level images, such as the component-tree [26] and its self-dual version, the tree of shapes [60, 61]. However, this simplification of multivalued spaces induces a loss of information.

(5)

To cope with this problem, efforts have been con-ducted to extend these data structures to more com-plex spaces. Nevertheless, preserving a tree structure still requires restrictive constraints on the value space [46], or a final simplification. More specifically, for the extension of the tree of shapes to multivalued image, it was proposed to marginally compute the tree of shapes for each colour channel, and merge them into a single tree [62]. But then, the merging decision does not rely on values anymore but on properties computed in a shape-space [63].

Since a natural total order on multivariate data is usually not obvious, some approaches try to deal with the inherent natural partial order. Consequently, a true extension of such hierarchies to multivalued spaces leads to a data structure that is no longer a tree, but a DAG. This is the case for the notion of component-graph [45], that extends the component-tree. This approach uses directly the partial ordering of values and manip-ulates the underlying graph structure. The higher rich-ness and structural complexity of the component-graph, with respect to the component-tree, induces algorithmic issues when considering the classical anti-extensive fil-tering process developed in [26, 32]. This is in particular the case for handling spatial complexity [47], pruning policies and image reconstruction [48].

Recently, a new notion of shaping [64] was intro-duced as an efficient way to improve the framework of anti-extensive filtering of [26, 32], by considering a two-layer component-tree for grey-level image processing [64, 65]. In [17], we proposed to associate both notions of component-graphs and shaping for the effective pro-cessing of multivalued images, opening the way to new paradigms for connected filtering based on hierarchical representations.

3 Background notions

This article is set in the framework of vertex-valued graphs. We recall some basic notions and notations on graphs. They will allow us to describe the component-tree and the component-graph in a simple and unified formalism, and to discuss, in Secs. 4 and 5, how to carry out shaping on component-graphs to handle multival-ued images. For the sake of clarity, Sec. 3 is written in a self-contained way.

3.1 Order relations

Let Γ be a finite set of elements. Let ≤ be a (binary) relation on Γ . We say that ≤ is an order relation, and

that (Γ, ≤) is an ordered set, if ∀x, y, z, the relation ≤ satisfies the following conditions:

(i) x ≤ x (reflexivity);

(ii) (x ≤ y ∧ y ≤ z) ⇒ (x ≤ z) (transitivity); and (iii) (x ≤ y ∧ y ≤ x) ⇒ (x = y) (antisymmetry).

Moreover, we say that ≤ is a total (resp. partial) order relation, and that (Γ, ≤) is a totally (resp. partially) ordered set, if ≤ is total (resp. partial), i.e. if ∀x, y ∈ Γ, (x ≤ y ∨ y ≤ x) (resp. if ∃x, y ∈ Γ, (x 6≤ y ∧ y 6≤ x)). The word “partial” indicates that there is no guarantee that two elements can be compared

An ordered set can be modelled via its Hasse dia-gram, which depicts the covering relation. More pre-cisely, the Hasse diagram of an ordered set (Γ, ≤) is its transitive reduction, i.e. the strictly ordered set (Γ, ≺) such that for all x, y ∈ Γ , we have x ≺ y iff y covers x, i.e. x < y and there is no z ∈ Γ such that x < z < y. (In the sequel, illustrations are drawn so that elements are placed above the elements they cover.) The resulting diagram provides a compact and lossless description of the order relation ≤.

3.2 Vertex-valued graphs

We define a graph G as a couple (Γ, a), where Γ is a nonempty finite set, and a is a binary relation on Γ . The elements of Γ are called vertices or points. If two vertices x, y of Γ satisfy x a y, we say that they are adjacent; any such couple (x, y) is called an edge. A subgraph G0 of G is a graph (Γ0, a) such that Γ0 is a subset of Γ , equipped with the restriction of a to Γ0.

In this work, we consider irreflexive graphs, i.e. we never have x a x. These irreflexive graphs are further-more non-directed graphs, i.e. x a y ⇔ y a x; the edges (x, y) and (y, x) are then the same.

In G, a path between two vertices x and y is defined as a sequence of distinct vertices of G from x to y such that any two successive vertices are adjacent. In this case, we say that x and y are connected in G. If this path exists and is unique for any two vertices of the graph, then the graph is a tree.

We say that G is connected if any two vertices of G are connected. The connected components of G are the maximal sets of vertices that can be linked by a path. The set of all these connected components is noted C[G]; it is a partition of Γ (i.e. a set P of nonempty disjoint subsets of Γ whose union is Γ ).

Let F : Γ → V be a function such that V is a nonempty set canonically equipped with an order re-lation ≤. The triple (G, V, F) is called a vertex-valued graph (or valued graph, for brief). We now define the

(6)

notions of component-tree and component-graph, based on this notion of valued graph.

3.3 Component-tree

Let (G, V, F) be a valued graph. In the sequel of this section, we assume that ≤ is a total order on V, and that G is connected, i.e. C[G] = {Γ } contains a unique connected component. Since Γ is finite, so is the set F (Γ ) = {F (x) | x ∈ Γ } ⊆ V. Without loss of general-ity, we can assume that V = F(Γ ) and is then finite. In particular, (V, ≤) admits a minimum, noted ⊥.

For any v ∈ V, we define the threshold set Γv by

Γv = {x ∈ Γ | v ≤ F (x)} (1)

Any such threshold set Γv induces a subgraph Gv =

(Γv, a) of G. For any v, v0∈ V we have v ≤ v0⇔ Γv0 ⊆

Γv. In addition, for any connected component Xv0 of

C[Gv0], there exists a unique connected component Xv

of C[Gv] such that Xv0 ⊆ Xv.

We note Ψ the set of all the connected components of the subgraphs Gv obtained by successive

threshold-ings of G

Ψ = [

v∈V

C[Gv] (2)

The component-tree [26] of (G, V, F), noted CT, is the Hasse diagram of the partially ordered set (Ψ, ⊆). We can observe that X ∈ Ψ can correspond to several connected components in distinct threshold sets Γv ⊆

Γ for successive values v ∈ V. Then, each X ∈ Ψ is intrinsically associated in CT to a value l(X), defined as the maximal value of V which generates this connected component by thresholding of F .

As suggested by its denomination, the component-tree has a component-tree structure. Its vertices are also called nodes. Among them, the largest component is the max-imum for the Hasse diagram, namely the set Γ , ob-tained as the unique connected component of G = G⊥;

it is the root of the tree.

On the opposite side, the leaves are the minimal elements of the Hasse diagram, i.e. the nodes of Ψ that do not strictly include any other nodes (see Fig. 1). The order relation ≤ between nodes defines a parenthood relationship: a node X1 ∈ Ψ is the parent of a node

X2∈ Ψ if X2⊂ X1and if there is no other node X3∈ Ψ

such that X2⊂ X3⊂ X1. In that case, we also say that

X2is a child of X1.

For image processing purposes, each node of CT generally stores a value: either an energy (for global optimization) or an attribute (for local selection); this value is most often real. In both cases, this valuation

(a) (G, V, F) A B C D,E K L M N F,G H,I,J O,P (b) CT A (c) Γ0 D H B D (d) Γ1 D I C F E (e) Γ2 J C G K O (f) Γ3 N C LM P (g) Γ4

Fig. 1 (a) A grey-level image, viewed as a valued graph (G, V, F), where V = [[0, 4]] ⊂ Z (from 0 in black; to 4 in white) equipped with the canonical order relation ≤. (c–g)

Thresholded sets Γv⊆ Γ (in white) for v varying from 0 to 4.

(b) The component-tree CT associated to (G, V, F). The let-ters (A–P) in nodes correspond to the associated connected components in (c–g).

is modelled by a function V : Ψ → R. In other words, such enriched component-tree can be itself interpreted as a valued graph (CT, R, V).

3.4 Component-graph

Let (G, V, F) be a valued graph. We still assume that V = F (Γ ) is finite and that (V, ≤) admits a minimum ⊥. The graph G also remains connected, but from now on the order relation ≤ on V need not be total.

We extend the notion of connected component in the following way. Let X ∈ C[Gv] be a connected

com-ponent of the threshold set Γv inducing Gv at value v.

(Contrary to totally ordered sets, there may exist sev-eral values vi ≤ v (i ∈ N) such that X ⊆ Xi ∈ C[Gv i]

while viis a maximal value lower than v.) We define the

couple K = (X, v) as a valued connected component. We note Θ the set of all the valued connected com-ponents of G, with respect to its successive thresholds, defined as

Θ = [

v∈V

C[Gv] × {v} (3)

From the order relation ≤ defined on V, and the inclusion relation ⊆, we define the order relation E on the valued connected components of Θ as

(X1, v1) E (X2, v2) ⇔ (X1⊂ X2)∨(X1= X2∧v2≤ v1)

(7)

This order1, which intuitively mixes the inclusion and the value orders between connected components in a lexicographic way, can be considered as an extension of the inclusion relation to valued connected components. The component-graph [45] CG associated to the val-ued graph (G, V, F) is the Hasse diagram of the par-tially ordered set (Θ, E). It does not necessarily have a tree structure; indeed several paths may exist between two nodes. This derives from the fact that a node can be the child of several parents (and not only one, as in a tree). When a node is the child of two parent nodes that are not comparable in values, either these parent nodes meet without inclusion, i.e. they are not mutually included, or one of them is included in the other.

The component-graph still has a unique greatest node that is the maximum for the Hasse diagram, namely (Γ, ⊥); it is the root of the graph. Similarly to the component-tree, the component-graph still has leaves, that are the minimal elements of the Hasse diagram.

Three variants of the component-graph exist, rely-ing on different subsets of valued connected compon-ents:

1. Θ represents all the valued connected components induced by G;

2. ˙Θ corresponds to the set of the valued connected components of maximal values considering all con-nected components. The set of nodes ˙Θ provides at least one occurrence of a valued connected com-ponent for each possible support X induced by the image, while removing those that are hidden as the value v is lower; and

3. ¨Θ gathers the valued connected components which are generators of F , i.e. those that actually contrib-ute to the definition of the image support.

˙

Θ = {(X, v) ∈ Θ | ∀(X, v0) ∈ Θ, v 6< v0} (5) ¨

Θ = {(X, v) ∈ Θ | ∃x ∈ X, v = F (x)} (6)

Based on these definitions, we observe that ¨

Θ ⊆ ˙Θ ⊆ Θ (7)

The Θ- (resp. ˙Θ-, resp. ¨Θ-) component-graph of F is the Hasse diagram of the partially ordered set (Θ, E) (resp. ( ˙Θ, E), resp. ( ¨Θ, E)).

1 Practically, when ≤ is a total order, the component-graph

and the component-tree are isomorphic [45]. Consequently, it would make sense to also consider the valued connected com-ponents and the order E for building the component-tree, as the threshold value that leads to the generation of a connected component is useful for image modelling and reconstruction; see Eq. (10).

The three variants of component-graphs present verse relationships between computational cost and in-formation richness. The set Θ, that models all the val-ued connected components in the image, is the most informative but also the most costly. The set ˙Θ, which gathers the valued connected components with max-imal level, is intermediate in terms of both cost and information. Finally the set ¨Θ, that is reduced to the minimal set of valued connected components needed to define the image support, is the least costly and inform-ative. The relevance of each component-graph directly depends on the targeted image processing application. For instance, when (V, ≤) is a hierarchy of concepts, e.g. an ontology, it is mandatory to compute all the nodes for a complete description of the semantics of the image [46]; then the Θ-component-graph is relevant. By con-trast, when performing anti-extensive filtering on col-our images, it is important to avoid the appearance, in the result, of colours non-present in the input data [47]; then the ¨Θ-component-graph is relevant.

An example of multivalued image F and its asso-ciated value set V are provided in Fig. 2(a) and (b). Fig. 2(c–p) depicts the various valued connected com-ponents obtained from this image. More precisely, the support X of a valued connected component is repres-ented in white in each subfigure, while v is given by the value at which the image has been thresholded in the subfigure. The three variants of component-graphs are illustrated in Fig. 2(q–s).

The component-graph is a relevant extension of the component-tree, as (i) both data structures are com-pliant for totally ordered sets (V, ≤), hence compatible for grey-level images, and (ii) the component-graph sat-isfies the image (de)composition model associated to component-tree, defined later in Eq. (10). In addition, similarly to component-trees, an attribute value can be stored in each node of the component-graph to char-acterize the corresponding component. The local node selection based on attributes can lead to filtering or segmentation strategies. This valuation can also be in-terpreted as a function A : Θ → R. Then, such enriched component-graph is also interpreted as a valued graph (CG, R, A).

4 Shape-space analysis of multivalued images: Theory

A (discrete) image I is a mapping from a finite spatial domain Ω (the image support, i.e. the set of its pixels / voxels) to a value space V possibly equipped with an order relation ≤. For any x ∈ Ω, I(x) ∈ V is the value

(8)

(a) (G, V, F) (0,3) (1,3) (2,2) (1,2) (2,1) (1,1) (1,0) (0,2) (2,3) (3,2) (3,1) (3,0) (2,0) (0,1) (0,0) (b) (V, ≺) B F (c) Γ(0,1) C E (d) Γ(1,0) D K (e) Γ(0,2) I L (f) Γ(1,1) J P R (g) Γ(2,0) X H Q (h) Γ(0,3) N O (i) Γ(1,2) M U W (j) Γ(2,1) G (k) Γ(3,1) X S V (l) Γ(1,3) T Y Z (m) Γ(2,2) AA (n) Γ(3,1) X X AB (o) Γ(2,3) AC (p) Γ(3,2) (q) Θ-CG (r) ˙Θ-CG (s) ¨Θ-CG

Fig. 2 (a) A multivalued image, viewed as a valued graph (G, V, F), where F : Γ → V and V = {(0, 1), (1, 0), (0, 2), (1, 1), (2, 0), (0, 3), (1, 2), (2, 1), (3, 0), (1, 3), (2, 2), (3, 1), (2, 3), (3, 2)}. (b) The Hasse diagram of the ordered set (V, ≤). For

the sake of readability, each value of V is associated to an arbitrary colour. (c–p) Threshold sets Γvfor v ∈ V. (q–s) The Θ, ˙Θ,

¨

Θ-component-graphs of F . The letters (A–AC) in nodes correspond to the associated connected components in (c–p). We can distinguish different types of valued connected components: they are either “completely visible” (e.g. the salmon X of value (2, 3)), or “partially visible” (e.g. the green G of value (3, 1)), or “totally hidden” (e.g. the brown B of value (0, 1)). Those that

are either partially or totally visible participate to the formation of the image support and then belong to Θ, ˙Θ, ¨Θ. Those that

are invisible cannot belong to ¨Θ but do belong to Θ. Among this set, the valued connected components that also belong to ˙Θ

(9)

of I at x: I : Ω → V x 7→ I(x) = v (8)

Various choices are available for V, such as V = Rn or

V = Zn. The case of n > 1 corresponds to images with several bands of values; we then have V = V1×. . .×Vn,

and this Cartesian product induces a complete lattice. Each mapping Ii : Ω → Vi is called a band of the

multivalued image, and v is a n-dimensional vector. In order to develop morphological hierarchies, such as component-trees or component-graphs, it is required to know the order ≤ on V. If (V, ≤) is a totally (resp. partially) ordered set, we say that I is a grey-level (resp. multivalued) image.

For any X ⊆ Ω and any v ∈ V, we define the cylin-der function C(X,v) of support X and value v, as:

C(X,v) : Ω → V x 7→ v if x ∈ X ⊥ otherwise (9)

In addition, to develop connected operators, it is necessary to handle the structure of Ω, i.e. to know the adjacency between its points, leading to a graph S. An image is then modelled as a valued graph (S, V, I).

4.1 Anti-extensive filtering with the component-tree The component-tree and the component-graph are im-age lossless models. Indeed, if we consider the imim-age I in its functional form, i.e. as a mapping I : Ω → V, then I can be fully recovered from the (de)composition formula of Eq. (10). More precisely, the image can be expressed as the point-wise supremum of the nodes of its associated component-tree (Ψ ) or component-graph (Θ): I = _ (X,v)∈Θ C(X,v)= _ X∈Ψ C(X,l(X)) (10)

In the framework of component-trees (i.e. for grey-level images, i.e. when ≤ is a total order), this formula leads to a well defined image for Ψ but also for any subset of nodes bΨ ⊆ Ψ . Then, it is possible to filter the image I by discarding some of the nodes of its hier-archical representation, and reconstructing a resulting image from the preserved nodes. Each point x ∈ Ω in the filtered image presents a value that is lower or equal to the initial image; the induced operators are then anti-extensive. This anti-extensive filtering scheme was formalized for grey-level images in [26, 32]. It ba-sically consists of three successive steps:

(i) construction of the component-tree CT associated to the image I;

(ii) reduction of the component-tree by selection of nodes b

Ψ ⊆ Ψ ; and

(iii) reconstruction of the result image bI ≤ I from the reduced component-tree dCT. (S, V, I) −−−−→ (CT, R, V)(i)   y   y(ii) (S, V, bI) ←−−−− (d(iii) CT, R, V| bΨ) (11)

Step (i) is carried out from a wide range of available component-tree construction methods [30], while Step (iii) is straightforward from Eq. (10).

The core of the process is Step (ii). It implies to choose a subset of nodes bΨ ⊆ Ψ . This choice is based on (i) a selection criterion, i.e. a Boolean predicate related to the valuation V : Ψ → R that indicates if a node sat-isfies a required property; and (ii) a reduction policy to determine which parts of the component-tree should be kept or removed. The nature of the valuation V guides the decision of preserving or discarding a node. If V models an increasing attribute, the removal of a node implies that of all its descendants. Contrary, if V mod-els a non-increasing attribute, some rejected nodes can have preserved descendants. In other words, the valid-ity of the predicate for a given node does not imply its validity for the rest of the branch. Several classical policies have been defined for handling such situation, including in particular the Min, Direct and Max ones [26, 32]:

– Min: a node is removed if it does not fulfill the cri-terion, or at least one of its parent node has been removed;

– Max: a node is removed if it does not fulfill the cri-terion, and all of its children nodes have been re-moved;

– Direct: a node is removed if it does not fulfill the criterion.

The Min and Max policies have a (sub)linear computa-tional cost but might lack coherence in regards to the criterion. Indeed, they might discard/preserve nodes that meet/do not meet the criterion, depending on their position in the tree. The Direct policy preserves exactly the nodes fulfilling the criterion, but it relies on an ex-haustive scanning of all the nodes in the tree. Its com-putational cost is then equal to the tree size.

4.2 Coupling shaping and component-graphs

In this section, we describe how component-graphs (that extend the component-tree from a spectral point of

(10)

view) and shaping (that extends the component-tree from a conceptual point of view) can be associated for the effective processing of multivalued images.

4.2.1 Extension of the anti-extensive filtering to component-graphs

The component-graph also satisfies the (de)composition formula classically associated to the component-tree; see Eq. (10). Indeed, an image I can be represented via the cylinder functions induced by the nodes Θ of its component-graph. In principle, we can then extend the above anti-extensive filtering to images taking their val-ues in any value space V, without the assumption that ≤ is a total order. We have to consider a component-graph instead of a component-tree. This allows us to process any image in the same framework as initially proposed in [26, 32]: (S, V, I) −−−−→(i) (CG, R, A)   y   y(ii) (S, V, bI) ←−−−− (d(iii) CG, R, A| bΘ) (12)

However, due to the more complex structure of mul-tivalued images and component-graphs, extending this framework is not straightforward. In particular, it raises two difficulties. First, as the data structure is no longer a tree, Step (ii) is now more complex. Indeed, even if pruning policies, defined for component-trees, remain consistent for component-graphs, they have to be adap-ted for dealing with non-linear bottom-up or top-down node parsing. Second, Step (iii) becomes an ill-posed problem, depending on the nature of the order ≤ and the organization of the preserved nodes bΘ. This issue is inherent to the component-graph structure. Indeed, if a node with several non-comparable parents is removed during the selection, the reconstruction becomes sub-ject to arbitrary decisions, due to non-determinism at the parent nodes intersection. In addition, structural stability may be lost as the result image can corres-pond to a set of nodes that are different from those of the initial image.

4.2.2 Shaping: anti-extensive filtering in the shape-space

The paradigm of shaping [64] proposes to extend the framework of anti-extensive filtering to non-monotonic attributes, for grey-level image processing. It consists of performing the filtering on a double layer of component-trees, i.e. on the component-tree of the component-tree of the image, that transforms any attribute into a mono-tonic one.

The first (inner) layer corresponds to the compo-nent-tree CT of the image. The paradigm of shaping is to consider this component-tree CT itself as an im-age, and to build a second (outer) layer of component-tree from it. Indeed, from a functional point of view, a component-tree can be defined as a mapping CT : Ψ → V, where points are replaced by nodes, while in-tensities correspond to attribute values. Two nodes of the component-tree are adjacent if one of them is the parent of the other. This approach is tractable only if the space of attribute values V is equipped with a total order relation, i.e. can be modelled as (a subset of) R or Z. This is the case for most attributes, in particular numerical ones. In this case, it is then possible to build a component-tree of this first tree CT.

This “tree of tree” CT0 is processed as any other component-tree and we can perform anti-extensive fil-tering with it. It is then possible to process any grey-level image in the framework initially proposed in [26, 32], by performing node selection in a data structure that is no longer defined at the image level, but at a higher semantic level. The virtue of this new tree is that the attributes computed from the nodes of CT are now increasing in CT0. This allows us to perform real-time threshold-based node selection. The overall procedure remains quasi-linear in time and space, since we only duplicate the standard component-tree anti-extensive filtering process. (S, V, I) −−−−→ (CT, R, V)(i) (i 0) −−−−→ (CT0, R, V)   y   y(ii) (S, V, bI) ←−−−− (d(iii) CT, R, V| bΨ) (iii0) ←−−−− (dCT0, R, V| bΨ0) (13) The main limitation of this framework is that it con-siders a tree as intermediate data structure, thus limit-ing its use to grey-level images.

4.2.3 From “a tree on a tree” to “a tree on a graph” The notion of valued graphs sheds light on the common structure of images, trees and component-graphs. In particular, it allows us to describe them with a simple and unified formalism. As a side effect, it em-phasises the fact that shape-space filtering does not ne-cessarily require a tree as inner layer; it can also accept a graph. The cornerstone of this work is the generaliza-tion of the initial shaping paradigm. It can be used not only do build a “tree on a tree” but also a “tree on a graph”. This simple idea, summarized by Diagram (14),

(11)

allows us, in theory, to process any image via a shape-based filtering. (S, V, I) −−−−→(i) (CG, R, A) (i 0) −−−−→ (CT, R, A)   y   y(ii) (S, V, bI) ←−−−− (d(iii) CG, R, A| bΘ) (iii 0) ←−−−− (dCT, R, A| bΨ) (14) Based on the above remarks, this approach has the following virtues:

– it avoids the complex selection of nodes directly in the component-graph, since this task is indirectly carried out on the outer-layer component-tree; – it extends the initial shaping approach beyond

grey-level images to multivalued images;

– it inherits the good properties of shape-space filter-ing from increasfilter-ing criteria, among which real time and interactive node selection at higher semantic level.

Nevertheless, behind this simple idea, and its intrinsic advantages, some algorithmic issues have to be tackled, in particular for the two reconstruction steps (iii), from the component-tree to the component-graph and then to the image. In Sec. 5, we propose some algorithmic solutions to these issues.

5 Shape-space analysis of multivalued images: Algorithmics

We now provide an algorithmic discussion about each step of the filtering framework depicted in Diagram (14), for handling multivalued images in the shape-space. This algorithmic discussion is provided under the as-sumption that the considered multivalued images are multiband data, i.e. the set of values V is composed of k spectral bands Vi, each equipped with a total order. In

particular, we consider the canonical partial order ≤ on V defined by (vi)ki=1 ≤ (wi)ki=1 ⇔ ∀i ∈ [[1, k]], vi ≤ wi.

This hypothesis is motivated by the high frequency of such images in current image processing applications, which justifies to study them in priority.

5.1 Component-graph construction

Three variants of component-graphs were introduced in [45] (see Sec. 3.4), in particular to simplify CG by considering smaller subsets of Θ. In Step (i) of our framework, that builds the first layer component-graph CG from the multivalued image (S, V, I), we will often

choose to consider the lightest version ( ¨Θ) of component-graph (Fig. 2(s)), i.e. the one that represents only the nodes which actually contribute to the construction of the image according to Eq. (10). In the case of ¨Θ, we use the component-graph construction algorithm proposed in [47]. (However, an algorithm was recently proposed in [66, 67] for building, more generally, the component-graphs Θ; then, the proposed framework is, of course, also tractable for such “complete” component-graphs.) For the sake of readability, we will now note ¨Θ as Θ.

The choice of working with the ¨Θ-component-graphs motivated by several facts. First, we are considering multiband images, that generally correspond to “real” images, where values at each point have a physical mean-ing (by contrast with semantic-content images considered for instance in [46]). The ¨Θ-component-graph is the only which guarantees that no new value will be intro-duced via a node initially hidden in the graph; this is a reasonable property in this context. Experimentally, it was observed in [47] that such component-graph was indeed relevant for filtering (denoising, simplification) purposes.

Second, from a space complexity point of view, this component-graph is more efficient than the other two. Indeed, by construction, each node is visible in the mod-elled image. This means that at least one point of the image is directly and uniquely modelled by one node of the component-graph. A corollary of this property is that the number of nodes within ¨Θ is bounded by the number of points of the image; in other words, the space complexity of the graph is (in the worst case) lin-ear with respect to the image size. Since no intermediate superlinear data structure is required for its construc-tion, which may lead to extra computational cost, the building of this component-graph also presents a linear time complexity.

5.2 Component-graph valuation

At this stage, an attribute can be associated to each node of Θ, in the component-graph CG, to retrieve a structure of valued graph, namely (CG, R, A) in Dia-gram (14).

We consider here an attribute taking its values in R, namely a set where all values are comparable. While alternative choices are possible (see Sec. 7), we assume that a valuation A : Θ → R contains enough inform-ation to accurately filter the nodes, while leading to a valued graph that authorizes the building of a tree structure as second layer.

The criteria possibly modelled by A for each node K = (X, v) ∈ Θ can depend on:

(12)

(1) spectral properties, relying on image intensities, tex-ture, etc. (i.e. the information stored in the v part, e.g. intensity mean, extrema): then, we generally have A : V → R;

(2) geometric properties, relying on spatial information of the image (i.e. the information stored in the X part, e.g. compacity, flatness): then, we generally have A : 2Ω → R;

(3) structural properties, relying on the relationship be-tween the node and its neighbourhood within the component-graph. By contrast with spectral and geo-metric properties, that describe the node locally (i.e. on its own support), structural properties describe the node with respect to its environment. For in-stance, relevant information can be related to branch length, position within the graph, number of chil-dren / parent nodes, etc. The definition of such properties then requires to observe the component-graph in a local or semi-global way (generally around the node of interest);

or any combination of some of these three classes. In the case of multiple criteria, standard linear combin-ations can be considered (mean, weighed functions), but also nonlinear combinations (min, max, median). The main difficulty then consists of making the criteria values relevantly comparable; this generally requires a normalization step.

Contrary to the other kinds of component-graphs, the chosen version of CG is relatively light. As a con-sequence, a criterion of type (3) would be weakly relev-ant, as all “internal” nodes of the graph are not mod-elled, thus making the graph structurally sparse. In-deed, the elimination of nodes from the richer variants of CG may hide information and introduce a bias in the graph structure, with respect to this specific type of attributes.

Then, only geometric criteria (type (2)) are con-sidered here, for building the component-tree of the second layer. In particular, this choice is coherent with the paradigm of shaping —initially designed to focus on higher level semantics— and also motivated by the fact that the spectral handling (criteria of type (1)) of the image is intrinsically carried out at the first layer of the structure. Indeed, the component-graph already models the order between values in V via its structure. The spectral handling then occurs both before (during the component-graph construction) or after the shaping stage (during the image reconstruction).

Another possibility is to define on this outer compo-nent-tree a second attribute V from which will be pro-cessed the tree pruning. In order to preserve the good properties of such filtering, it is essential that this new attribute V keep the same behaviour as the first

attrib-ute A, i.e. an increasing or decreasing evolution along the tree. An example can be given by considering as attribute V the gap between the attribute value Ak

of the node K and the values Al of the leaves of its

branch. Such criterion remains increasing, thus author-izing a (relative) thresholding approach, similar to the first strategy but with a fine behaviour.

5.3 Component-tree construction and pruning

From the valued graph (CG, R, A) associated to the component-graph, a shape-based component-tree can now be defined. This tree is the data structure that will be considered for the pruning process (Step (ii)).

In practice, the structures of interest can be of two kinds, depending on the adopted processing paradigm: they are either structures to be preserved in the im-age (for segmentation) or structures to be removed (for filtering or denoising).

Two basic policies can be considered to build the component-tree CT, guided by the nature of the attrib-ute A, and more specifically by the correlation between the extremal values of A and the structures of interest. When these structures of interest correspond to the low-est values of A, a min-tree (Fig. 3(b)) is chosen, i.e. the root has the highest value, while the leaves have the lowest; when the structures of interest correspond to the highest values of A, a dual max-tree (Fig. 3(c)) is adopted. The inversion of the attribute allows one to switch from a representation to the other.

Once the min-/max-tree has been built, the prun-ing process is carried out in a way that depends on the kind of processing paradigm. In the first case (seg-mentation), i.e. when structures of interest have to be preserved, relevant nodes are selected by preserving the distal parts, i.e. the branches of the tree (Fig. 3(d)). In the second case (filtering), i.e. when structures of in-terest have to be eliminated, the pruning consists of removing those distal parts of the tree, and preserving the proximal part (Fig. 3(e)). In each case, the principle is to compute the most discriminative cut in the tree, and to preserve nodes located either below or above this cut, according to the chosen paradigm.

Practically, each node Y ∈ Ψ of the component-tree CT is a connected component gathering nodes of a subgraph of CG, for a given threshold value with re-spect to A. This threshold value then constitutes the valuation of this node. Following the above classifica-tion of properties (spectral, geometric, structural, see Sec. 5.2), this new valuation A —that was obtained from a valuation based on geometric properties— is now a valuation based on spectral properties in the

(13)

con-M N X AB AC H G A 6 1 3 1 5 1 2 5 (a) CG {N-AC} X {AB} {H} {G} {A} 1 2 3 4 5 6 {X-M} 6 5 3 2 1 1 (b) Ψmin {N-AC} {AB} {H} {A} 6 5 4 3 2 1 6 5 3 2 {X-M-G} 1 (c) Ψmax 1 2 3 4 5 6 6 2 1 {A} {N-AC} 5 {H} 3 {AB} X {X-M} 1{G} (d) Ψleaves min N-AC X AB H G A 1 2 3 4 5 6 X-M 6 5 3 2 1 1 (e) Ψroot min

Fig. 3 Component-tree construction and pruning. (a) The component-graph valued with attribute A. (b,c) The min-tree

Ψmin and max-tree Ψmaxbuilt from Θ. In these representations, a node of the component-tree may correspond to a node of

the component-graph (e.g. leaves such as G in Ψmin) or a set of nodes of the component-graph (e.g. {A} gathers all the nodes

of {A,G,H,M,N,X,AB,AC} in a same node in Ψmin). The red line represents a cut in the tree. Depending on the policy, the

(14)

text of shaping, since the attribute of the component-graph becomes the valuation of the component-tree, in the shape-space. In addition, it defines a monotonic cri-terion, allowing for an easy selection by thresholding, and avoiding the use of any specific (e.g. Min, Max) pruning policies.

5.4 Component-graph filtering

A “standard” component-tree —defined from a grey-level image— contains nodes which represent connec-ted components of points of the image, obtained at a given threshold value. By contrast, the component-tree CT, defined at the outer layer of the shape-space model —computed from the valued graph (CG, R, A)— con-tains nodes that are connected components of Θ, which are themselves connected components of Ω. Such nodes Y ∈ Ψ are thus defined as {Ki= (Xi, vi)}ki=1 ⊆ Θ, with

k ≥ 1.

Once the pruning of the component-tree has reduced the number of nodes Y preserved in bΨ , two main ways can be considered to determine which nodes Ki ∈ Y

have to be preserved in the resulting pruned component-graph dCG, i.e. which nodes should form bΘ; see Fig. 4.

A first way is to keep all the Ki contributing to the

preserved Y , since they all provide spectral information in the image.

But since spectral information is already taken into consideration within the inner layer component-graph, a second way is to exclusively consider spatial inform-ation carried by the preserved nodes Y . Indeed, each node Ki∈ Y is either included in another node Kj∈ Y ,

or is a maximal element in Y with respect to the E (and the ⊆) relation. When dealing with geometric criteria, only these latter —maximal— nodes, that contribute to define the support Sk

i=1Xi of Y in Ω, are of interest.

In other words, if Y is preserved in bΨ , only the nodes Ki = (Xi, vi) of Y such that Xi is a maximal subset

of Ω within Y may be preserved (both spatially and spectrally) in the filtered image. We note bY ⊆ Y the subset of Y formed by such nodes.

Then, the other nodes of Y are not taken into ac-count and are lost in the representation. Practically, this is not a problem in general. Indeed, most2 of the

2 Erratum: In [17, p. 453], it was written that “any node

K ∈ Θ belongs to bY for at least one Y ∈ Ψ ”. This is not

always true. A trivial counter-example is the case where Θ has at least 2 nodes, while the same value is given by A to each K ∈ Θ, thus leading to a degenerated component-tree CT composed by a single node. However, for “reasonable” valuation functions A, and in particular those taking their values in R, it is quite probable that most nodes K ∈ Θ

belong to bY for at least one Y ∈ Ψ .

A C B A C B AUB

(a)

(b)

(c)

contribution spa tial

Fig. 4 Component-graph filtering. (a) A node Y from the

outer layer component-tree preserved in bΨ after tree pruning.

This node Y is composed of three nodes of the component-graph CG: A, B and C such that C ⊂ A and C ⊂ B. Two choices can be made for the filtering of Y : either (b) all nodes

(A, B and C) are preserved in bY as they all contribute to

the spectral values of the component; or (c) only A and B are

kept in bY when C is eliminated, since A∪B defines the spatial

support of Y while C does not participate to the definition of the boundaries of Y .

nodes K ∈ Θ belong to bY for at least one Y ∈ Ψ . In other words, even by preserving a strict part of the nodes within the elected Y , each node K still has a chance to be finally preserved, thus minimizing the risks of erroneous removals.

The main difference between the initially proposed shaping paradigm (“a tree on a tree”) and the present one (“a tree on a graph”) is that the first defines the support of any Y ∈ Ψ from a single node K ∈ Θ, while the second can require several nodes of Θ since values of V may be non-comparable.

5.5 Image filtering

Our data structure considered for processing an image is composed of two layers of component-graph / tree. Then, the final filtering of the image has to go success-ively through these two layers, leading to two filtering steps.

(1) The first step is a temporary reconstruction at the component-tree level. It consists of reconstructing regions of the image corresponding to each reduced node bY , associated to each node Y ∈ bΨ .

(2) A given node K ∈ Θ may belong to cYj, for several

nodes Yj∈ bΨ . Then, the second step —which leads

to the final reconstruction of the image— handles the conflicting intersections between those regions Yj.

(15)

We recall that we deal with multivalued images, and we assume that the value space (V, ≤) is structured as a complete lattice. As a consequence, any subset of values admits an infimum and a supremum; this assumption is used hereafter for reconstruction purpose.

For the first step, various approaches can be con-sidered to reconstruct image regions associated to the nodes bY of the reduced component-tree. The first, strai-ghtforward possibility is to preserve all values vi

asso-ciated to each node Ki of the reduced node bY ; and

delegate the real image valuation to the second step. We do not describe how to do that in the sequel of the paper. Instead, we focus on the second possibility that consists of assigning a unique value v to the whole re-duced node bY , thus creating a flat zone of value v, over the whole set of nodes (Ki, vi) ∈ bY . In this case, we

mainly have two options for defining v:

– v can be set as the supremum of all the vi of each

node Ki = (Xi, vi) ∈ bY . This policy leads to the

loss of the anti-extensivity property of the designed filters. Indeed, the choice of the supremum creates connected components possibly presenting a high value that did not exist in the initial set of con-nected components of the original image; or – v can be defined as the infimum of all the viof each

node Ki ∈ bY . This policy ensures to preserve the

property of anti-extensivity of the subsequent filters. This strategy is justified by the fact that the node Y has been preserved with respect to a geometrical attribute, computed for the union of all supports Xi of the Ki. In such conditions, the least common

value associated to all these nodes can be relevantly considered. However, this strategy tends to “flatten” the intensities in the image and create connected components valued with intensities that are lower or equal to that of the existing support.

The second step consists of handling the conflicts between the values assigned to nodes K belonging to several nodes Yj∈ bΨ .

(i) In the case where v was defined by a supremum paradigm:

– the value finally assigned to the node K can be defined as the supremum of all the conflicted val-ues of K. This may create connected compon-ents valued with intensities higher than in the original image, due to the composition of two successive supremum operators. The reconstruc-tion of the filtered image can then be formalized as: b I∨∨= _ Y ∈ bΨ C(S (X,v)∈ bYX, W (X,v)∈ bYv) (15)

– alternatively, the value finally assigned to the node K can be defined as the infimum of all the values in conflict. This policy tends to attenuate the effects of the initial supremum operator. In particular, the result will be closer to the initial image than with the above policy. However, it does not ensure to retrieve anti-extensivity. The reconstruction of the filtered image can be form-alized as: b I∧∨= ^ Y ∈ bΨ C(S (X,v)∈ bYX, W (X,v)∈ bYv) (16)

(ii) In the case where v was defined by an infimum paradigm:

– the value finally assigned to the node K can be defined as the supremum of all the values in con-flict. This policy is justified by the fact that a node Y ∈ Ψ , defined as the union of several nodes of Θ, should not lose its geometry in the filtered image. It allows us to offset the flatten-ing of intensities (due to the infimum policy at step one) and to come up with intensities closer to those of the original support. The reconstruc-tion of the filtered image can be formalized as:

b I∨∧= _ Y ∈ bΨ C(S (X,v)∈ bYX, V (X,v)∈ bYv) (17)

– alternatively, the value assigned to the node K can be defined as the infimum of the values in conflict. This policy will tend to completely flat-ten the inflat-tensity in the image; the resulting sup-port may be spectrally far from the real intensit-ies. Besides, the choice of the infimum may result in the loss of the geometry of individual nodes Y due to the building of large flat components. The reconstruction of the filtered image can be formalized as: b I∧∧= ^ Y ∈ bΨ C(S (X,v)∈ bYX, V (X,v)∈ bYv) (18)

In practice, there is no universally good strategy, within the four proposed above; the relevance of an ap-proach actually depends on the purpose. However, it is important to notice that the first two, based on an initial supremum operator do guarantee neither extens-ivity nor anti-extensextens-ivity. Indeed, we have:

b

I∨∨≥ bI∧∨ (19)

but we can neither ensure bI∨∨, bI∧∨ ≥ I nor ≤ I.

By contrast, the last two strategies do guarantee anti-extensivity:

(16)

(a) (b) (c)

Fig. 5 Coupled CT (a) and PET (b) images. (b) Ground-truth of the lesions, in purple. (c) Multivalued shape-based processing

from (a+b), visualized in the PET value space. (a,b) Courtesy M. Meignan, Hˆopitaux Universitaires Henri-Mondor, Lymphoma

Academic Research Organization, Cr´eteil, France.

In particular, bI∨∧ is the closer result image that

re-mains below I.

These various strategies, although presented in the general case of handling all bands of a multivalued im-age, can be restricted to a subset of bands. In the case where only one band is considered, the reconstructed image is a grey-level one, and the supremum and in-fimum on V considered above are simply replaced by the maximum and minimum in the considered band.

6 Illustrations

We now present two application cases involving shape-space analysis on component-graphs. Both deal with image segmentation, but each in a different way. The first application case deals with segmentation of mul-timodal images, and the component-graph is then used for modeling these modalities in a unified way. The second application case deals with segmentation fusion, and the component-graph is then used for modeling the various segmentation results as a richer alternative to fuzzy paradigms.

In both experiments, the ¨Θ-component-graph is used. On the one hand, it was shown in [45] that the space cost of this component-graph is O(|Ω|), i.e. linear (and possibly sublinear) with respect to the size of the im-age support. On the other hand, it was observed in [47]

that the construction of this component-graph has a computational cost O(|Ω|2), experimentally corrected to O(|Ω|1,5).

6.1 Multimodal medical image segmentation

We consider a first application example in the con-text of multimodal medical imaging. (This application example was initially provided in [17]; it is presented here with more details.) The image is a function tak-ing its values in Z × N with two bands correspond-ing each to a given imagcorrespond-ing modality, namely morpho-logical X-ray Computed Tomography (CT) and func-tional Positron Emission Tomography (PET). The seg-mentation of such images is designed to emphasise tu-mours based on their shape and metabolic activity.

Our purpose is not to prove that our results reach the state of the art for PET-CT segmentation (see [68, 69] for recent surveys on this active research topic). We aim to qualitatively emphasise that the proposed frame-work is versatile enough for encompassing a wide range of applications. In particular, we aim to show that it opens the way to alternative possibilities of image pro-cessing of multivalued images via connected operators. Positron Emission Tomography visualizes metabolic activity characterized by the intensity of an injected ra-diotracer (here,18F-FDG). It is routinely used in

(17)

can-cer imaging for diagnosis and characterization of ma-lignant tissues, corresponding to FDG hyperfixations (black regions surimposed in purple in Fig. 5(b)). PET images are classically associated to X-ray computed tomography images (Fig. 5(a)), for visualizing the ana-tomy.

The interpretation of PET images requires a thor-ough knowledge of the normal patterns of the FDG uptake. Indeed, FDG uptake reflects glucose metabol-ism. In particular, FDG uptakes are seen in cancerous lesions, but also in various organs such as the brain, the heart, the liver, the bladder. Then, CT images can provide complementary information by localizing FDG uptakes corresponding to physiological sites. Conseque-ntly, it is relevant to process them as a unique bivalued image in order to more accurately extract the lesions and their activity. The idea is to highlight tumours, i.e. maximal intensities in the PET image (represented as black regions) and discriminate those corresponding to physiological uptakes, using the CT information.

In contrast to PET images, where the canonical or-der ≤ on N = VP ET captures the semantics of

meta-bolic activity, this order is partly meaningless on Z with respect to the Hounsfield scale in CT. Consequently, we apply a non-injective mapping mH : Z → N on

CT to the regions known as physiological uptakes on the PET. More precisely, we associate the lowest val-ues of VCT = N to tissues of extremal Hounsfield

in-tensities. Such mapping can be defined by mH(v) =

max{M − |v − K|, 0} with M > 0 and K ∈ [100, 300], thus providing the highest signal M for soft tissues (with Hounsfield values around [100, 300]), and lowest values for non-relevant tissues (for instance fat and wa-ter, around [−100, 0] or bones around [700, 1 000]). The order ≤ on VCT for the resulting image associates the

least values in the CT data to tissues which are more likely to induce false positives in PET. The value space is subsampled to 256 values for both PET and CT, lead-ing to a space of V = VCT × VP ET of 2562 = 65 536

distinct values.

In the case of adults lymphomas, lymphatic lesions in the thorax are characterized as compact masses. Con-sequently, the criterion A : Θ → R chosen for lesions filtering is the compactness factor [65]. The outer layer component-tree is then built as a max-tree with respect to the value set R of A. Our purpose is here to select the nodes of highest values, i.e. in the distal parts of the branches, in a segmentation paradigm. The processed image is finally reconstructed following the policy pro-posed in Eq. (17).

For visualization purposes, the results are only de-picted in the VP ET value band (VCT does not present

any interest in this context), see Fig. 5(c). We observe

a satisfactory spatial accuracy between the detection of lesions (Fig. 5(c)) and the ground-truth (purple areas in Fig. 5(b)). In particular, all lymphatic lesions are de-tected, whereas potential false positives were discarded (e.g. the bladder physiological uptake).

6.2 Extended fuzzy framework for segmentation fusion We now consider a second application case, still in the context of segmentation. However, it differs from the previous as the notion of component-graph is no longer used for modeling the structure of the value space, but for handling the fusion of many segmentation results from a spatial point of view.

Let us suppose that, for a given image I defined on a support Ω, we have k ≥ 2 segmentation results for a same structure of interest. These k segmentation results are binary sets Si ⊆ Ω such that for any i ∈

[[1, k]], x ∈ Si means that the point x belongs to the

i-th segmentation result.

Our purpose is to fuse these k segmentation results. We define the function

S : Ω → {0, 1}k x 7→ (xi)ki=1 (21) where xi= 1 (resp. 0) if x ∈ Si (resp. x /∈ Si). In other

words, S provides, for each point of the image support, a k-uple of binary values that express the result of the k segmentation results at this point.

A classical way to carry out segmentation fusion is to define a fuzzy function F : Ω → [0, 1] such that for any x ∈ Ω, we have F (x) = 1

kkS(x)k1, where k.k1 is

the `1norm. In other words, we compute the

probabil-ity that x belongs to the fused segmentation result, by assuming that all the k segmentation results are equi-probable. This function F is a grey-level image, and it is possible to build a standard component-tree for allow-ing the user to relevantly carry out local thresholdallow-ing on this fuzzy fusion map. However, the function F “flat-tens” the vectorial information carried by the k-uples of binary values.

For improving this fuzzy framework for segmenta-tion fusion, we consider the set {0, 1}k as the value set

V, with a partial order ≤ defined as

(xi)ki=1≤ (yi)ki=1⇔ ∀i ∈ [[1, k]], xi≤ yi (22)

Then, (V, ≤) is isomorphic to the power-set of a set of k elements endowed with the inclusion relation. In particular, (V, ≤) is a complete lattice.

We can then build the component-graph G associ-ated to the “image” S. Each node K = (X, (vi)ki=1) ∈ Θ

(18)

(a) (b) (c) (d) Fig. 6 (a) Original image. (b–d) Manual segmentations, from the segmentation evaluation database [72].

corresponds to a maximal connected region X ⊆ Ω such that there exist (at least) v segmentation results Si ∈ Ω that satisfy X ⊆ Si. In particular, the node

(Ω, (0)k

i=1) is the root of G.

This component-graph of S is much richer than the component-tree of F . Indeed, it carries not only inform-ation related to the `1norm of the k-uples, but also

ad-ditional information on their composition with respect to the k segmentation results Si.

We build a second-layer component-tree on G, for actually handling it. To this end, we need a valuation function A on the nodes K = (X, (vi)ki=1) of Θ. A

relev-ant way consists of considering the mean gradient value along the border of X. Indeed, the higher this value, the most probable X is a significant standalone region within the image I. This valuation A : Θ → R is a mixed spectral/geometric attribute on Θ, as it depends on the support X of the nodes, but also on the value of the border points of X in I.

The second-layer to be built is a max-tree, and the nodes to be preserved are located on the distal part of the tree. In other words, we build a tree CTrootmax. The

pruning can then be carried out by a simple threshold-ing process along the branches of the tree. Finally, since our purpose is to carry out segmentation, reconstruc-tion policies are trivial here, and only consist of the union of the node supports.

To illustrate this strategy, we consider an image extracted from the segmentation evaluation database3

[72] consisting of grey-level (natural) images along with their manual delineations (ground-truths) from three humans. The chosen example image is provided in Fig. 6. The three manual segmentations are fused using the function S given in Eq. (21). Then, each point of S has a value v ∈ {0, 1}3

= V. Since V is endowed with the (partial) component-wise ordering ≤, extremal value of S are (0, 0, 0) (the point does not belong to any mentation) and (1, 1, 1) (the point belongs to all seg-mentations).

3 http://www.wisdom.weizmann.ac.il/~vision/Seg_

Evaluation_DB/index.html

-1:((0,0,0) a=67500 g=0) 0:((0,0,0) a=67500 g=0)

1:((0,0,1) a=23642 g=29) 4:((0,1,0) a=21159 g=56) 7:((1,0,0) a=17922 g=63) 2:((0,1,1) a=20951 g=55) 5:((1,0,1) a=17847 g=62) 6:((1,1,0) a=17662 g=64)

3:((1,1,1) a=17632 g=62)

Fig. 7 Component-graph G of the segmentation fusion image S. Each node represents a connected component of intersect-ing segmentations. In each node, the first number represents the node identifier (−1 denotes a fictitious root); then the at-tributes are: the value of the node, the area of the node and the mean gradient value of the node.

The component-graph G constructed from the im-age S is represented in Fig. 7. From G, we build the tree (max-tree) CT (Fig. 8). This component-tree is based on the successive thresholdings of the mean gradient attribute. The leaves represent the set of ex-tremal regions of G according to the mean gradient cri-terion. For instance, in Fig. 8, the leaf node at level h = 64 represents the isolated node of G at level v = (1, 1, 0). Therefore, this node represents the connected compon-ent with the highest mean gradicompon-ent on its contour, res-ulting from the intersection between the manual seg-mentations (see Fig. 6(b,c)). It can be interpreted as the optimal fusion of segmentations according to the chosen criterion. This result is illustrated by Fig. 9.

7 Conclusion

By coupling the two recently introduced notions of shap-ing and component-graph, we opened the way to the de-velopment of new connected operators based on mor-phological hierarchies, to process multivalued images. This work constitutes a first algorithmic contribution to such an approach, in the field of multivalued math-ematical morphology. Beyond encouraging results

Références

Documents relatifs

The main drawbacks of this method are the need for a textured scene, because the focus measure is based on the high frequency content of the scene, and a large number of

Using this tech- nology the aim of this study was to explore whether a PCA-based model could be applied for accurately predicting internal skeleton points, such as joint centers

processing ultrasonic axial transmission signals measured in long cortical

• Development of a deformable model segmentation in a probabilisti framework: A major problem in medi al image pro essing is the automati.. segmentation of anatomi al

This empirical study has been made on two images conventionally used in hyperspectral image processing domain: i) the Pavia image, which represents the campus of Pavia university

After reminding the AFB1 biosynthetic pathway and its internal gene regulation, we describe other genes found to interact with AFB1 synthesis according to their role in

Given the predominance of housing fabric over other types of building, to a very large degree the kind and quality of housing is what gives form to the city:

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des