Point Sets

(1)

Histopathalogy

Nicolas Loménie

(2)

Discussion Overview

• Setting up the Digital Histopathology Challenge ;

• Radiometric Filtering ;

• Geometric Filtering ;

2/48

(3)

Prewitt Filtering

3/48

Parametric and Nonparametric Recognition by Computer : An Application to Leukocyte Image Processing Judith M. S. Prewitt . Advances in Computers -

1972

(4)

Point Sets

• A methodological new trend : point reconstruction, sparse representation, points of interest, PCL library ;

• A more geometric approach to image analysis issues : to overcome radiometric instability and redundancy ;

• A re-energized theoretical topic : visual point set processing ;

4/48

The Geometric Science of Information 28-08-2013 - 30-08-2013 Paris http: // www. gsi2013. org

(5)

Mathematical Morphology

• A well-established theoretical framework : lattice theory (Gallois, abduction,

Minkovsky operators etc.)

• A global concept : image processing and reasoning as well ;

• A new theoretical topic : morphology and spatial relations over point sets

(ANR ASTRID DESCRiBE 2012-2014) ;

5/48

(6)

Digital Histopathology

• An ongoing big challenge : academic, industrial, societal ;

• A complete ground test : closed universe, no digital model, possibility of ground truth ;

• A new avenue to put together semantic filtering and image processing ;

6/48

Pathology Innovation Centre of Excellence (PICOE). Digital Histopathology : A New Frontier in Canadian Healthcare. White Paper. Jan. 2012. GE Healthcare.

(7)

MICO Project

• Objective : Automated Breast Cancer Grading

• 3-year Project

• February 2011 to mid-2014

• 3.2 million ¤

• Funded by ANR

Agence Nationale de la Recherche French National Research Agency

7/48

(8)

Project Partners

IPAL — University Joseph Fourier Image Processing, Ontologies, Reasoning

Pitié-Salpêtrière Hospital Pathologists, Expertise on breast cancer

University Pierre and Marie Curie Contextual graphs, GPU

TRIBVN User interface Agfa HealthCare Ontologies, Reasoning

Thales Semantic middleware

8/48

(9)

Outline

• Setting up the Digital Histopathology Challenge;

9/48

(10)

Whole Slide Image (WSI)

Biopsy glass slide

Ü Scan at

40X Resolution :

0.2456µm per pixel

Ü

Whole Slide Image 8 GPixels

• Multi-page TIFF image

• File size (compressed) : Usually 1 to 2 GB

• Image size (uncompressed) : 15 to 25 GB

10/48

(11)

Whole Slide Image

1.25X

Identification of tumor area

11/48

(12)

Whole Slide Image

1.25X

Identification of tumor area

11/48

(13)

Whole Slide Image

12/48

(14)

Whole Slide Image

12/48

(15)

Whole Slide Image

12/48

• International breast cancer grading system recommended by the World Health Organization

• 3 criteria :

1. Gland (acinus) formation (score 1 - 2 - 3) 2. Nuclear atypia / pleomorphism (score 1 - 2 - 3) 3. Mitosis counts (score 1 - 2 - 3)

• Final grading : Add scores for acinus formation, nuclear atypia and mitosis count.

• If total score = 3, 4 or 5Ügrade 1

• If total score = 6 or 7Ügrade 2

• If total score = 8 or 9Ügrade 3

13/48

Acinus : Any cluster of cells that resembles a many-lobed “berry”, such as a raspberry. Acinus is Latin for berry.

(23)

Nottingham Grading System

Gland Formation

Score 1 More than 75% of the whole carcinoma forms acini Score 2 10 ∼75% of the whole

carcinoma forms acini Score 3 Less than 10% of the whole

carcinoma forms acini

14/48

(24)

Nottingham Grading System

Nuclear Atypia

Score 1 Nuclei only slightly larger than benign breast epithelium (<1.5×normal area) ; minor variation in size, shape and chromatin pattern

Score 2 Nuclei distinctly enlarged (1.5∼2×

normal area), often vesicular, nucleoli visible ;

may be distinctly variable in size and shape but not always

Score 3 Markedly enlarged vesicular nuclei (>2×

normal area), nucleoli often prominent ; generally marked variation in size and shape but atypia not necessarily extreme

15/48

(25)

Mitosis

Ê Prophase : The chromatin condenses into a highly ordered structure called a chromosome.

ËMetaphase: Chromosomes align in the middle of the cell.

Ì Anaphase: Each chromatid moves to opposite poles of the cell, the opposite ends of the mitotic spindle, near the mi- crotubule organizing centers.

ÍTelophase : Two daughter nuclei form in the cell.

16/48

(26)

Nottingham Grading System

Mitosis Counts

• Scan sections to find area with most mitotic activity (often at tumor edge).

• In this area count definite mitoses in 10 consecutive fields.

• Skip fields with few carcinoma cells or obvious necrosis.

For a microscope field diameter of 58 mm (or a digitized square image 512×512µm²) :

Score 1 Score 2 Score 3

≤9 mitosis 10∼19 mitosis ≥20 mitosis

16/48

(27)

Edge-based

• J.M.S. Prewitt "Object Enhancement and Extraction" in

"Picture processing and Psychopictorics",Academic Press,1970

• Edge Detector : Prewitt filter

http://en.wikipedia.org/wiki/Prewitt_operator

• Provides the gradient of the signal : ∇I(x,y) for every radiometric pixel...

20/48

(37)

Cell Nuclei Extractor

Edge-based

21/48

(38)

Active Contours

22/48

Chan–Vese Segmentation - Pascal Getreuer - Published in Image Processing On Line on 2012–08 –08.

Active Contours Without Edges - Tony F. Chan, Member, IEEE, and Luminita A. Vese - IEEE Transactions on Image Processing - VOL. 10, NO. 2,

FEBRUARY 2001

(39)

Active Contours

22/48

FEBRUARY 2001

(40)

Active Contours

22/48

FEBRUARY 2001

(41)

Outline

• Geometric Filtering;

23/48

(42)

Geometric Processing

Point Set and Digital Histopathology ?

24/48

A Hybrid Classification Model for Digital Pathology Using Structural and Statistical - Pattern Recognition - Erdem Ozdemir and Cigdem Gunduz-Demir -

IEEE Transactions on Image Processing - VOL. 32, NO. 2, FEBRUARY 2013

(43)

Geometric Processing

Point Set and Delaunay triangulation

Del A Delaunay triangulation for a set S of points in a plane is a triangular meshDel(S) such that no point in S is inside the circumcircle of any triangle inDel(S).

24/48

(44)

Implementing the Delaunay Triangulation

We need a specific doubly linked list Data Structure called

HalfEdge composed of :

• int Origin ;

• HalfEdge *Twin ;

• Face *Adjacent ;

• HalfEdge *Next,*Last ; At the same time, we need a tree structure Del whose vertices are structures called Facecomposed of :

• HalfEdge *Edges[3] ;

• Face *Fils[3] ;

25/48

(45)

Validity check

Validity checking of edge AB : when a new triangle PAB is created : if the circumscribed circle to PAB does not contain the point Q, the edge is valid ; else we need to swap edge AB and PQ.

25/48

(46)

Implementing the Delaunay Triangulation

Algorithm 1: Delaunay Triangulation Algorithm

Require: S is a point set with n pointsPq forq from 0 to n-1 ; Require: Delis a tree ;

Require: H is a list ;

Construction of the triangle enclosing all points : creation of three dummy pointsPn,Pn+1 andPn+2with coordinates(0,0,3M),(0,3M,0 and (3M,0,0)whereM is the maximum coordinate of the points in S ; Assign this triangle to the root ofDel then initialize the listHwith the corresponding edges ;

for alli such that 0≤i ≤n−1do

Compute the leaf triangle inDel PqPrPs that contains the pointPi; Insert inDel the three trianglesPiPqPr,PiPrPs andPiPsPq and update the structuresDel andH subsequently ;

LEGALIZE(PqPr) LEGALIZE(PrPs) LEGALIZE(PsPq) end for

Remove the triangles sharing a vertex with the virtual enclosing triangle.

25/48

(47)

Implementing the Delaunay Triangulation

Algorithm 2: Procedure LEGALIZE

Require: An edgePqPr

Extract the two adjacent trianglesPqPr, letPt andPi be the two new vertices ;

if ILLEGAL(PqPr,Pi,Pt)then

Replace the two trianglesPqPrPtandPqPrPi with the two triangle PtPqPi andPiPrPt then update the structuresDel andHin accordance ; LEGALIZE(PqPt)

LEGALIZE(PrPt) end if

Algorithm 3: Function ILLEGAL

Require: An EdgePqPr and two verticesPi andPt

if Pt in contained in the circumscribe circlePiPqPr then Return 1

else Return 0 end if

25/48

(48)

Shape Definition

α-objects

(a) 2D point setS; (b) Delaunay triangulationDel(S)or∞-complex(S) ;

(c)αopt-complex (as a simplicial complex)Cαopt(S)triangulated byDelαopt(S); (d)αopt-shape as a

polytopeS_α_opt(S) (a) (b)

(c) (d)

26/48

H. Edelsbrunner, E.P. Mucke, E.P., Three-dimensionnal alpha-shapes, ACM Transactions on Graphics 13 (1), pp. 43-72 (1994)

(49)

Shape Analysis

Binarization of a Point SetS

A spectrum ofα−objects derived from the Edelsbruner’s modeling : fromS0

to the meshed convex hullS∞ ofS.

∀S, α-bin(S) ={T ∈Del(S)|ρ_T < α} ≡Del_α(S)

|α-bin(S)|=|Del_α(S)|=|S_α(S)|and α_opt =2∗median_T∈Del(S)(ρ)

27/48

(50)

Shape Analysis

A Morphological Study

In the α-object framework :

∀k∈N,e_T^k =max{e_T^k⁻¹0 |T⁰∈neighbor(T)}

∀k∈N,d_T^k=min{d_T^k⁻¹0 |T⁰∈neighbor(T)}

and e_T⁰ =d_T⁰ =ρT

neighbor(T) = {T⁰ ∈Del|T⁰∩T 6=∅ and |T⁰|=|T|=3}

28/48

(51)

Shape Analysis

ErodingS

(a) A point setSin<²;

(b) Its binarized representation ;

(c)α-eroded(S) forα=αopt .

∀T,e_T=max{ρ_T|T⁰∈neighbor(T)}

∀T,d_T=min{ρ_T|T⁰∈neighbor(T)}

α-eroded(S) ={T⁰ ∈Del|e_T⁰ < α}

28/48

(52)

Shape Analysis

DilatingS

(c)α-dilated(S) forα=αopt .

∀T,e_T=max{ρ_T|T⁰∈neighbor(T)}

∀T,d_T=min{ρ_T|T⁰∈neighbor(T)}

α-dilated(S) ={T⁰ ∈Del|d_T⁰ < α}

28/48

(53)

Shape Analysis

Properties of the newα-objects(S)

k acting as the size of the structuring element α^k-eroded(S) = {T⁰ ∈Del|e_T^k0 < α

and |T⁰|=3}

α^k-dilated(S) = {T⁰ ∈Del|d_T^k0 < α and |T⁰|=3}

(α^k-dilated(S))^C = {T⁰ ∈Del|e_T^k0 > α and |T⁰|=3}

29/48

(54)

Shape Analysis

Properties of the newα-objects(S)

Duality of the transformations : (a) In gray, the α-complex and in black, theα-eroded complex (b) In black, the α-dilated of the

complementaryα-complex

(a) (b)

29/48

(55)

Shape Analysis

Duality Edge/Face

Successive Erosions in Contour Mode

λ^ke =min{eT^k , e_T^k0} and µ^ke=max{eT^k , e_T^k0} λ^ko=min{oT^k , oT^k⁰} and µ^ko=max{oT^k , oT^k⁰}

Morph. operator Region Mode Contour Mode α^k−eroded α∈[e^k,∞[ α∈[λ^ke, µ^ke] α^k−opened α∈[o^k,∞[ α∈[λ^ko, µ^ko]

29/48

(56)

Shape Analysis

Playing with the newα-objects(S)

29/48

MorphoMesh Package - Java ImageJ plugin :

http: // www. math-info. univ-paris5. fr/ ~lomn/ Data/ MorphoMesh. zip

(57)

Shape Analysis

Opening and ClosingS

(c)α-open(S) forα=αopt .

oT^k=min{e^k_T0|T⁰∈neigh(T)}

=minT0 ∈neigh(T){maxT00 ∈neigh(T0){e_T^k−100 }}

α^k-opened={T ∈Del|oT^k < αand|T|=3}

30/48

(58)

Digital Histopathology

31/48

(59)

Outline

• A Proposal for a new ANR project ?;

32/48

(60)

Augmented Reality

Question :

How to Upgrade the Visual Experienceof Tissue Image Exploration ?

33/48

(61)

Augmented Reality : Take Away Experience

New Display : Novel Practice

• As an Incentive to drive IT innovation ;

• Digital Display in Histopathology is at least Changing Reality.

• Augmented Vision : Eye-tracking technology for Saliency Map ;

34/48

(62)

Structural Analysis

Question :

How to Perform a Geometrical Analysisof Tissue Images ?

35/48

(63)

Structural Analysis : Experience Take Away

Working on Geometric Primitives : Point Set Extractor for Tissue Images

Thorough assessment of seed cell extractors :

Shijian LU’s Method Maya ALSHEH ALI’s Method

I2R, A*STAR, Singapore SIP Group

Visiting in June 2013 Visiting Ecole Polytech. Montreal

36/48

(64)

Structural Analysis : Experience Take Away

RoI Detection based on Mesh Morphological Operators

The FlexMIm project (FUI) AP-HP (27 departments)

37/48

(65)

Image Mining

Question :

How toExtract Knowledge from Tissue Images ?

Very different from Face Recognition issues ;

Expertise from theVISAVI project - ANR RNTL 2007-2009in collaboration with Nicole Vincent, Rabie Hachémi, Thomas Pons, Isabelle Bouvarelle etc.

38/48

(66)

Image Mining : Take Away Experience

Ontological Perception

A Medical Terminology→

Like ADICAP, SNOMED

→ A Digital Ontology For Micro-Semiology

39/48

(67)

Image Mining : Take Away Experience

Learning is a Bottleneck : Tissue Screening with Bio-codes Two structural bio-codes for various breast cancer based on the nuclei organization analysis. The bio-codeBC2 is based on the Euler Number computed over the three mesh representations

Cancer BC1 based on BC2

Type

EN,CC,MS based on EN

Normal cells 111 101

Ductal hyperplasia 010 000

Atypical ductal 110 101

DCIS 000 110

DCIS-MI 220 112

Invasive ductal 550 115

40/48

(68)

Image Mining : Take Away Experience

Learning is a Bottleneck : Tissue Screening with Bio-codes

(a)From left to right : a representationDelαopt of a tubular bio-structure, the opening of order 2 o²(Delαopt) and the difference between the two meshes

Delαopt−o²(Delαopt); (b)Idem for DCIS -like bio-structure. These two bio-structures are respectively encoded with the bio-codes101 and110.

(a)

(b)

40/48

Publication (aimed at IEEE Transactions on Medical Imaging) : Nicolas Lomenie, Ludovic Roux, Gilles Le Naour, Shijian Lu, Frédérique Capron, Point Set Processing for Histopathological Image Analysis : a visual slide study

(69)

Image Mining : Take Away Experience

Image examples of various cell nuclei organizations at the same magnification : (a)(c) DCIS or DCIS-like configurations of various sizes ; (b) tubule or gland formations.

40/48

(70)

Image Mining : Take Away Experience

Nuclei extractor assessment

Method TP FP FN

Method 1 1166 691 1 Method 2 390 395 777

Method Recall Precision F-measure Comp. Load

Method 1 0.999 0.628 0.771 ≈30s

Method 2 0.334 0.497 0.400 ≈10 ms

40/48

(71)

Image Mining : Take Away Experience

(a)Magnification x20 ; (b) Magnification x40 (about 2000 x 2000 pixels) : Fine detection point set extractor (Method 1) ; (c) Coarse detection point set extractor (Method 2). These bio-structures are all encoded into the structural bio-code

’110’

(a)

(b)

(c)

40/48

(72)

Image Mining : Take Away Experience

Efficiency of the cell configurations classifier by structural filtering using DCIS(S). A DCIS being considered as an abnormal configuration and Tubule as a normal gland in the tissue.

Type Nb samples Correct Biocodes Method 1 Method 2

DCIS(S) =⁰110⁰ 10 9 8

DCISpost(S) =⁰110⁰ 10 9 9

Tubule(S) =⁰101⁰ 10 10 10

40/48

(73)

Spatial Reasoning

Question :

How to improve/emulate the Cognitive Exploration of Tissue Images ?

41/48

(74)

Spatial Reasoning

Mitoses Counting through Ontological Modeling/Reasoning :

With the Protégé^TM framework :

−→Nucleus(?x)∧hasIntensity(?x,?value)

∧swrlb:lessThan(?value,110,0)∧hasCircularity(?y,?cir)

∧swrlb:lessThan(?cir,0.75)∧hasRound(?z,?round)

∧swrlb:lessThan(?round,0.65)→Mitosis(?x)

−→Mitosis(?x)∧Neoplasm(?y)

∧hasNeoplasmPeriphery(?y,?z)→sqwrl:select(?x,?z)

42/48

(75)

Spatial Reasoning : Take Away Experience

A Need for a Versatile Framework in Bioimaging :

New collaboration with Laurent Wendling (and Pascal Matsakis, Canada) : CombiningForce Histograms and Mesh Morphological Operators.

43/48

(76)

Spatial Reasoning : Take Away Experience

A Narrow Stream of Research to Enlarge

Qualitative modeling

→ Quantitative

assessment

43/48

Ruttenberg BE, Luna G, Lewis GP, Fisher SK, Singh AK : Quantifying spatial relationships from whole retinal images. Bioinformatics (Bioimage informatics).

29(7) :940-6 (2013)

Randell, D.A.[David A.], Landini, G.[Gabriel], Galton, A., Discrete Mereotopology for Spatial Reasoning in Automated Histological Image Analysis, PAMI(35), No. 3, March 2013, pp. 568-581.

(77)

Spatial Reasoning : Take Away Experience

Geometric/Topological Reasoning

(a) (b)

(c) (d)

(a) A 1024 x 1024 pixels sub-image out of a WSI ; (b)Delα(S); (c) Focus on Ductal Carcinoma In Situ (DCIS)areas witho²(Delα(S)); (d) Focus on normal cells withDelα(S)∩(o²(Delα(S))∩Delα(S))^c

44/48

(78)

Spatial Reasoning : Take Away Experience

Geometric/Topological Reasoning

(a) (b) (c)

(d) (e) (f)

Point set (a) without artifactsM; (d) and with artifactsMb; (b) Binarization ofM; (e) Binarization deMb; (c) Opening ofM; (f) (g) Opening ofMb.

44/48

(79)

Spatial Reasoning : Take Away Experience

Computational Topology & Math. Morph. & Simplicial Complexes

44/48

Kyoshitaka Kimori1, Norio Baba, Nobuhiro Morone. Extended morphological processing : a practical method for automatic spot detection of biological markers from microscopic images. BMC Bioinformatics,11:373, (2010) Olivo-Marin J-C : Extraction of spots in biological images using multiscale products. Pattern Recognition,35:1989-96, (2002).

(80)

Wrap up

• Augmented Vision for Digital Histopathology ;

• Digital Ontology ;

• Assessment of point set extractor within WSI ;

• Structural Analysis by biocodes ;

• Geometric/Topological Analysis/Reasoning within WSI ;

45/48

(81)

Horizon

Digital histopathology and graphs Emerging topic (France and US)

http://www.computational-pathology.org/

46/48

Lomenie and Racoceanu. (2012). Point set morphological filtering and semantic spatial configuration modeling : application to microscopic image and

bio-structure analysis,Pattern Recognition,45(8) :2894-2911.

Ta, Lezoray, Elmoataz and Schupp. (2009). Graph-based Tools for Microscopic Cellular Image Segmentation,Pattern Recognition,42(6) :1113-25.

Basavanhally, Ganesan, Agner, Monaco, Feldman, Tomaszewski. (2010) Computerized image-based detection and grading of lymphocytic infiltration in HER2+ breast cancer histopathology,IEEE Trans Biomed Eng,,57:642-653.

(82)

Horizon

O. Lezoray, M. Gurcan, A. Can, J.-C. Olivo-Marin, Whole Slide Microscopic Image Processing - Special Issue Editorial,

Computerized Medical Imaging and Graphics, Vol. 35, n˚7-8, pp.

493-495, 2011.

Gurcan MN, Boucheron LE, Can A, Madabhushi A, Rajpoot NM, Yener B. (2009) Histopathological image analysis : a review.IEEE Reviews in Biomedical Engineering,2 : 147-171.

Cigdem Gunduz, Bülent Yener,and S. Humayun Gultekin, The cell graphs of cancer Bioinformatics (2004) 20 (suppl 1) : i145-i151 doi :10.1093/bioinformatics/bth933

Andrei Chekkoury, Parmeshwar Khurd, Jie Ni, Claus Bahlmann, Ali Kamen, Amar Patel, Leo Grady, Maneesh Singh, Martin Groher, Nassir Navab, Elizabeth Krupinski, Jeffrey Johnson, Anna Graham and Ronald Weinstein, "Automated Malignancy Detection in Breast Histopathological Images", Proc. of SPIE 2012 (winner of Best Student Paper award).

Parmeshwar Khurd, Leo Grady, Ali Kamen, Summer Gibbs-Strauss, Elizabeth M. Genega and John V. Frangioni, "Network Cycle Features : Application to Computer-Aided Gleason Grading of Prostate Cancer Histopathological Images", Proc. of ISBI 2011.

M. Toutain, O. Lézoray, F. Audigié, V. Busoni, G. Rossi, F. Parillo, A. Elmoataz, Analysis of whole slide images of equine tendinopathy, ICIAR (International Conference on Image Analysis and

Recognition), Vol. LNCS 7325, pp. 440-447, 2012.

47/48

(83)

Thank You for your Attention

Nicolas Loménie, ing, Ph.D.

Laboratoire d’Informatique de Paris Descartes (LIPADE)

Equipe d’Acceuil EA 2917

Groupe Systèmes Intelligents de Perception (SIP) http ://w3.mi.parisdescartes.fr/sip-lab/

Image & Pervasive Access Lab (IPAL)

International Joint Research Unit — UMI CNRS 2955 http ://ipal.cnrs.fr/

48/48