Lung tissue analysis: from local visual descriptors to global modeling

Thesis

Reference

Lung tissue analysis: from local visual descriptors to global modeling

DICENTE CID, Yashin

Abstract

Medical imaging plays an important role in patient diagnosis and treatment planning. A standard procedure to assess a respiratory disease is a CT scan of the chest, where radiologists can detect subtle alterations in the lung tissue. This thesis aims at describing the lung tissue in CT scans, both from a local and a global perspective. It explores all the steps involved in the pipeline for the automatic analysis of the lung tissue: the initial lung segmentation, the division of lung fields into subregions, the extraction of local biomedical features, and the assembly of local features to form a global model. A new tissue descriptor is presented, as well as a novel graph-based model that provides a global characterization of the lung tissue. In addition, this thesis describes a new on-line platform where clinicians can extract state-of-the-art computerized image-based features.

DICENTE CID, Yashin. Lung tissue analysis: from local visual descriptors to global modeling. Thèse de doctorat : Univ. Genève, 2018, no. Sc. 5248

URN : urn:nbn:ch:unige-1113942

DOI : 10.13097/archive-ouverte/unige:111394

Available at:

http://archive-ouverte.unige.ch/unige:111394

Disclaimer: layout of this document may differ from the published version.

1 / 1

UNIVERSIT ´E DE GEN `EVE

D´epartement d’informatique FACULT ´E DES SCIENCES

Professeur Dr. St´ephane Marchand–Maillet D´epartement de radiologie et informatique m´edicale FACULT ´E DE M ´EDECINE Professeur Dr. Henning M¨uller

Lung Tissue Analysis:

From Local Visual Descriptors To Global Modeling

TH` ESE

pr´esent´ee `a la Facult´e des sciences de l’Universit´e de Gen`eve pour obtenir le grade de Docteur `es sciences, mention informatique

par

Yashin Dicente Cid

de

Barcelone (Espagne)

Th`ese N^o 5248

GEN `EVE 2018

fesseur titulaire et codirecteur de th`ese (Facult´e de m´edecine, D´epartement de radiologie et informatique et Haute Ecole Sp´ecialis´ee de Suisse Occidentale, Sierre, Valais, Suisse), Mon- sieur P.-A. POLETTI, professeur associ´e (Facult´e de m´edecine, D´epartement de radiologie et informatique), Monsieur K. BATMANGHELICH, docteur (Department of Biomedical Informatics, University of Pittsburgh, U.S.A.) autorise l’impression de la pr´esente th`ese, sans exprimer d’opinion sur les propositions qui y sont ´enonc´ees.

Gen`eve, le 27 aoˆut 2018

Th`ese - 5248 -

Le Doyen

Contents

Abstract v

R´esum´e vii

Resumen ix

Resum xi

Acknowledgements xiii

1 Introduction 1

1.1 Motivation . . . 1

1.2 Thesis Objectives . . . 2

1.3 Thesis Overview . . . 2

1.4 Scientific Contributions . . . 4

2 Computerized Biomedical Image Analysis 5 2.1 Computerized Texture Analysis . . . 5

2.2 Higher–Order Riesz–Wavelet Transform: Background . . . 7

2.2.1 Notation . . . 8

2.2.2 Higher–Order Riesz Transform of 3D Signals . . . 8

2.2.3 Steerability . . . 9

2.2.4 3D Riesz–Wavelet Pyramid . . . 10

2.2.5 Tensor–based Estimation of Local Image Orientations . . . 10

2.2.6 Rotation–Covariance . . . 10

2.2.7 Uni–directional and Multi–directional Riesz Components . . . 11

2.3 Locally–Oriented Higher–Order 3D Riesz–Wavelet Transform . . . 11

2.3.1 Local Image Orientation Estimation Based on M–D Riesz Repre- sentations . . . 12

2.3.2 Local Image Orientation Estimation Based on Uni–Directional Riesz Components . . . 13

2.3.3 Local Image Orientation Estimation Based on 1st–Order Riesz Com- ponents . . . 14

2.4 Evaluation of the Proposed Alignments . . . 14

2.4.1 RFAI Database . . . 14

2.4.2 Experimental Setup . . . 17

2.4.3 Quantitative Evaluation . . . 18

2.4.4 Qualitative Evaluation . . . 23 i

2.5 Biomedical Image Features in Practice: The QuantImage Platform . . . 25

2.5.1 Multi–Modality Alignment . . . 27

2.5.2 Intensity–Based Features . . . 27

2.5.3 Distance Features: Measures of Cancer Invasiveness . . . 30

2.5.4 Texture Features . . . 32

2.5.5 Adoption of the QuantImage Platform by Health Professionals . . . 36

2.6 Summary . . . 36

3 Segmentation and Division of the Lung Fields 39 3.1 Automatic Lung Segmentation in CT Images . . . 39

3.1.1 Respiratory System Segmentation . . . 40

3.1.2 Removing Trachea and Primary Bronchi . . . 40

3.1.3 Right–Left Lung Identification and Mask Refinement . . . 41

3.2 Lung Segmentation Evaluation . . . 43

3.2.1 VISCERAL Anatomy3 Benchmark . . . 43

3.2.2 Evaluation on a Large–Scale Lung CT Dataset . . . 44

3.2.3 Supervised Evaluation on the ImageCLEF Tuberculosis Dataset . . 45

3.2.4 Pulmonary Hypertension and Pulmonary Embolism Dataset . . . . 46

3.3 Lung Atlases . . . 48

3.3.1 Geometric Atlas of the Lungs . . . 48

3.3.2 Texture–based Atlas of the Lungs . . . 49

3.4 Summary . . . 54

4 Biomedical Image Analysis of the Lung Parenchyma 57 4.1 Age–Related Evolution of Lung Tissue Visual Features . . . 57

4.1.1 Dataset . . . 59

4.1.2 Biomedical Image Features Extracted . . . 59

4.1.3 Analysis of the Age–Related Trends . . . 62

4.1.4 Interpretation of the Age–Related Trends Found . . . 66

4.1.5 Conclusions . . . 68

4.2 Vessel–based Biomedical Features on Dual Energy CT Scans . . . 69

4.2.1 Dataset: PE DECT Subset . . . 69

4.2.2 Energy–based Vessel–Prisms . . . 69

4.2.3 Vs–prism Classification and Validation . . . 70

4.2.4 Results . . . 72

4.2.5 Discussion . . . 73

4.3 Texture Distribution on Pulmonary Circulatory Diseases . . . 76

4.3.1 Dataset: PH and PE CT Subset . . . 77

4.3.2 Analysis of the Texture–based Atlases . . . 77

4.4 Summary . . . 79

5 Graph–based Models for Lung Characterization 81 5.1 Graph Model of the Lungs . . . 81

5.2 Undirected Complete Graph Using Local DECT Features . . . 83

5.2.1 Dataset: PH and PE DECT Subset . . . 84

5.2.2 Intensity–based Graph Model . . . 84

5.2.3 Graph Model Classification . . . 86

5.2.4 Quantitative Evaluation of the Graph Model . . . 86

CONTENTS iii

5.2.5 Qualitative Evaluation . . . 87

5.3 Texture–based Graph Model of the Lungs . . . 89

5.3.1 3D Texture Features . . . 89

5.3.2 Graph Architectures . . . 89

5.3.3 Graph–based Patient Descriptors . . . 91

5.3.4 Participation in the ImageCLEF 2017 Tuberculosis Task . . . 91

5.4 Undirected Vs. Directed Texture–based Graph . . . 99

5.4.1 Adding Directionality to the Texture–based Graph . . . 99

5.4.2 Graph–based Patient Descriptor . . . 100

5.4.3 Quantitative Evaluation on the PH and PE Dataset . . . 100

5.4.4 Qualitative Evaluation . . . 101

5.5 Summary . . . 103

6 Conclusions and Perspectives 105

List of Figures 107

List of Tables 114

Bibliography 116

Nomenclature 137

Acronyms 143

Abstract

Medical imaging plays an important role in diagnosis and treatment planning, in particular for the assessment of respiratory diseases. Respiratory diseases are among the leading causes of death worldwide, thus, an early and accurate diagnosis of them can have a major impact. A standard procedure to assess a respiratory disease is a Computed Tomography (CT) scan of the chest, where radiologists can detect subtle alterations in the lung tissue that could help to correctly identify and diagnose the disease. However, when there is a general distribution of the disease in the lungs, the pathological changes can be so elusive that more (invasive) studies may be required to establish a diagnosis. Overall, a miss rate as high as 30 % and a false positive rate of up to 15 % has been reported in radiology.

Therefore, many Computer–Aided Diagnosis (CAD) systems have been designed, usually based on characterizing the lung tissue.

CAD systems often include computerized image–based texture descriptors for tissue analysis. The complex structure of the lung parenchyma requires powerful texture descrip- tors that encode directional information in a rotation–invariant manner. Nonetheless, this is not trivial in 3D volumes and it remains a challenging task in computerized image anal- ysis. State–of–the–art approaches have attempted to diagnose pulmonary pathologies by quantifying local defects in the lung parenchyma. However, in some respiratory diseases, it is not the proportion of abnormal tissue that differentiates them, but rather the spa- tial distribution of the affected regions across the lung. Providing advanced computerized techniques to health professionals in a user–friendly platform may help the detection and diagnosis of these complex cases.

This thesis aims at describing the lung tissue in CT scans, both from a local and a global perspective. A new local 3D texture descriptor with the aforementioned properties is presented, as well as a novel graph–based model that provides a global characterization of the lung tissue. In addition, this thesis describes a new on–line platform where clinicians can extract state–of–the–art computerized image–based features. Moreover, this thesis explores all the steps involved in the pipeline for the automatic analysis of the lung tissue:

the initial lung segmentation, the division of lung fields into subregions, the extraction of local biomedical features, and the assembly of local features to form a global model.

The methods presented in this thesis have been extensively tested in four datasets, together accounting for more than 14,000 CT scans from over 10,000 patients. They include patients with very diverse diseases, such as tuberculosis and pulmonary circulatory pathologies. The designed experiments and the obtained results confirm the robustness of the techniques detailed in this thesis.

v

R´ esum´ e

L’imagerie m´edicale joue un rˆole important dans le diagnostic et la planification du traite- ment, en particulier pour l’´evaluation de maladies respiratoires. Les maladies respiratoires sont parmi les principales causes de mortalit´e dans le monde et, par cons´equent, un diag- nostic pr´ecoce et pr´ecis d’entre eux peut avoir un impact majeur. Une proc´edure standard pour l’´evaluation d’une malade respiratoire est une tomographie du thorax, qui peut per- mettre aux radiologues de d´etecter des alt´erations subtiles dans le tissu pulmonaire, afin de les aider `a identifier et diagnostiquer correctement la maladie. Cependant, lorsqu’il y a une distribution g´en´eralis´ee de la maladie dans les poumons, les changements pathologiques peuvent ˆetre si insaisissables qu’ils requi`erent plus d’´etudes (invasives) afin d’´etablir un diagnostic. Dans l’ensemble, un taux de faux n´egatifs pouvant aller jusqu’`a 30 % et un taux de faux positifs allant jusqu’`a 15 % ont ´et´e rapport´es en radiologie. Par cons´equent, de nombreux syst`emes de diagnostic assist´e par ordinateur (CAD pour son acronyme en anglais) ont ´et´e con¸cus, g´en´eralement bas´es sur la caract´erisation du tissu pulmonaire.

Les syst`emes CAD comprennent souvent des descripteurs de texture bas´es sur l’image pour l’analyse de tissu. La structure complexe du parenchyme pulmonaire n´ecessite de puissants descripteurs de texture qui encodent l’information directionnelle de mani`ere in- variante aux rotations. N´eanmoins, ce n’est pas trivial pour des volumes en 3D et cela reste une tˆache exigeante dans le domaine de l’analyse d’image informatis´ee. Des approches con- formes `a l’´etat de l’art tentent de diagnostiquer des pathologies pulmonaires en quantifiant des d´efauts locaux dans le parenchyme pulmonaire. Cependant, dans certaines maladies respiratoires, ce n’est pas la proportion de tissu anormal qui les diff´erencie, mais plutˆot la distribution spatiale des r´egions affect´ees dans le poumon. Fournir des techniques in- formatiques avanc´ees aux professionnels de la sant´e `a travers une plate–forme conviviale peut aider `a la d´etection et au diagnostic de ces cas complexes.

Cette th`ese vise `a d´ecrire le tissu pulmonaire dans les tomographies, `a la fois d’un point de vue local et global. Un nouveau descripteur de texture 3D avec les propri´et´es susmentionn´ees est pr´esent´e, ainsi qu’un mod`ele novateur bas´e sur des graphes qui fournit une caract´erisation globale du tissu pulmonaire. En outre, cette th`ese d´ecrit une nouvelle plate-forme en ligne permettant `a des cliniciens d’extraire des caract´eristiques informa- tis´ees bas´ees sur l’image `a la pointe de la technologie. De plus, cette th`ese explore toutes les

´etapes du processus d’analyse automatique du tissu pulmonaire : la segmentation initiale des poumons, la division des poumons en sous-r´egions, l’extraction des caract´eristiques biom´edicales locales et l’assemblage de ces caract´eristiques locales pour former un mod`ele global. Les m´ethodes pr´esent´ees ont ´et´e test´ees de mani`ere extensive dans quatre ensem- bles de donn´ees, repr´esentant ensemble plus de 14’000 tomographies provenant de plus de 10’000 patients. Ils incluaient des patients avec des maladies tr`es diverses, telles que la tuberculose ou les pathologies circulatoires pulmonaires. Les exp´eriences con¸cues et les r´esultats obtenus confirment la robustesse des techniques d´etaill´ees dans cette th`ese.

vii

Resumen

Las im´agenes m´edicas desempe˜nan un papel importante en el diagn´ostico y la planificaci´on del tratamiento, en particular, para la evaluaci´on de enfermedades respiratorias. Las enfermedades respiratorias se encuentran entre las principales causas de muerte en todo el mundo, por lo tanto, un diagn´ostico temprano y preciso puede tener un gran impacto.

Un procedimiento est´andar para evaluar una enfermedad respiratoria es una tomograf´ıa computarizada (CT por su acr´onimo en ingl´es) del t´orax, donde los radi´ologos pueden detectar alteraciones sutiles en el tejido pulmonar que podr´ıan ayudar a identificar y diagnosticar correctamente la enfermedad. Sin embargo, cuando hay una distribuci´on generalizada de la enfermedad en los pulmones, los cambios patol´ogicos pueden ser tan elusivos que pueden requerirse m´as estudios (invasivos) para establecer un diagn´ostico.

En general, una tasa de error de hasta el 30 % y una tasa de falsos positivos de hasta el 15 % ha sido reportada en radiolog´ıa. Debido a esto, se han dise˜nado muchos sistemas de diagn´ostico asistido por computadora (CAD por su acr´onimo en ingl´es), generalmente basados en la caracterizaci´on del tejido pulmonar.

Los sistemas CAD a menudo incluyen descriptores visuales de textura computarizados para el an´alisis de tejidos. La estructura compleja del par´enquima pulmonar requiere po- tentes descriptores de textura que codifiquen informaci´on direccional de manera invariante a rotaciones. No obstante, esto no es trivial en vol´umenes 3D y sigue siendo una tarea dif´ıcil en el an´alisis computarizado de im´agenes. Los m´etodos en el estado del arte han intentado diagnosticar las patolog´ıas pulmonares mediante la cuantificaci´on de defectos locales en el par´enquima pulmonar. Sin embargo, en algunas enfermedades respiratorias, no es la proporci´on de tejido anormal lo que las diferencia, sino la distribuci´on espacial en el pulm´on de las regiones afectadas. Proporcionar t´ecnicas computarizadas avanzadas a los profesionales de la salud en una plataforma f´acil de usar puede ayudar a la detecci´on y el diagn´ostico de estos casos complejos.

Esta tesis tiene como objetivo describir el tejido pulmonar en CTs, tanto desde una perspectiva local como global. En ella se presenta un nuevo descriptor de textura local en 3D con las propiedades antes mencionadas, as´ı como un nuevo modelo basado en grafos que proporciona una caracterizaci´on global del tejido pulmonar. Adicionalmente, esta tesis describe una nueva plataforma en l´ınea donde los m´edicos pueden extraer caracter´ısticas visuales computarizadas de ´ultima generaci´on. Adem´as, esta tesis explora todos los pasos involucrados en el proceso para el an´alisis autom´atico del tejido pulmonar: la segmentaci´on inicial de los pulmones, la divisi´on de los campos pulmonares en subregiones, la extracci´on de caracter´ısticas biom´edicas locales y el ensamblaje de caracter´ısticas locales para formar un modelo global.

Los m´etodos presentados en esta tesis han sido ampliamente probados en cuatro con- juntos de datos, que suman m´as de 14.000 CTs de m´as de 10.000 pacientes. ´Estos incluyen pacientes con enfermedades muy diversas, como tuberculosis y patolog´ıas circulatorias pul-

ix

monares. Los experimentos dise˜nados y los resultados obtenidos confirman la solidez de las t´ecnicas detalladas en esta tesis.

Resum

Les imatges m`ediques tenen un paper important en el diagn`ostic i la planificaci´o del tracta- ment, en particular, per a l’avaluaci´o de malalties respirat`ories. Les malalties respirat`ories es troben entre les principals causes de mort a tot el m´on, per tant, un diagn`ostic preco¸c i prec´ıs pot tenir un gran impacte. Un procediment est`andard per avaluar una malaltia res- pirat`oria ´es una tomografia computeritzada (CT pel seu acr`onim en angl`es) del t`orax, on els radi`olegs poden detectar alteracions subtils en el teixit pulmonar que podrien ajudar a identificar i diagnosticar correctament la malaltia. No obstant, quan hi ha una distribuci´o generalitzada de la malaltia en els pulmons, els canvis patol`ogics poden ser tan elusius que poden requerir m´es estudis (invasius) per establir un diagn`ostic. En general, una taxa d’error de fins al 30 % i una taxa de falsos positius de fins al 15 % ha estat reportada en radiologia. A causa d’aix`o, s’han dissenyat molts sistemes de diagn`ostic assistit per ordinador (CAD pel seu acr`onim en angl`es), generalment basats en la caracteritzaci´o del teixit pulmonar.

Els sistemes CAD sovint inclouen descriptors visuals de textura computeritzats per a l’an`alisi de teixits. L’estructura complexa del par`enquima pulmonar requereix potents descriptors de textura que codifiquin informaci´o direccional de manera invariant a rota- cions. No obstant, aix`o no ´es trivial en volums 3D i segueix sent una tasca dif´ıcil en l’an`alisi computeritzat d’imatges. Els m`etodes en l’estat de l’art han intentat diagnosticar les patologies pulmonars mitjan¸cant la quantificaci´o de defectes locals en el par`enquima pulmonar. Malgrat aix`o, en algunes malalties respirat`ories, no ´es la proporci´o de teixit anormal el que les diferencia, sin´o la distribuci´o espacial en el pulm´o de les regions afec- tades. Proporcionar t`ecniques computeritzades avan¸cades als professionals de la salut en una plataforma f`acil d’utilitzar pot ajudar a la detecci´o i el diagn`ostic d’aquests casos complexos.

Aquesta tesi t´e com a objectiu descriure el teixit pulmonar en CTs, tant des d’una perspectiva local com global. S’hi presenta un nou descriptor de textura local en 3D amb les propietats abans esmentades, aix´ı com un nou model basat en grafs que proporciona una caracteritzaci´o global del teixit pulmonar. Adicionalment, aquesta tesi descriu una nova plataforma en l´ınia on els metges poden extreure caracter´ıstiques visuals computeritzades d’´ultima generaci´o. A m´es, aquesta tesi explora tots els passos involucrats en el proc´es per a l’an`alisi autom`atic del teixit pulmonar: la segmentaci´o inicial dels pulmons, la divisi´o dels camps pulmonars en subregions, l’extracci´o de caracter´ıstiques biom`ediques locals i l’assemblatge de caracter´ıstiques locals per formar un model global.

Els m`etodes presentats en aquesta tesi han estat `ampliament provats en quatre conjunts de dades, que sumen m´es de 14.000 CTs de m´es de 10.000 pacients. Aquests inclouen pacients amb malalties molt diverses, com tuberculosi i patologies circulat`ories pulmonars.

Els experiments dissenyats i els resultats obtinguts confirmen la solidesa de les t`ecniques detallades en aquesta tesi.

xi

Acknowledgments

This thesis has been possible thanks to the collaboration, effort and advice of many people.

First, I would like to thank my thesis advisors Henning M¨uller and St´ephane Marchand–

Maillet for their guidance and advice in this long work. Particularly, I sincerely would like to thank Henning for letting me follow my instincts and for giving me the enormous freedom that I have enjoyed all this time, both professionally and personally. Moreover, I would like to thank Pierre-Alexandre Poletti, who has been always available to help me and who has led the SNSF project (agreement 320030-146804) that has provided the necessary funds to make this thesis possible. Of course, without forgetting his colleague Alexandra Platon. Still in the professional field, I would like to thank Kayhan Batmanghelich for the numerous Skypes, for his critical comments and for being part of my thesis tribunal. His involvement has clearly improved the quality and clarity of this thesis.

I cannot forget all those people who during this stage have listened to me, have endured me, have discussed with me, have disagreed with me, have entertained me, have made me sweat and have made this Ph.D. an unforgettable experience. Thanks to the members of the MedGIFT group, to the interns and workmates of HES-SO, and to everyone with whom I have shared the lunchtime, the coffee hours, the ”ap´eros”, the ski resorts, the bike trails, etc. I would like to highlight (in strict order of appearance in this wonderful stage) and to give a special gr`acies, gracias, danke, merci, grazie and thanks to Pol, Oscar, Ale, Thomas, Antonio, Sergio, Roger, Stefano, Visara, Morgane, Michael, Alevtina, Adrien, Sebas, Nastya, Matteo, Gaetano, Francesca, Davide, Stephanie, and Vincent.

In a different category there is Alba, my colleague and friend that has been always by my side already for 11 years, from Amsterdam, passing by Barcelona and Sierre, and soon in the UK. Gracias por la inestimable ayuda que me llevas ofreciendo todos estos a˜nos.

Tambi´en quiero agradecer a todas esas personas que desde Barcelona me han empujado y me han dado fuerza para seguir adelante. Gracias a mis coleguill@s por estar disponibles para hacer una birra conmigo todas y cada una de las veces que he pasado por Barcelona.

Much´ısimas gracias a Borja, Carles, Laura y Toni por estar en los mejores y en los peores momentos de esta etapa.

El meu m´es profund agra¨ıment a la meva fam´ılia mataronina, la Pilar i la Marta. El vostre ajut ha estat indispensable per dur a terme aquesta aventura. No tinc paraules per agrair-los que hagin cuidat i continuin cuidant tan b´e d’en Seeker. Gr`acies pel vostre caliu, hospitalitat i per les moltes visites.

Mi familia no se merece menos que un GRACIAS en may´usculas. Gracias a mis padres por los valores que me han ense˜nado, por su apoyo y por su comprensi´on. Gracias a mi hermano Uriel por sus muchas visitas, por sus muchos viajes al aeropuerto, por siempre estar al otro lado del tel´efono y por cuidar a los nuestros. Muchas gracias tambi´en a mi t´ıo y a mi yaya por sus ´animos, cari˜no y ayuda.

Y para acabar, mi mayor agradecimiento para la que hoy es mi esposa. Por apoyarme xiii

en todas mis decisiones, por siempre escuchar mis tonter´ıas, por estar a mi lado en los peores momentos, por hacerme crecer como persona, por ponerme retos, y por cuidarme.

Sense tu no hagu´es tingut mai el coratge d’arribar tan lluny.

Gr`acies Meritxell!

Chapter 1

Introduction

1.1 Motivation

Respiratory diseases are among the leading causes of death worldwide, affecting more than 1 billion people in the world in 2013 [73]. Only lung infections (mostly pneumonia and tuberculosis), lung cancer, and Chronic Obstructive Pulmonary Disease (COPD) together accounted for 9.6 million deaths worldwide in 2015 [213]. In the same year, approximately 6.5 million patients with diseases of the respiratory system were discharged from EU hospitals¹. Therefore, an early and accurate diagnosis of respiratory diseases can have a major impact in health services.

Medical imaging plays an important role in diagnosis and treatment planning. A stan- dard procedure to assess respiratory diseases is a Computed Tomography (CT) scan of the chest, which allows a more detailed visualization of thoracic structures than traditional chest radiography (X–ray) [90]. An important task in the radiology workflow is to detect subtle alterations in patient scans that could help to correctly identify and diagnose dis- eases. However, when there is a widespread distribution of the disease in an organ, e.g., the lungs, the pathological changes can be so elusive that they require more (invasive) studies to establish a diagnosis [199]. Moreover, radiologists face the time consuming task of visually inspecting these large 3D scans. Overall, a miss rate as high as 30 % and a false positive rate of up to 15 % has been reported in radiology [121]. For assisting the radiolo- gists’ image interpretation, computerized analysis of medical images has been implemented clinically [66]. This is generally known as Computer–Aided Diagnosis (CAD).

High resolution 3D scans have made it possible to represent many medical structures as isotropic 3D Solid Textures (3DSTs). These images consist of textured dense 3D volumes, containing 2D textures in all planes of R³. Visualizing 3DSTs is difficult and full 3D analysis is not trivial even though many approaches exist [41]. Lung CT images have been the target for analysis and decision support for a long period of time, such as for nodule detection [93] and for the diagnosis of interstitial lung diseases [3]. Many techniques have developed image–based texture descriptors to characterize lung tissue [50, 215]. The lung parenchyma is composed of tubular structures forming a fractal branching tree [114], creating 3DST with strong directional components. Since these structures can be found with any orientation, rotation–invariant texture descriptors seem appropriate. However, fully leveraging the directional information of textures is a challenging problem, which

1http://ec.europa.eu/eurostat/statistics-explained/index.php/Respiratory_diseases_

statistics, as of April 5, 2018.

1

is even more difficult in 3D. In addition, the Local Organization of Image Directions (LOID), or how directional structures intersect, is fundamental for texture segregation [15].

Consequently, an inherent dilemma of directional texture analysis is to either use texture operators that are insensitive to directions and invariant to rotations, or use directional operators that are able to characterize the LOID but not in a rotation–invariant fashion.

Therefore, there is a need in tissue image analysis of 3D texture descriptors both describing the 3DST in a rotation–invariant fashion and encoding local directional information.

Besides, most of the work using texture to detect affected patients has focused on quantifying the proportion of abnormal texture present [18, 47, 85]. Other studies have presented region–based approaches to analyze the parenchyma [54, 218]. However, in some respiratory diseases, it is not the proportion of abnormal tissue that differentiates them but rather the spatial distribution across the lung of these textures, as in the case of Pulmonary Embolism (PE) and Pulmonary Hypertension (PH) [70]. A holistic model describing the entire lung parenchyma, instead of local independent assessments of alterations, is desirable to improve the clinical evaluation of a patient by providing a global pathological status of the lungs.

1.2 Thesis Objectives

This thesis has two main objectives, both focus on the analysis of the lung tissue, each from a local or a global point of view. The first objective is to create rotation–invariant local texture descriptors, maintaining the directionality properties of the lung tissue. This is targeted by improving a state–of–the–art 3D texture descriptor based on Riesz–wavelets.

The second objective is to design a general model of the lung capable to characterize the global aspect of the lung parenchyma in a holistic manner. In this case, the new model is based on a graph–based approach, where the lungs are seen as a graph entity, with nodes and edges based on a lung division or atlas, and regional visual descriptors. The pipeline to build this model is composed of three main steps, all of them investigated in this thesis, and becoming secondary but necessary objectives. These, in a few words, consist of: 1) automatically segmenting the lung fields in CT scans of patients presenting anomalies in the lung parenchyma; 2) dividing the lung fields into regions with similar tissue; and 3) analyzing current biomedical features and developing new visual descriptors to express the regional lung tissue in CT and Dual Energy CT (DECT) scans.

1.3 Thesis Overview

This thesis is structured as a journey from general computer vision to a specific graph–

based model of the lungs, following the required steps to build the latter. Figure 1.1 depicts how this thesis is divided and how the different chapters are related.

Chapter 2 contains my two broad contributions, first into the computer vision area, and second in the computerized biomedical imaging field. It starts by introducing an extension to the state–of–the–art Riesz–wavelet 3D texture descriptor. This extension allows the use of the higher–order 3D Riesz transform in a rotation–covariance manner (rotation–invariance with no loss of directionality), and it will be used as the main texture descriptor in this thesis. Moreover, the same chapter presents the QuantImage platform, an online web–service developed to facilitate the extraction of computerized biomedical features, including the new aligned 3D Riesz–wavelet transform, to non computer vision

1.3. THESIS OVERVIEW 3

Biomedical Image Analysis (Chapter 2)

Lung Segmentation

and Division (Chapter 3)

Lung Biomedical

Features (Chapter 4)

Lung Graph Models (Chapter 5)

Figure 1.1: Diagram containing the links between the four main chapters in this thesis.

experts, such as, clinicians.

In Chapter 3 we move into the domain of lung CT imaging, particularly, on how to automatically segment and divide the lung fields. These are primary steps in most CAD systems for lung diseases. A new automatic lung segmentation algorithm is introduced and analyzed in several datasets. Moreover, this chapter presents two lung division algorithms, one based on geometrical information and the other based on textural information obtained with the Riesz–wavelet descriptor.

Chapter 4 contains several analyses of visual descriptors extracted inside the lungs. It first describes a large study on age–related trends of state–of–the–art biomedical image features, carried out on more than 12,000 CT scans. Then, it presents some DECT–

specific visual features developed for PE detection. Finally, a texture distribution analysis performed for the distinction of PH and PE diseases is explained.

The novel graph–based model of the lungs is explicitly defined in Chapter 5. This model is able to combine local biomedical lung features in a holistic fashion based on a lung division,i.e., combining the techniques described in the three previous chapters. The general definition of the new framework is followed by the experiments performed using several architectures of the graph. These explore the potential of complete or incomplete, directed or undirected, and intensity– or texture–based graph lung models.

Finally, Chapter 6 includes the conclusions and perspectives of the methods presented in this thesis.

1.4 Scientific Contributions

The scientific contributions of this thesis can be grouped in the following six topics:

Computerized texture descriptors: In [64] a 3D rotation–invariant texture descrip- tor based on the Riesz–wavelet transform was presented. This descriptor was successfully applied for multimodal medical case retrieval in [105]. Moreover, in combination with a covariance–based descriptor designed in [34, 36], a 3D Riesz–covariance texture model for prediction of nodule recurrence in lung CT images was developed [35].

Biomedical feature extraction tools: A new web–based platform for 3D radiomics feature extraction was created in [56]. Thanks to this platform, several medical articles were published [24, 43, 182]. Moreover, some of the biomedical imaging techniques ex- plained in this thesis were included into the electronic Patient Annotation Device (ePAD) platform in [181], a web–based tool for quantitative imaging.

Lung segmentation and division on CT scans: The lungs are the main anatomical structures analyzed in this thesis. Several works were published focused on automatically segmenting the lung fields in CT images [58, 106, 108]. Additionally, a division of the lungs into texture–based subregions is presented in [65].

Biomedical image analysis of the lung tissue: Age–related correlations of state–of–

the–art biomedical image features inside of the lungs were analyzed in [61]. For this work, a large dataset containing more than 12,000 CT scans from patients with ages between 0 and 106 years was used. In [57], vessel–based features were designed to detect PE on DECT images. On the other hand, the classification of patients with Tuberculosis (TB) was attempted using deep–learning in [189] and using an structure Bag of Words (BoW) representation of the patients in [62].

Graphs for lung modeling: The novel graph–based framework for lung modeling pre- sented in this thesis has been applied to patients with pulmonary vascular pathologies in [63] and [59]. The former used simple statistics extracted from DECT images while the latter extracted 3D texture features from standard CT scans. Finally, it was also applied to model patients with TB in [55], obtaining the best results in the detection of Multi–Drug Resistance (MDR).

Organization of the ImageCLEF 2017 Tuberculosis task: The Image Retrieval Track of the CLEF (ImageCLEF) 2017 challenge included for first time a task using CT scans of patients with TB [103]. Thanks to the expertise obtained during this thesis on the analysis of lung CT scans I became the responsible of designing the task, preparing the data of the patients, creating the training and test sets, and evaluating and analyzing the techniques developed by the participants [60].

Chapter 2

Computerized Biomedical Image Analysis

This chapter first describes my theoretical contribution in the field of computerized texture analysis, introducing an improvement on a gold–standard texture descriptor: the locally–

oriented higher–order 3D Riesz–wavelet transform. This new descriptor was published in [64] and its implementation is available to the community². Second, I present a web–

based platform for helping to bring closer the biomedical community and the computer vision tools. This software is extensively detailed in [56].

2.1 Computerized Texture Analysis

Imaging techniques in areas such as biomedical imaging [46], material analysis [98], and structural geology [219], allow the acquisition of 3D Solid Texture (3DST) images. These images consist of textured dense 3D volumes, where 2D textures can be found in all planes ofR³. Textures existing in more than two dimensions cannot be fully visualized by humans.

Only virtual navigation in multi–planar rendering or semi–transparent visualization allows visualizing subregions in 2D projections or slices. Therefore, computational approaches are required because humans are not able to fully visualize, interpret and quantify 3DST properties. Figure 2.1 shows two examples of 3DST images.

Figure 2.1: Two examples of 3DST images where a spherical section is removed to show the solidity of the interior. These examples correspond to two images of the Reconnaissance de Formes, Analyse d’Images (RFAI) database (see Section 2.4.1), specifically from the datasetsFourier (left) andGeometric (right).

2http://publications.hevs.ch/index.php/publications/show/2035, as of March 12, 2018.

5

The automatic analysis of 3DST remains a challenging topic. Spatial scales and di- rections in images are fundamental for texture discrimination [16, 195]. In most cases, systems capable of acquiring 3D images also provide voxel sizes in physical dimensions and controlled viewpoints (such as Computed Tomography (CT) or Magnetic Resonance Imaging (MRI)). In this case, the analysis of image scales requires describing 3DST at multiple scales and not using scale–invariant descriptors, the latter entailing the risk of regrouping patterns of different nature. Scale is itself a powerful discriminative property.

Much work has been done on 2D texture analysis since the 1970s and many ap- proaches exist, as it is less complex than in 3D [46]. Popular approaches are gray–level matrices (co–occurrence [95], run–length [82], and size–zone [196]), Local Binary Pattern (LBP) [156], wavelets and filterbanks (Laplacian of Gaussian (LoG) [142], Gabor [128], maximum–response [207], Riesz [45, 202]), fractals [167], and learned representations (dic- tionary learning [139], deep convolutional neural networks [33]). In natural images, 2D analysis also often includes visual descriptors such as color, shape, and appearance [150]

that complement texture. Three–dimensional extensions of color descriptors are relatively straightforward. Many 3D shape descriptors exist and they are usually based on 3D sur- face models but can also be based on solid 3D information [20]. For image classification and object recognition, these features are usually aggregated in the form of visual words or Fisher vectors. This chapter aims only at describing texture properties in 3DST, and it does not cover color or shape features that can be combined with texture.

First attempts for describing 3DST were based on extending 2D texture descriptors to 3D. Fehr et al. extended LBPs [156] to 3D in [71]. 2D LBPs encode the local orga- nizations of image directions by constructing binary codes over circular neighborhoods.

Histograms of pattern occurrences can be used as a texture descriptor. Image rotations result in circular shifts of the LBPs and rotation–invariance can be obtained by using the magnitude of the discrete Fourier transform of the LBPs [2]. When extending LBPs to 3D, defining ordered spherical neighborhoods is not trivial, making it difficult to obtain rotation–invariance. Solutions were proposed using spherical harmonics [72] and cylindri- cal neighborhoods [18]. However, the radius of the neighborhood (scale) required by LBPs is controlled by a parameter that needs to be optimized for every application. Another set of descriptors used to describe 3DST is based on Gray–Level Co–occurrence Matri- ces (GLCMs) [118, 123]. A GLCM contains the counts of all co–occurrences between two voxel values separated by a predefined distance (radius) and in a specific direction. In 3D, the directions are defined for every surface point of a spherical neighborhood centered at each voxel. The radius of the neighborhood defines the scale. Texture descriptors can be obtained from each matrix by calculating, for example, the homogeneity, entropy or energy of a matrix in a specific direction and scale. Rotation–invariant features can be obtained by summing the descriptors obtained from GLCMs in all directions, i.e., a subset of 13 uniformly distributed directions. This procedure discards the directionality of the descriptor. Similarly to 3D LBPs, the scales need to be defined manually and re- quire optimization. GLCMs usually require a reduction of the bit depth used to describe voxel values: the number of possible co–occurrences is extremely large, leading to unstable values or extremely sparsely populated matrices. Run–Length encoding (RLE) [82] was extended to 3D in [214]. Similarly to GLCMs, this method requires reducing gray–levels and choosing directions. The requirement of making arbitrary choices entails the risk of losing important information. However, in RLE, the scale is not fixed and the descriptor can encode several dominant scales (run–lengths).

While these methods presented simple solutions to deal with texture directionality

2.2. HIGHER–ORDER RIESZ–WAVELET TRANSFORM: BACKGROUND 7

and rotation–invariance, they do not encode image scales and directions in a systematic fashion. Wavelets constitute an attractive solution to fully exploit texture information at multiple scales through multi–resolution image representations, where their Fourier transform is systematically split into a series of dyadic bands [140]. Consequently, wavelets were successfully used for texture analysis in 2D [9, 45, 128, 168, 201] and 3D [12, 35, 49].

Fully leveraging the directional information of textures is a challenging problem, which is even more difficult in 3D. In most cases, local rotation–invariance is required because class–specific texture primitives (or textons) can be found in any local orientation [109]. In addition, the Local Organization of Image Directions (LOID), or how directional structures intersect, is fundamental for texture segregation [15]. Therefore, an inherent dilemma of directional texture analysis is to either use (i) texture operators that are insensitive to directions and invariant to rotations (such as measurements based on gray–level matrices averaged over multiple directions or isotropic filters and wavelets) or (ii) use directional operators that are able to characterize the LOID but not in a rotation–invariant fashion.

In particular, isotropic–wavelets are rotation–invariant but they are not able to encode the directionality of the texture. The latter can be recovered by coupling each isotropic bandpass filter with directional filters, resulting in a loss of rotation–invariance.

The d–dimensional Riesz–wavelet transform combines isotropic–wavelets and direc- tional all–pass filters [32, 203]. The Riesz filters of order N behave like local Nth–order partial image derivatives. Several papers have already shown the benefits of the 3D Riesz–

wavelet transform when analyzing 3DST [37, 47, 49, 105, 107]. The order of the Riesz transform controls the richness of the directional filterbank, where increasingly complex structures can be described with higher orders. A key property of Riesz–wavelets is steer- ability [78, 204, 205]. This means that the local response of each Riesz component to an image rotated by an arbitrary angle can be derived analytically from a linear com- bination of the responses of all components of the filterbank. Therefore, the descriptor can be locally aligned analytically (meaning that it does not require convolving the im- age with the rotated filter) without losing the directional information. This property is known as rotation–covariance and it allows comparing textures with arbitrary local orien- tations [45, 47]. The steerability of the 3D Riesz–wavelet transform when the local image orientations are known is described in the literature [32, 202]. The estimation of the image orientation in the context of complex 3D texture patterns is challenging. Chenouardet al.

proposed to estimate local image orientations when using 1st–order Riesz–wavelets [31]

that encode simple image gradients. In most cases, the characterization of complex struc- tures containing subtle directional information requires using higher–order Riesz filters.

In the following sections, I present three methods for estimating local texture orienta- tions from Riesz transforms of any order in the context of 3DST analysis. Moreover, these three methods are evaluated and compared on a synthetic dataset of 3DSTs.

2.2 Higher–Order Riesz–Wavelet Transform: Background

This section presents all the preliminary knowledge on the higher–order Riesz–wavelet transform required on this thesis. Based on this knowledge, Section 2.3 presents the three methods I developed to estimate the local texture orientation from higher–order Riesz–

wavelet responses. The structure of this section is as follows: The notation, common to the state–of–the–art, is first introduced. Then, the higher–order Riesz transform is defined for a 3D image. The following subsection shows how to steer the Riesz components when the orientation of the original image is known. The coupling of the Riesz transform with

isotropic–wavelets is explained in the next subsection. It also explains how the steering of the Riesz components commutes with the isotropic–wavelet framework. The approach proposed by Chenouardet al. [31] to estimate the local image orientation from 1st–order Riesz components is then introduced. The formal definition of rotation–covariance follows.

Finally, I define a differentiation of the Riesz components based on their profile. This distinction will be used in the methods proposed in this chapter.

2.2.1 Notation

2.2.1.1 The Image as a 3D Signal

A 3D image is considered a 3D signalf indexed by the continuous–domain space variable x= (x1, x2, x3)∈R³. The 3–dimensional Fourier transformF of f is noted as:

f(x)←→^F f(ω) =ˆ Z

R³

f(x)e^−jhω,xidx₁dx₂dx₃, (2.1) with the pulsation vectorω= (ω1, ω2, ω3)∈R³.

2.2.1.2 Multi–Index Notation

Following the notation used in [32], let us consider 3D index vectors of the formn= (n1, n2, n3)∈N³. The following multi–index notations and operators are used:

• Sum of components: |n|=n1+n2+n3,

• Max of components: max(n) = max(n₁, n₂, n₃),

• Factorial: n! =n1!n2!n3!,

• Exponentiation of a vector v = (v₁, v₂, v₃)∈R³: vⁿ = v₁ⁿ¹vⁿ₂²vⁿ₃³. 2.2.2 Higher–Order Riesz Transform of 3D Signals

Unser et al. presented in [205] the Nth–order Riesz transform of a d–dimensional signal R^(N){f}. In the case of 3–dimensional signals, the Riesz transform of orderN ∈Nis com- posed ofM = ^(N+2)(N+1)₂ components that are denoted byRⁿ, withn= (n1, n2, n3)∈N³ such that |n| = N. Given ω = (ω₁, ω₂, ω₃) ∈ R³, each of these operators is an all–pass filter with the directional frequency response defined in Fourier as

dRⁿ(ω) = (−j)^N

r N! n₁!n₂!n₃!

ω₁ⁿ¹ωⁿ₂²ω₃ⁿ³ (ω₁ⁿ¹+ωⁿ₂² +ω₃ⁿ³)^N2

. (2.2)

Figures 2.2 and 2.3 show a 3D visualization of the 1st– and 2nd–order Riesz filters, respectively. Since the support of the components is allR³, the filters are convolved with a Gaussian kernel in order to visualize their profile. The 2nd–order Riesz filters shown in Figure 2.3 are used to illustrate properties of the higher–order components in further sections.

2.2. HIGHER–ORDER RIESZ–WAVELET TRANSFORM 9

G∗ R^(1,0,0) G∗ R^(0,1,0) G∗ R^(0,0,1)

Figure 2.2: 1st–order Riesz filters Rⁿ convolved with Gaussian kernels G(x). The three filters have exactly the same profile but different orientations. They are oriented along each of the three directionsx₁,x₂ and x₃ ofR³.

G∗ R^(2,0,0) G∗ R^(0,2,0) G∗ R^(0,0,2) G∗ R^(1,1,0) G∗ R^(1,0,1) G∗ R^(0,1,1) Figure 2.3: 2nd–order Riesz filtersRⁿ convolved with Gaussian kernelsG(x). There are two groups of three filters with the same profile but different orientations: {G∗ R^(2,0,0), G∗ R^(0,2,0), G∗ R^(0,0,2)}and {G∗ R^(1,1,0), G∗ R^(1,0,1), G∗ R^(0,1,1)}.

2.2.3 Steerability

Unser et al. showed in [205] that the Riesz filterbanks are steerable [78], which means that the local response of each componentRⁿ of an image f(x) rotated by an arbitrary rotation matrix R∈ R^3×3 (represented by fR) can be derived analytically from a linear combination of the responses of all components of the filterbank, using a steering matrix S_R as

R{f_R}=S_RR{f}, (2.3)

whereR denotes the Riesz transform of any order.

Moreover, Unseret al. demonstrated in Theorem 1 of [202] that for a given 3D rotation matrix R = (r₁,r₂,r₃)^T with r_i ∈ R³, the steering matrix S_R with elements s_n,m (in multi–index notation) is defined as

s_n,m = rm!

n!

X

|k₁|=n₁

X

|k₂|=n₂

X

|k₃|=n₃

δ_k1+k2+k3,m· n!

k1!k2!k3!r^k₁¹r^k₂²r^k₃³, (2.4) whereki ∈N³ and δk1+k2+k3,m is the Kronecker symbol used to exclude the summation terms withk1+k2+k3 6=m. This steering matrix preserves the inner–structure of the Riesz filters, i.e., rotating their coefficients in a coherent way by preserving the angles between the elements of the filterbank.

A consequence of this Theorem is that, if the rotation matrixRis known, it is possible to analytically align the Riesz coefficients applying the steering matrixS_R, where the Riesz transform off(x) is computed only once.

2.2.4 3D Riesz–Wavelet Pyramid

The Riesz transform has the property to map any frame ofL₂(R³) (Lebesgue space of finite energy functions [178]), including wavelet frames, intoL₂(R³) since it preserves the inner product ofL2(R³) [203]. The Riesz–wavelet construction proposed by Chenouardet al. in [31] is the one used in this chapter. This consists in applying the 3D Riesz transform to the coefficients of an isotropic–wavelet pyramid³ to build a steerable wavelet transform in 3D. In this case, the Riesz and wavelet transforms can be commuted. The Riesz transform is applied to each scale of the isotropic pyramid defined by the wavelet functionψ. When using an isotropic primary wavelet, the directionality information is encoded at each scale of the pyramid by the Riesz transform only. The Riesz–wavelet coefficients can thus be steered using the same steerability matrixS_R as the Riesz coefficients (see Equation 2.3).

Specifically, the coefficientsq_k,U(x) corresponding to a rotation of the Riesz–wavelet atoms by the unitary matrixU are computed as:

qk,U(x) =R_U{ψk∗f}(x) =UR{ψk∗f}(x) =Uqk(x), (2.5) whereψ_k is theψ isotropic wavelet at scale k.

2.2.5 Tensor–based Estimation of Local Image Orientations

As it is mentioned before, the objective is to estimate the orientation of the image using the Riesz responses. In [31], Chenouardet al. presented a tensor–based estimation of the local image orientations based on the 1st–order Riesz transform at a position x0. This estimation relies on computing the eigenvectors of the tensor matrixJ(x0), with

J(x0) =





R²₁{g∗f}(x₀) R₁R₂{g∗f}(x₀) R₁R₃{g∗f}(x₀) R₂R₁{g∗f}(x0) R²₂{g∗f}(x0) R₂R₃{g∗f}(x0) R₃R₁{g∗f}(x0) R₃R₂{g∗f}(x0) R²₃{g∗f}(x0)



, (2.6) whereg(x) is the regularization function of the orientation map (see [31]), in this case a 3D Gaussian window, andR= (R^(1,0,0),R^(0,1,0),R^(0,0,1)) = (R₁,R₂,R₃).

The collection of eigenvectors of J(x0) sorted by eigenvalue, defines a rotation matrix U_g. For each location x₀, the resulting matrix U_g maximizes the energy of the first component of the rotated Riesz transform UgR{f}. It then maximizes the residual energy for the second component, and then for the third.

2.2.6 Rotation–Covariance

Orienting the Riesz operatorsR_i locally following the approach explained in Section 2.2.5 is referred to asrotation–covariance (represented byR^RC{f}(x)), where the organization of image directions is characterized independently from its local orientation [45, 47]. This differs from the monogenic signal [31], since all the scales of the wavelet frame are aligned using the same rotation matrixUg derived from the highest image resolution regularized byg(x). It also differs from rotation–invariant operators since R^RC{f}(x) is directional.

Rotation–covariance is warranted while the Riesz operators are steered coherently. In the case of the 2nd–order Riesz transform, if after steering the components, the response corresponding to the componentG∗ R^(0,2,0) is moved to the componentG∗ R^(2,0,0), then, to preserve the relation between all components, the response to the componentG∗R^(0,1,1)

3A redundant pyramid is used to ensure translation–invariance of the wavelet transform.

2.3. LOCALLY–ORIENTED HIGHER–ORDER 3D RIESZ–WAVELET TRANSFORM11

needs to be moved to the component G∗ R^(1,0,1) (see Figure 2.3). The steering matrix S_R defined in Equation 2.4 preserves the coherence between the components. In the case of the 1st–order Riesz transform, as defined in Section 2.2.5, given a rotation matrixUg, the resulting steering matrix S_Ug = PU_g, where P∈ R^3×3 is a permutation matrix. P does not affect the property of rotation–covariance, due to the equivalence of the profiles of the 1st–order Riesz components (see Figure 2.2).

2.2.7 Uni–directional and Multi–directional Riesz Components

In the following sections, I distinguish uni– from multi–directional components. This distinction is based on the directional profiles of the Riesz components.

Definition 2.2.1. The uni–directional Riesz components are defined as the components Rⁿ with max(n) = N. The multi–directional Riesz components are then those with max(n)< N.

As an example, Figure 2.3 contains three uni–directional components (G∗ R^(2,0,0), G∗R^(0,2,0), andG∗R^(0,0,2)) and three multi–directional components (G∗R^(1,1,0),G∗R^(1,0,1), andG∗ R^(0,1,1)). For a 3–dimensional signal there are always three uni–directional Riesz components. These filters are encoding the variations of a signal only along one of the three main directions of the image f(x). In the particular case of the 1st–order Riesz transform, all components are uni–directional (see Figure 2.2).

2.3 Locally–Oriented Higher–Order 3D Riesz–Wavelet Trans- form

The previous sections showed that the Riesz–wavelet transform is affected by the orien- tation of the image. Therefore, an image f and a rotated version of the same image f_R have different Riesz–wavelet transform (see Figure 2.4). However, if this rotation R is known, then the Riesz filters can be steered by applying the steering matrix SR (that can be derived analytically from R), producing then the same response tof and f_R (see Figure 2.4). This technique provides rotation–covariance to the Riesz–wavelet transform (rotation–invariance with directional information). In addition, Chenouardet al.[31] pre- sented a tensor–based technique to estimate the local image orientations based on the 1st–order Riesz responses. However, no solution has been proposed yet for estimating the local image orientations based on the higher–order Riesz–wavelets transforms. Based on this, two relevant open questions are:

• “How can we estimate local image orientations based on higher–order Riesz trans- forms?”

• “Can we use the tensor–based technique as defined by Chenouard et al. for this task?”

In this section, I propose three approaches for estimating local image orientations based on 3D Riesz transforms of any order.

Figure 2.4: Schematic example showing the effect in the Riesz transform R of applying a rotationR to an image f. This effect can be neutralized by using the steering matrix SR. In this case, R{f}=SRR{fR}.

2.3.1 Local Image Orientation Estimation Based on M–D Riesz Repre- sentations

This section presents the straightforward extension of the tensor–based technique ex- plained in Section 2.2.5 from the 1st– to theNth–order. TheNth–order Riesz transform is composed ofM components with M = ^(N+2)(N₂ ⁺¹⁾ (see Section 2.2.2). This extension consists in building a matrixJ(x0)∈R^M×M with all the Riesz components (R₁, . . . ,R_M) as

J(x₀) =







R²₁{g∗f}(x0) · · · R₁R_M{g∗f}(x0)

... ...

R_MR₁{g∗f}(x0) · · · R²_M{g∗f}(x0)





. (2.7)

Following the procedure detailed in Section 2.2.5, an M–dimensional unitary matrix Ug

can be built from a sorted collection of the eigenvectors ofJ(x₀). Then,U_g can be used as a rotation matrix in R^M to align the Riesz transform. Equation 2.8 shows the final form of this aligned transform represented byR^MD.

R^MD{f}=U_gR{f} (2.8)

This procedure yields rotation–invariant texture descriptors but it does not rotate the Riesz components coherently, breaking then their directional inner structure. In this case, U_g cannot be used as a steering matrix following Equation 2.4 (see Section 2.2.6).

This method reorders the responses of the Riesz filters independently from the directional pattern that they are encoding, discarding the property of rotation–covariance as well as the ability to characterize the local organization of image directions. The Riesz filters (R₁, . . . ,R_M) do not have the same profile when N > 1. This was not the case in Equation 2.6, where the three filters only differed in their orientation and yet had the same profile (see Figures 2.2 and 2.3). Proposition 2.3.1 demonstrates that the result of applying Ug to a given Riesz vector Ralways produces the same aligned vector, independently of

2.3. LOCALLY–ORIENTED HIGHER–ORDER 3D RIESZ–WAVELET 13

the initial order of the components. This implies that the inner–structure of the Riesz filterbanks is not preserved with the application ofU_g.

Proposition 2.3.1. Let R(x0) ∈ R^M be the Nth–order Riesz transform at the position x₀ ∈ R³ of f(x₀), and let J(x₀) be the matrix defined in Equation 2.7. Let U be the matrix composed of the sorted collection of eigenvectors of J(x₀). Given a permutation matrixP∈R^M×M, we define R(x) =˜ PR(x),∀x∈R³. Let J(x˜ 0) be the matrix defined in Equation 2.7 for R. Then,˜ ∀ P, UR(x₀) = ˜UR(x˜ ₀), where U˜ is composed of the sorted collection of eigenvectors ofJ(x˜ ₀).

Proof. To simplify the notation, the position x0 is omitted.

It is straightforward that ˜J=PJP⁻¹. This expression corresponds to a change of the basis and the eigenvalues of the matrices ˜J and J are the same. Let Λ be the diagonal matrix containing the eigenvalues ofJand ˜Jordered in descending order. Then,

˜J=PJP⁻¹ =PU^TΛ(U^T)⁻¹P⁻¹=PU^TΛ(PU^T)⁻¹. (2.9) Since ˜J= ˜U^TΛ( ˜U^T)⁻¹, then ˜U^T =PU^T,i.e., ˜U=UP^T.Hence,

U˜R˜ =UP^TPR=UR, (2.10)

because by definitionP^T =P⁻¹ as Pis a permutation matrix.

The limitation of this approach is exemplified when two different patterns yield iden- tical overall filterbank responses but are distributed in components with different profiles.

In this case, aligning the Riesz components with theM–dimensional matrixUg produces the same vector ˜R for both texture patterns, even if they differ.

This method entails the risk of mixing image responses to Riesz components with different profiles, creating a rotation–invariant but not rotation–covariant descriptor,i.e., not preserving the directional relations between filters. The method only assigns the response of the filter with the highest energy to the first component of the Riesz transform, no matter which is the first component. The resulting feature vector is always the same for any sorting of the Riesz components. This technique is referred to asM–dimensional alignment in the following sections.

2.3.2 Local Image Orientation Estimation Based on Uni–Directional Riesz Components

This method only uses the Riesz coefficients corresponding to the uni–directional Riesz components (see Definition 2.2.1) to estimate local image orientations. These three compo- nents are orthogonal and encode theNth–order image derivatives along each 3D direction (^∂N

∂x^N₁ ,_∂x^∂NN 2

,_∂x^∂NN 3

). Their profiles are rotated versions of each other and encode the same type of texture patterns but along different directions. EveryNth–order Riesz transform withN >0 contains exactly three uni–directional Riesz components.

For this alignment, the tensorJ(x0) proposed in Equation 2.6 is constructed using the response at the position x₀ to these three filters. This tensor is referred to as J^Ud(x₀).

The corresponding rotation matrixU^Ud_g can be computed from the eigenvectors ofJ^Ud(x0) (see Section 2.2.5). Equation 2.4 yields the steering matrixS_UUd

g , that is M ×M. This steering matrix is used to align the Riesz components as shown in Equation 2.11.

R^Ud{f}=S_UUd

g R{f}. (2.11)

In this particular case, the estimation of the local orientation is solely based on the uni–directional components of theNth–order Riesz transform. All Riesz components are subsequently aligned with the M–dimensional steering matrix S_UUd

g . This technique is referred to asuni–directional alignment in the following sections.

2.3.3 Local Image Orientation Estimation Based on 1st–Order Riesz Components

The estimation of the local orientation described in Section 2.2.5 can be used to align higher–order filters. Using the 1st–order filters, the local orientations of the image are estimated by the technique explained in Section 2.2.5, yielding the matrix U_g. In this case, this unitary matrix is referred to as U^O1_g since only the 1st–order filters are used.

Applying then Equation 2.4, theM–dimensional steering matrixS_U^O1

g for theNth–order Riesz transform is computed. Then, theM–dimensional vector of Riesz components can be aligned applying this steering matrix (see Equation 2.12).

R^O1{f}=S_UO1

g R{f}. (2.12)

This technique is referred to as 1st–order alignment in the following sections. In this case, the local orientation is computed based on image gradients. Since the estimation of the local orientations is always based on the 1st–order Riesz components, it remains stable when the order of the transform increases. Thanks to the steerability property, the three uni–directional components of the 1st–order Riesz transform cover all 3D image directions.

However, this coverage is not complete when only using uni–directional components of the higher–order Riesz transform. In addition, the 1st–order Riesz transform of the image is required for this alignment, but since higher–order Riesz transforms can be obtained from recursions of the 1st–order Riesz transforms, this can be kept for further estimation of the local orientations.

2.4 Evaluation of the Proposed Alignments

The methods presented in this chapter were evaluated using the Reconnaissance de Formes, Analyse d’Images (RFAI) database⁴ of 3D synthetic textured images [164]. This is one of the few 3DST databases available with clear ground truth. It contains several datasets of 3DST images, along with the information regarding the textures that compose each volume. The datasets called Fourier and Geometric were used for the evaluation of the methods presented in this chapter. Later on, for the development of a texture–based atlas of the lungs, the available dataset for texture segmentation is also used (see Section 3.3.2 in Chapter 3).

2.4.1 RFAI Database

The RFAI database contains two datasets designed for texture classification, theFourier andGeometric datasets, both containing images of size 64×64×64 voxels. TheFourier dataset contains 15 classes of texture built from synthetic distributions in the Fourier domain. On the other hand, the Geometric dataset contains 25 classes of texture con- structed using random positioning of geometric shapes such as spheres, cubes, and ellipses.

4http://www.rfai.li.univ-tours.fr/PublicData/3D_Textures/3Dsynthetic_images_database.

html, as of March 8, 2018.