Towards Weakly-Supervised Medical Image Segmentation of Organs at Risk Using
Deep Convolutional Neural Networks
Rosana El Jurdi(1,2), Caroline Petitjean(1)
,
Paul Honeine(1),
Fahed Abdallah (2)(1)
LITIS Lab, Université de Rouen Normandie, France
(2)
Université Libanaise, Hadath, Beyrouth, Liban
Introduction
2
Medical Image segmentation
the process of isolating anatomical objects of interests for analysis and treatment.
Cancer domain:
treatment through irradiation planning requires accurate delineation of target tumorized areas and organs at risk on CT image.
manually performed by doctors and specialists within the domain
Solution: Automatic medical image segmentation for organs at risk
Introduction
Soft Organs At Risk (OAR) in CT-images:
hard to detect
same grey levels with neighboring tissues
low contrast and high noise levels
encompass segmentation ambiguities
4
Convolutional Neural Networks:
able to learn spatial features as well as intensity based features
known for their success within natural image segmentation
Inadequacies:
rely on large amounts of annotated data in order to gain their generalization ability
unavailability of annotations within the medical domain
partial annotations (bounding boxes, seeds) are more common
Objective:
pushing forward to weakly supervised learning the problem Medical Image Segmentation of Thoracic Organs at Risk in CT-images
Highlights:
exploit a selection of weakly supervised models present within the state of the art
propose our own weakly supervised model for medical image segmentation
comparative study of the multiple methods utilized
Introduction
Related Works in Weakly Supervised Image Segmentation
Computer Vision Model
Fully Supervised Model
Correction Network Label Estimate
State of art Model Computer vision Label Est. Training Correction Network Simple Does it by Khoreva et al.
(CVPR, 2017)
GrabCut like algorithms Unmodified Prior cues BoxSup by Jifeng et al.
(ICCV, 2015) Unsupervised region
proposal methods Modified to choose
suitable labels Fixed masks DeepCut by Rajchl et al.
(IEEE Medical Image Trans, 2017) Replace GMM in GrabCut
with CNN EM method: alternates between finding label estimates and updating model parameters
Related Works in Weakly Supervised Image Segmentation
6
Computer Vision Model
Fully Supervised Model
Correction Network Label Estimate
State-of-the-art models Computer vision Label Est. Training Correction Network Simple Does it by Khoreva et al.
(CVPR, 2017)
GrabCut like algorithms Unmodified Prior cues BoxSup by Jifeng et al.
(ICCV, 2015) Unsupervised region
proposal methods Modified to choose
suitable labels Fixed masks DeepCut by Rajchl et al.
(IEEE Medical Image Trans, 2017) Replace GMM in GrabCut
with CNN EM method: alternates between finding label estimates and updating model parameters Intensity Based algorithms
Problem: Intensity based algorithms proven to be insufficient
Reason: lack of contrast between soft organs within our CT-images
Inference
CT BB
Train Set
CT BB GT
Test
Ancillary Model
Label estimates
train train
CT BB 𝐺𝑇
Train
Primary Model Train
Evaluation CT: CT-IMAGE BB: Bounding Box 𝑮𝑻: Estimated
Ground Truth GT: Ground Truth
Framework : Workflow
Framework : Workflow
8
Inference
CT BB
Train Set
CT BB GT
Test
Ancillary Model
Label estimates
train train
CT BB 𝐺𝑇
Train
Primary Model Train
Evaluation
Circular Generator
CT: CT-IMAGE BB: Bounding Box 𝑮𝑻: Estimated
Ground Truth GT: Ground Truth
Inspired by: Mostafa et al. “Weakly Supervised Semantic Image Segmentation with Self-correcting Networks.” CoRR abs/1811.07073 (2018)
Framework : Workflow
Inference
CT BB
•Weakly Supervised Train Set (1244 samples)
•Tiny Fully Supervised Train Set (200 samples)
CT BB GT
Test
Ancillary Model
BB-Unet Label estimates
train
CT BB 𝐺𝑇
Train
Primary Model U-net Train
Evaluation
train
CT: CT-IMAGE BB: Bounding Box 𝑮𝑻: Estimated
Ground Truth GT: Ground Truth
CT BB GT
10
Framework : Unet Model Structure
Filter
Max pool with ratio d d1
d2
d3
d
Intersection of 2 inputs
Framework: BB-Unet Model for Label Estimation
Filter
Max pool with ratio d d1
d2
d3
d
Intersection of 2 inputs
256 256
Setup: Epochs: 550, Loss: dice, Batch size= 5 12
Framework: BB-Unet Model for Label Estimation
Segmentation of Thoracic Organs at Risk (SegTHOR) Dataset
13
# Patient # Heart Slices
Train Set 36 1477
Validation Set 4 155
Test Set 20 693
Total 60 2325
The LITIS lab released the dataset at the 16th IEEE International Symposium for Biomedical Imaging (ISBI 2019) .
Four thoracic organs at risk were manually segmented by an expert radiotherapist
Work done so far on SegTHOR is purely supervised
Primary focus is on segmentation of the heart
Preprocessing steps :
• cropped to their contour boundary
• resize images to 400x400
• pixel values clipped -1000 , 3000
• normalization by mean and standard deviation
• no data augmentation was performed
Results
14
Dice similarity percentage # Mean dice # standard dev. # Max dice # Min dice
Circular Naïve Est. 54.43 22.46 77.91 0.0
Grabcut Like Model 64.37 32.67 92.93 0.0
Proposed models BB Label Est. 83.19 (4) 13.322 96.364 9.091
BB∩Image Label Est. 84.47 (3) 5.97 95.96 60.62
CC Label Est. 91.69 (1) 11.27 98.77 23.82
CC∩Image Label Est. 86.79 (2) 15.03 98.54 5.12
Unet Full Supervision 91.53 11.12 98.79 10.67
BB-Unet Full Supervision 95.29 3.51 98.55 75.65
Naïve Model
54.43
Grabcut Model
64.37
BB Model
83.19
BB∩ Im Model
84.47
CC∩Im Model
86.79
CC- Model
91.69
BB-Unet Full Sup.
95.29
Conclusion and Future Work
Conclusion:
Proposition: a new weakly supervised framework for OAR segmentation of CT images
Proposed the BB-Unet model as ancillary trained in order to infer masks from bounding boxes
Trained a Primary Unet model on label estimates provided by the ancillary model to perform the segmentation task
Future Work:
Consider other training approaches of the BB-Unet
Extend binary segmentation into a multi-label one
Address the class-imbalance problem
Explore 3D weakly supervised algorithms
