Automatic Annotation Tool to Support Supervised Machine Learning for Scaphoid Fracture Detection

Article

Reference

Automatic Annotation Tool to Support Supervised Machine Learning for Scaphoid Fracture Detection

FOUFI, Vasiliki, et al.

Abstract

The aim of this work is to develop and validate an automatic annotation tool for the detection and bone localization of scaphoid fractures in radiology reports. To achieve this goal, a rule-based method using a Natural Language Processing (NLP) tool was applied. Finite state automata were constructed to detect, classify and annotate reports. An evaluation of the method on a manually annotated dataset has shown 96,8% of total match.

FOUFI, Vasiliki, et al . Automatic Annotation Tool to Support Supervised Machine Learning for Scaphoid Fracture Detection. Studies in Health Technology and Informatics , 2018, vol.

255, p. 210-214

DOI : 10.3233/978-1-61499-921-8-210 PMID : 30306938

Available at:

http://archive-ouverte.unige.ch/unige:128922

Disclaimer: layout of this document may differ from the published version.

1 / 1

Automatic Annotation Tool to Support Supervised Machine Learning for Scaphoid

Fracture Detection

Vasiliki FOUFI^a1, Sébastien LANTERI^b, Christophe GAUDET-BLAVIGNAC^a, Pascal REMY^a, Xavier MONTET^c and Christian LOVIS^a

a Division of Medical Information Sciences Geneva University Hospitals and University of Geneva

b ESIEE Paris

c Division of Radiology Geneva University Hospitals

Abstract. The aim of this work is to develop and validate an automatic annotation tool for the detection and bone localization of scaphoid fractures in radiology reports.

To achieve this goal, a rule-based method using a Natural Language Processing (NLP) tool was applied. Finite state automata were constructed to detect, classify and annotate reports. An evaluation of the method on a manually annotated dataset has shown 96,8% of total match.

Keywords. Scaphoid fracture, radiology report, automatic annotation, finite state automata, Natural Language Processing (NLP).

1. Introduction

Until recently, classification of radiology reports and decision making has mostly been a manual process [1]. However, systematic reviews have shown that CDSS have improved practitioner performance in 64% of the studies [2] and that significantly improved clinical practice in 68% of trials [3]. The purpose of this study is to develop an automatic annotation tool to support supervised machine learning for scaphoid fracture detection. The scaphoid is the most frequently fractured carpal bone, accounting for 71%

of all carpal bone fractures. [4] Although most studies about fracture detection have been conducted on images [5], [6], [7], research has also been performed on narrative data. In particular, text classification methods on wrist x-ray reports for the identification of wrist fracture patients have been applied in [8]. A support vector machine (SVM) algorithm was able to identify fractures in free-text radiology notes, achieving an overall F-measure of 91.3%. Machine learning algorithms and a combination of stemmed token bigram features, negation features, and SNOMED-CT concept features related to morphologic abnormalities and disorders have been applied in [9]. In [10], a text search algorithm that classified radiology reports into the categories ‘fracture’, ‘normal’ and ‘neither normal nor fracture’ was developed. In this paper, automatic annotation of the presence, absence

1 Corresponding Author, Division of Medical Information Sciences, Geneva University Hospitals and University of Geneva, E-mail: [email protected].

© 2018 The authors and IOS Press.

This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0).

doi:10.3233/978-1-61499-921-8-210

or suspicion of a scaphoid fracture in free-text radiology reports written in French using a rule-based method based on morphosyntactic pattern recognition via finite state automata is presented.

2. Method

In this work, a rule-based tool that automatically detects, classifies and annotates wrist fracture in X-ray radiology reports is presented. In this scope, an information retrieval approach is followed based on morphosyntactic pattern recognition. More precisely, finite state automata via the Unitex NLP system [11] were constructed in order to detect and extract the targeted information, i.e. the expressions that refer to the presence, absence or suspicion of wrist fracture. Unitex is a corpus processing system based on electronic dictionaries and local grammars for corpus analysis and annotation. After the pre-processing phase (tokenization, sentence splitting and part-of-speech tagging), the system looks for the features described in the paths (Figure 1) and proceeds to the annotation task and attribution of the corresponding tag (yes, no or uncertain). In the present study, the corpus consists of 250 wrist X-ray radiography reports that were both manually annotated and automatically processed.

2.1. Manual annotation of reports

A dataset of 250 X-ray radiography reports was manually annotated with values ‘yes’,

‘no’ and ‘uncertain’ for reports where the findings lacked of clarity and were difficult to draw a conclusion. This manually annotated dataset served as a gold standard corpus for the development of the automatic annotation tool.

2.2. Construction of finite state automata

In order to detect text structures that describe the presence, absence or suspicion of wrist fracture, eight finite state automata were constructed via the Unitex NLP tool. As shown in Figure 1, the main finite state automaton consists of three major paths that correspond to the three basic values: ‘yes’ for the presence of a fracture, ‘no’ for absence and

‘uncertain’ for uncertain diagnosis. Each path recalls electronic dictionaries of simple and compound medical terms (e.g. fracture ‘fracture’, os scaphoïde ‘scaphoid bone’) and sub-paths with expressions of negation (e.g. pas de fracture ‘no fracture’).² The findings were categorized as positive, negative or uncertain if they represented one or more of the features described in the paths. Next, transducers were added to the automata in order to produce annotated outputs that categorize the reports into positive (<Fracture: oui>), negative (<Fracture: non>) or uncertain (<Diagnostic: uncertain>).

2 The importance of capturing negations from free-text to correctly identifying the presence or absence of fractures and other abnormalities is also pointed out in ⁹.

V. Foufi et al. / Automatic Annotation Tool to Support Supervised Machine Learning 211

Figure 1. Finite state automaton.

2.3. Synthesis of outputs

Unitex detects text structures at a high precision by analyzing the text at the sentence level. This means that for a single report, we could have more than one outputs, tagged with yes, no, and/or uncertain. A program was then developed in Python in order to automatically synthetize the various sentential outputs and give an overall conclusion.

For instance:

Fracture extra-articulaire de l’extrémité distale du radius de type A2 selon la classification A0 <Fracture = YES>

Pas d’autre fracture visible <Fracture = NO>

Final conclusion: <Fracture = YES>

3. Results

The work presented in this article is part of an ongoing research project. As a consequence, the dataset described above was used both to develop the rules and as a validation set. The finite state automata were applied on a dataset of 250 X-ray radiography reports. Some representative examples of the automatic annotation are displayed in Table 1:

Table 1. Examples of the automatic annotation of fractures.

Information retrieval Automatic annotation

Fracture transverse du tiers moyen de l’os scaphoïde ‘Transverse fracture of the middle third of the scaphoid bone’

Yes

Doute sur une fracture ‘Doubt over a fracture’ Uncertain Pas de lésion ostéo-articulaire ‘No osteo-articular lesion No

Afterwards, an automatic comparison between the automatic and the manual (gold standard) outputs was performed. The results are depicted in the following pie chart (Figure 2):

The tool achieved 96.8% of total matching between the automatic and the manual output and only 3,2% of the automatic outputs are not in accordance with the manual annotation. Figure 3 displays the percentages per value, ‘yes’, ‘no’, ‘uncertain’. At this point, the finite state automata are being enriched in order to localize automatically the fractured bone. In its current version, the tool manages to perform bone localization on 40% of the documents. Example: Fracture transverse non déplacée du col de l’os scaphoïde <Fracture = YES> <type = scaphoïde>.

4. Discussion

In this paper, we presented an automatic annotation tool that annotates wrist radiology reports written in French with ‘yes’, ‘no’, and ‘uncertain’ for the presence, absence or suspicion of fracture. Based on a dataset of 250 reports, finite state automata were constructed and tested in comparison with a gold standard dataset. This research is performed on textual data. It is well known that textual data constitutes a challenge for medical sciences since it contains valuable information difficult to handle because of the heterogenous and “unstructured” nature. For that reason, large datasets must be used to be able to predict and cover all possible structures and therefore construct finite state automata as complete as possible. However, typographic errors in reports influence the results of the automatic annotation tool. For instance, if the word fracture is miswritten (e.g. fracturre), the system is not capable of detecting the term. Equally, if the words are jointed without spaces between them, the system cannot recognize them. The next steps will be the automatic localization of the fractured bone. Then, the tool will be evaluated on a blind dataset. On manual annotation, a second expert should annotate the same set of documents, as multiple annotators are considered necessary in order to calculate an inter-annotator agreement and construct a validated gold standard annotated dataset.

5. Conclusion

The main focus of the present research is to build an automatic annotation tool for scaphoid fractures in radiology reports. The proposed method has been proven effective at identifying fractures in reports and an evaluation has shown promising results. In the next steps, the efficiency of the system at classifying radiology reports will be evaluated on a blind dataset. The ultimate goal is the development of a supervised machine learning algorithm for the automatic classification of large datasets of X-ray radiography reports.

Figure 3. Results per value.

Figure 2. Results of the automatic annotation.

V. Foufi et al. / Automatic Annotation Tool to Support Supervised Machine Learning 213

References

[1] A. Tiwari, S. Fodeh, S. Baccei, M. Rosen, Automatic Classification of Critical Findings in Radiology Reports, Proceedings of The First Workshop Medical Informatics and Healthcare held with the 23rd SIGKDD Conference on Knowledge Discovery and Data Mining, PMLR 69 (2017), 35–39.

[2] A.X. Garg, N.K. J. Adhikari, H. McDonald, Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review, JAMA 293 (10) (2005), 1223–1238.

[3] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach, Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success, BMJ 330 (7494) (2005), 765. doi:10.1136/bmj.38398.500764.8F (published 14 March 2005).

[4] C.A. Boles, Scaphoid Fracture Imaging: Practice Essentials, Radiography, Computed Tomography (2018).

Available: https://emedicine.medscape.com.

[5] J.E. Burns, J. Yao, H. Muñoz, R.M. Summers, Automated Detection, Localization, and Classification of Traumatic Vertebral Body Fractures in the Thoracic and Lumbar Spine at CT, Radiology 278 (1) (2016), 64–73.

[6] I. Hacihaliloglu, R. Abugharbieh, A. J. Hodgson, R. N. Rohling, P. Guy, Automatic bone localization and fracture detection from volumetric ultrasound images using 3-D local phase features, Ultrasound Med Biol 38 (1) (2012), 128–144.

[7] R. Tadeusiewicz, M.R. Ogiela, Picture languages in automatic radiological palm interpretation, Int. J. Appl.

Math. Comput. Sci. 15 (2) (2005), 305–312.

[8] B. de Bruijn, A. Cranney, S. O’Donnell, J. D. Martin, A. J. Forster, Identifying Wrist Fracture Patients with High Accuracy by Automatic Categorization of X-ray Reports, J Am Med Inform Assoc 13 (6) (2006), 696–698.

[9] G. Zuccon, A.S. Wagholikar, A.N. Nguyen, L. Butt, K. Chu, S. Martin, J. Greenslade, Automatic Classification of Free-Text Radiology Reports to Identify Limb Fractures using Machine Learning and the SNOMED CT Ontology, AMIA Jt Summits Transl Sci Proc 2013 (2013), 300–304.

[10] B. J. Thomas, H. Ouellette, E. F. Halpern, D. I. Rosenthal, Automated computer-assisted categorization of radiology reports, AJR Am J Roentgenol 184 (2) (2005), 687–690.

[11] S. Paumier, Unitex 3.1, User Manual. Available: http://unitexgramlab.org. [Accessed: 03-Jul-2018].