MobileDr : adapting the BabelDr medical translator for improved usability on mobile devices

(1)

Master

Reference

MobileDr : adapting the BabelDr medical translator for improved usability on mobile devices

SEKERA, Isabelle

Abstract

In an international city like Geneva, doctors often have to treat patients with whom they share no common language. In the absence of professional interpreters, the web application BabelDr plays a key role. It is currently used at the HUG mainly on desktop computers.

However, in some situations, doctors need to move around and use mobile devices. This work focuses on the adaptation of BabelDr to a more graphically intuitive, efficient and user-friendly text interface. The new resulting web application is called MobileDr. To obtain opinions on the usability of MobileDr, eight doctors evaluated both BabelDr and MobileDr on smartphones.

Feedback was provided throughout identical questionnaires. Results show a considerable increase in the general satisfaction rate in favor of MobileDr.

SEKERA, Isabelle. MobileDr : adapting the BabelDr medical translator for improved usability on mobile devices. Master : Univ. Genève, 2021

Available at:

http://archive-ouverte.unige.ch/unige:151202

Disclaimer: layout of this document may differ from the published version.

1 / 1

(2)

Master Thesis in Multilingual Communication Technology

MobileDr

Adapting the BabelDr medical translator for improved usability on mobile devices

December 20, 2020

Isabelle Sekera

Faculty of Translation and Interpreting

Supervisor: PhD Johanna Gerlach, Faculty of Translation and Interpreting Jury: Prof. Pierrette Bouillon, Faculty of Translation and Interpreting Dr. Hervé Spechbach, Geneva University Hospitals

University of Geneva

(3)

1

(4)

2

ACKNOWLEDGEMENTS

It is my pleasure to thank the director of the Department of Translation Technology, Prof.

Pierrette Bouillon, and my supervisor PhD Johanna Gerlach, for allowing my participation on the ongoing BabelDr project. It was an exceptional experience I was proud to be part of.

I would like to express my sincere gratitude to my supervisor PhD Johanna Gerlach for her constant support, patience and precious advices. Her availability helped me very much to remain focused during the pandemic situation which was sometimes not easy for me to get through. I particularly appreciated the kindness she showed me at every contact.

Many thanks to all participating physicians to have taken their time to evaluate both MobileDr and BabelDr web applications. Their remarks were of a valuable contribution for the whole project.

I am immensely grateful to all my family for their permanent encouragement and my friends for their warmhearted amiability.

(5)

3

INTRODUCTION

In an international city like Geneva, Switzerland, about 40% of inhabitants are foreigners (OCSTAT, 2020). The Geneva University Hospitals (HUG) indicate 8% of patients do not speak French at all and 30% speak French as a second language without necessarily having a sufficient level to understand a discussion on a medical topic (Hudelson, Dominicé-Dao &

Durieux-Paillard, 2013). Thus, chances are high for doctors to come across a patient with whom communication will be difficult to maintain because of the language barrier.

In the absence of professional interpreters, two major problems arise. First, approximate language knowledge can lead to misunderstandings, can cause the doctor to do unnecessary examinations and can possibly put the patient’s health at risk. Second, using a cloud based automatic translation tool (no matter its accuracy) does not guarantee the confidentiality required before, during and after a consultation.

Automatic translation is being improved all the time to facilitate communication between people who do not share a common language. Google Translate is one of the most popular online tools used worldwide. However, it has been observed that such tools, although based today on a neuronal system (Poibeau, 2019) can have limitations.

In specialized fields such as medicine, where particular terminology and exact formulations are required, automatic translation tools often give inaccurate or even wrong results. A study directed by Pr. Pierrette Bouillon from the Faculty of Translation and Interpreting (FTI) at the University of Geneva (UNIGE) and Dr. Hervé Spechbach from the HUG emphasizes that Google Translate has about 60% accuracy in the medical field, making the tool unreliable for usage by doctors (Patil & Davies, 2014; Bouillon, Gerlach, Spechbach, Tsourakis & Halimi, 2017; Spechbach et al., 2019).

Mobile applications downloadable from Google Play Store (Android) or from App Store (iOS) intended for a medical consultation do already exist. These applications store their data on servers external to the HUG, which does not satisfy the confidentiality requirements of the HUG, similarly to Google Translate (RTS, 2019).

(8)

6 For these reasons the FTI and the HUG collaborated to develop BabelDr, a web application which meets the required criteria of translation accuracy and consultation confidentiality.

Indeed, it contains a large pool of about 11’000 pre-translated sentences and it runs on a local HUG server. It is currently available in French for consultations with patients whose first language can be Albanian, Arabic, Dari, Farsi, Spanish, Tigrinya or Swiss sign language.

Context

Medical staff often needs to be more flexible and has to move within the hospital, for an emergency or to the patient’s bedside, or during outpatient services for an “in the field”

intervention.

One could think that every ill person feels comfortable coming to a hospital or a health facility to receive medical care as soon as possible. However, interviews have shown that it is not always the case; a large number of people in need of medical help but not speaking good enough French will avoid seeking assistance (HUG, 2010). This can worsen individual health or put in danger other people in case of a transmissible infection or disease. Also, if diseases are detected at an earlier stage, an increase in costs could be avoided if treated earlier (Wolff, Besson, Holst, Induni & Stalder, 2005).

Studies say that reasons for delay in seeking the care can be numerous: no health insurance, no permit or official documents, psychological fragility, misinformation, cultural differences, fear of authorities and others (Wyssmüller & Efionayi-Mäder, 2011; Santé sans-papiers, 2014). Moreover, regardless of their background, some people are scared to be hospitalized, so preliminary information and appropriate communications appear very helpful (Roy, 2003).

For undocumented migrants, the fear to be arrested or stigmatized due to non-valid documents or situation renders them vulnerable. They often prefer to remain anonymous, so their number can hardly be statistically estimated (Bilger & Hollomey, 2011). The choice of non- stigmatizing words could be determinant to reach a therapeutic alliance (Wenk-Clément, Singy & Bodenmann, 2020).

(9)

7 Evidently, it would be useful to facilitate communication between medical staff and these patients outside of the hospital as well, for whom specialized structures in Geneva were created for them. The HUG provide the Consultation ambulatoire mobile de soins communautaires (CAMSCO) [Mobile ambulatory consultation of community care] and the Programme de santé migrants (PSM) [Migrants health program]. Medical staff from these services can move around within a town.

Moreover, several associations propose to treat any person anonymously and confidentially.

Examples are: Quai9 (Quai9, n.d.) for drug dependence and Carrefour Addictions (Carrefour addictions, n.d.) for those suffering from alcohol, tobacco and excessive gambling, Centre LAVI (LAVI, n.d.) for victims of an offense, VIRES for offenders (VIRES, n.d.). The staff of these institutions could be confronted with medical problems of clients who do not speak their language and who avoid hospitals, and could facilitate the acceptation of a medical intervention. Therefore, the medical staff needs light and portable devices to communicate in foreign language and handle these special situations as fast as possible.

For extraordinary situations such as the recent COVID-19 pandemic, there is a need for rapid communication with allophone patients which allows maintaining social distancing. Portable devices become almost necessary and avoid the risk to expose interpreters to danger.

Motivation

BabelDr has proved to be well suited for the HUG emergency service. As a web application, it can be used on desktop or laptop computers in a doctor’s office, as well as on mobile devices. Yet, it was felt that its text interface is not structured optimally for mobile devices.

On small screens, there is a lack of space, so it would be useful to think about how to structure the available sentences of the original pool to make the navigation more user- friendly.

The use of a speech recognition tool, which is integrated in BabelDr, could be problematic in situations where there is a lot of noise due to the surrounding people and environment.

Moreover, because of their foreign accent, some non-native French speaking doctors could

(10)

8 have difficulties to use the tool. Therefore, it seemed appropriate to favor the adaptation of its text interface.

A small project linked to BabelDr was proposed during a course at the FTI of UNIGE. The whole concept of BabelDr raised a big interest, because it enables doctors to communicate with allophone patients through a device playing the role of an intermediate instead of an interpreter. One of the current development aspects of the whole project was to adapt BabelDr to mobile devices, as the HUG requested. The opportunity to be part of such a human adventure was taken.

The aim of this work was to adapt BabelDr, and more specifically the text component, for mobile devices and have it evaluated by different medical practitioners to test its usability.

This new application is called MobileDr throughout this thesis.

Research questions

The two research questions explored in this thesis are:

 Which web application (BabelDr or MobileDr) is the best suited for use on a mobile device, in terms of ease of usage and learning curve?

 Do medical practitioners working in different contexts agree on the features and functionalities required for a mobile medical translation application?

In order to answer these research questions we have carried out an evaluation with eight doctors from different medical settings, using both systems, and collecting opinions by the means of System Usability Scale questionnaires (SUS) (Bangor, Kortum & Miller, 2008).

Thesis structure

This Master thesis is structured as follows:

(11)

9 Chapter 1 describes the current state of art of medical translation applications. Four currently available smartphone applications, Care to Translate, Canopy Speak, MediBabble Translator and MedTranslate, are presented before introducing BabelDr.

Chapter 2 presents the new application MobileDr, whose development was based on BabelDr, focusing on the text interface. The main new features are a new layout, reduced number of sentences and sentence classification. The proposed structure follows as best as possible a classical medical consultation and the subdivision in medical specialties is explained from the medical point of view.

Chapter 3 gives an overview of the technologies involved in the implementation of the new application structures.

Chapter 4 presents the evaluation methodology and shows the results of the doctors’ usability evaluation based on the System Usability Scale (SUS) as well as their personal opinion about the new application. A complete analysis of results will be performed.

Chapter 5 proposes possible future improvements for MobileDr.

(12)

10

1 CURRENT STATE OF ART

Medical translation applications do already exist, whether on Android or on Apple operating systems. Section 1.1 presents four of the already existing mobile medical translation applications on the market: Care to Translate, Canopy Speak, MediBabble Translator and MedTranslate. Section 1.2 describes the BabelDr web application. In Section 1.3, the advantages and disadvantages of the main aspects of these applications are exposed, as well as the proposed features for MobileDr for comparison.

Other medical translation applications exist on iOS, like Universal Doctor Speaker, DrTranslate, HealthTrans, iTranslate Medical, The Medical Translator, etc. We chose to analyze only the applications which work, have recorded sentences and are free.

MedTranslate is a special case because there are no sentences to choose from and it works only with speech recognition and machine translation. Since this is a feature shared with BabelDr, it was interesting to consider this application as well.

It is recommended to download and try the applications. The applications were not tested for making a comparison in the first place, but to acquire an overall feeling of usage as a random user would have. After this, twenty features were chosen for their comparison. It should be noted that none of these applications meets the requirements of confidentiality of the HUG.

1.1 Presentation of existing medical translation applications

Since the specific field is medicine, there should be no possibility of translation errors, as it could seriously impact the patient’s diagnosis. Most applications presented below seem to work based on the same fundamental approach: no machine translation, but pre-translated selectable sentences. We can easily verify this by choosing several times long or difficult sentences; the translations shown will always be the same. However, the available languages, features and layout vary from one application to the other and this will determine the user’s preference for a specific application.

(13)

11 We will now introduce the four medical translation applications (Care to Translate, Canopy Speak, MediBabble Translator, and MedTranslate) to give an overview of what already exists.

Images in this chapter are screenshots of all these applications displayed on a smartphone.

1.1.1 Care to Translate

The application Care to Translate originated as a non-profit student project in Sweden and was developed as a downloadable application with a payed extension. The user does not need to create a profile to use the application as such. Another version of this application, called CTT Clinic, was probably developed only for professional healthcare providers. Indeed, the user must login with an already existing account which unfortunately cannot be directly created, so it was not possible to analyze it in this study. In what follows, the focus is only on the Care to Translate application.

When opening the application, the doctor can choose among 34 languages on both the doctor and the patient side (see Image 1).

Image 1 – 3 screenshots of possible settings, choice of type of user (doctor or patient) and choice of languages

(14)

12 On the doctor side, the sentences are classified in domains. Each domain consists of a domain name and a pictogram (see Image 2), which can help the doctor both linguistically and visually to situate himself within the structure. Since there are 30 domains, they cannot be all displayed on one single phone screen. This implies a scrolling up and down to see all available domains. When clicking on a domain, its sentences appear (see Image 2).

Image 2 – Doctor side: 4 screenshots of all 30 domains, sentences in a given domain and the warning sign

No order in the sentences has been found when going through the sentences of different domains. In domains like Presentation, Family, Front desk, etc., a disordered structure might not necessarily negatively affect a consultation. However, a chaotic order of sentences in domains of medical specialties (Psychiatry, Gynecology, etc.) could impact the consultation process with a doctor who might lose their train of thought (see Image 2).

When analyzing the sentences of a more evident specialty like Dental care, the lack of sentence order is clearly noticeable: the sentences range from a disease announcement (you have a gum disease) to naming body elements (the inside of your cheek), passing by a therapeutic intervention (I need to extract a tooth) (see Image 2). Evidently, all these sentences concern dental specialty. However, it is not so obvious to know in advance which

(15)

13 sentences can be found in this domain. The only clear aspect is that all of these sentences concern teeth.

From a medical perspective, these aforementioned three sentences could also be found in other domains than just the Dental Care domain: you have a gum disease in Diagnosis and information domain, the inside of your cheek in The body domain and I need to extract a tooth in Surgery domain. It is not clear whether every doctor would think the same way about the classification of sentences. In particular, during an emergency, the doctor will not have the time to navigate within the domains when searching for each question.

From a sentence formulation perspective, some available sentences are only single words or clauses. It is unclear when exactly a doctor would need to use the gums or a clause such as the inside of the cheek without turning it into a question or an affirmation. Indeed, it looks more like an answer to a question a patient might have asked. This shows that full sentences are not separated from clauses and words which seem to bear a resemblance to an answer.

It is worth mentioning that some sentences have a “warning sign” next to them, for example the sentence are you allergic to antibiotics? in the Medication domain (see Image 2). When clicking on such sentences, a special menu appears with the message “make sure that both question and answer are understood correctly”.

This might be for two reasons. Either the sentence contains a medical word which might be difficult to understand for a layperson, or the sentence could be wrongly understood depending on the context. Typically, confusion in the meaning can be created if a word exists in different homonym forms as homographs (words with same spelling but different meaning) or homophones (words with same pronunciation but different meaning) in a given language.

From this hypothesis, it can be concluded that a doctor will be warned (by the “warning sign”

and message) that the sentence he is about to use could possibly be problematic in its meaning. However, no concrete solution is proposed to him; even when asking a patient do you understand? and receiving a “yes” response, the doctor has no assurance that the patient understood the sentence, especially a problematic sentence.

From the patient side, a different set of 21 domains with corresponding pictograms is displayed, enabling him to express more than nodding and gesturing (see Image 3). Although

(16)

14 the possibilities are limited compared to the doctor side, the patient is able to ask for Orientation, express his Feelings, Ask for help, etc. With the number of domains, the patient also has to scroll down and up to see the whole domain set. It is unclear whether an elderly, suffering or confused patient would be able to confront this amount of information, given that he could only be familiarized with the application at the moment he enters the hospital.

From a sentence formulation perspective, there are single words (see Image 3) which could this time enhance confusion on the doctor side. For example, if the patient wants to express the fact that his grandfather had died from cancer and he does not find that sentence, he might just select grandfather and the doctor would probably not understand the intended meaning.

Image 3 – Patient side: 4 screenshots of 21 domains and examples of sentences in two domains.

In conclusion, the application Care to Translate has a doctor side and an independent patient side, which are independent, and provides 34 languages on both. Sentences are classified in domains consisting of a domain name and a corresponding pictogram. Domains can be of different types: specialties, diagnosis, posology, intervention and many others. In addition of full sentences, one can also find individual words and clauses. Potential usability issues related to the classification of sentences and numerous domains have been detected.

(17)

15 1.1.2 Canopy Speak

The digital health company called Canopy Innovations was founded in 2010 by Bill Tan and is head-quartered in New York, USA. One of its services is the Canopy Speak smartphone application, already used in some hospitals and healthcare systems. Its only source language so far is English. This implies it can be used efficiently only by healthcare providers who studied a medical field in English.

In order to use the application, one must create a profile and provide details such as personal information, work institution and the held occupation (see Image 4). It is also possible to contact directly an interpreter. This functionality has not been tested. The application consists of only the full doctor side with 15 target languages (see Image 4).

Image 4 – 3 screenshots of profile creation, language selection and domains

Canopy Speak has 10 domains (see Image 4) and up to 3 subdomain branches in a domain. To illustrate this branch structure, the way to reach the GI (gastrointestinal system) subdomain, one has to go through the path: Internal Medicine domain > History > HPI (history of the present illness) > GI (gastrointestinal system) (see Image 5). The only visibly traceable part of the path is the domain (here Internal Medicine) and first subdomain History (visible on the

(18)

16 blue toolbar in Image 5). For the subsequent subdomains, it is impossible to know the rest of the path retrospectively.

Image 5 – 3 screenshots of domain Internal Medicine, then subdomain branches History > HPI > GI

Selecting Abdominal Pain in the GI subdomain branch leads to a set of 56 sentences. At least three problematic issues were found at the very bottom of the list (see Image 6). When these questions are selected, their respective audios are played. The audios have clearly not been recorded by human voices, but rather by a text to speech engine with a slow unnatural pace, and it can be assumed that long sentences are constructed from short record segments.

Examples of problematic pronunciation for French as a target language follow:

Image 6 – 3 screenshots of Internal Medicine > History > HPI > GI > Abdominal Pain: wrong pronunciation

(19)

17 In the first example, the voice makes mistakes if there is a typography error in the translation.

For instance, in the sentence do you have a history of any sexually transmitted infections?

translated as avez-vous déjà eu des maladies sexuellement trnsmissibles ? (see Image 6), the underlined misspelled adjective “trnsmissibles “ comes out letter by letter as “t-r-n-missibles”

instead of “transmissibles” making it confusing.

Another incorrect pronunciation can be found in the sentence have you had an increased appetite?. It is translated as votre appétit a-t-il augmenté ? (see Image 6). The spelling of the underlined set of words is correct, but the voice pronounces distinctly almost every part of it:

“a tiret t il” (“tiret” means “hyphen”) comes out, yet it should be spoken out all at once as

“atil”.

As a third example, consider the sentence have you recently had abdominal surgery or any scope procedures?, translated as avez-vous récemment subit une chirurgie abdominale ou tout autre type d’intervention sur cette partie du corps ? (see Image 6). The noun “intervention” is spelled correctly, but the audio pauses while pronouncing (“interven-tion”). This shows that a word could be cut in the middle, even if spelled correctly, as if it was not known to the system. Furthermore, “tion” is pronounced with “t” whereas in this case, “tion” should be pronounced with “s” (“sion”). On the contrary, a misspelled word can be pronounced correctly. Indeed, in the same sentence, the conjugated verb “subit” is incorrectly spelled (it should be “subi”), but is still pronounced with accuracy. Based on our observations, the quality of the translations and audios has possibly not been validated.

In the same branching example Internal Medicine > History > HPI > GI > Abdominal Pain, it appears that the doctor has to click at least 5 times to select the desired sentence. Even when going through so many subdomains, he arrives to a final set of 56 sentences.

On top of that, it is worth mentioning there are also medical abbreviations in the application.

Hence, if the doctor doesn’t have a solid English medical background, it might be challenging to navigate through the subdomains without having to look for the acronyms. Moreover, as mentioned above, it is visually difficult to know in which subdomain we are situated at a given moment except for the domain and first subdomain (dark blue toolbar in Image 5).

(20)

18 There is a limited patient interface opened by selecting the three dots menu, which appears on the right-hand side of the selected sentence (see Image 7). In this menu, selecting Patient Response gives the patient a handful of sentences to choose from. The patient can say what he feels, needs or that he wishes to ask a question. Each of those options has its own set of subdomains, and the doctor can ultimately react to the answers at the end of the path. The options for the doctor to react are, however, very limited. Finally, it seems very easy for the doctor to overlook these options as the layout is not really user-friendly.

Image 7 – 4 screenshots of choosing to add patient’s response, response possibilities and doctor’s follow-up

In conclusion, Canopy Speak proposes 15 target languages and a subdivision in 10 domains.

The subdomain branches are numerous. Medical words are presented as acronyms. The recorded voice is clearly synthetic, with pronunciation mistakes. There is a very limited interface for the patient to express himself with restricted choices. The doctor has only a few options for the response.

(21)

19 1.1.3 MediBabble Translator

MediBabble Translator is only available on iOS and was created by NiteFloat Inc. It offers a limited choice of only English and Spanish as both target and source languages (see Image 8).

Even though there should be a possibility to download other languages, they failed to load when testing the application. The main menu presents 9 domains and each domain has a different number of subdomains (see Image 8).

There is an implicit separation into four categories. Three of them are explicit: Introductions

& Explanations, Physical Exam and Follow-up Questions. The remaining domains fall under the final category representing the medical history (see Image 8). On the right side of the screen the acronyms of their respective domains are presented when available (see Image 8).

Image 8 – 3 screenshots of the choice of languages, the domains, and an example of subdomains

When selecting one of these medical history domains, the History menu appears where the domains in the form of acronyms are visible on the top navigation bar (see Image 9). When selecting the different subdomains of History, some subdomains lead directly to a selection of sentences, while others require navigation through another branch of subdomains to reach the sentences. Some sentences have subtitles (for example Tobacco use) and some do not (see Image 9). The navigation is not very intuitive.

(22)

20

Image 9 – 3 screenshots of sentences examples with and without subtitles, and option to rotate the screen

When selecting a sentence, there is a possibility to rotate the phone of 90° and to obtain an enlarged version of the translation, probably to show it to the patient in case the audio record was insufficient to comprehend (see Image 10). In particular, it could be useful for an elderly or visually impaired person. In such cases however, it could be inconvenient for the doctor since he would have to rotate the phone for each sentence during the consultation.

On top of that, there is a history of recently played sentences (see Image 10), that probably is meant to remind which sentences were already used. It cannot be cleared inside the application. One has to reinstall the application to clear the history. This seems rather impractical.

To improve usability, several changes might be considered. For example, to create a separate page for each translation, write it in big characters, keep the phone straight and allow history clearing in the application.

(23)

21 Image 10 – 3 screenshots of selected question, screen rotation to get the translation and history

In conclusion, MediBabble Translator offers so far only English and Spanish as source and target languages. It consists of 9 domains, each of them subdivided in a different number of subdomain branches. Some sentences have describing subtitles. It is possible to rotate the screen and see the translated sentence in a bigger size. There is also a history of the last translated sentences, but there is no way to clear it in a simple way. MediBabble Translator works only on iOS.

1.1.4 MedTranslate

The last tested application is particularly interesting because, to our knowledge, it is the only medical translation application (except BabelDr) using speech recognition and direct translation. It was developed by BYU Creative Works and is only available on iOS. Its instructions (like Hold to Record Voice) are only in English, but the application offers 35 source and target languages (see Image 11). There are neither domains, nor choice of sentences, but rather only the speech recognition and automatic translation tools.

(24)

22

Image 11 – 3 screenshots of main menu with speech recognition tool, languages and recognized sentence

A random medical question in French was tried and perfectly recognized (see Image 11):

avez-vous mal quelque part? [do you feel pain somewhere?]. The speech recognition tool worked remarkably well, but the translation tool converted the previous sentence into do you hurt somewhere?.

One of the buttons appearing below on the screen enables to correct the translation by typing the right one on the virtual keyboard and then it is possible to record the new translation (see Image 12). We corrected the sentence to are you hurt somewhere?. However, the tool recognized “heard” instead of “hurt” and rated the record as incorrect, displaying a red

“unlike” icon, probably because it was not pronounced by a native English speaker (see Image 12). This was a try from French to English, which is a quite common combination.

To test the speech recognition and the translation tools with a less common source language, we used the Czech language. The sentence in Czech was pronounced by a native speaker and was perfectly recognized (see Image 12): bolí vás něco? [literally: is something hurting you?].

Its translation was: does it hurt somewhere?, which is grammatically and syntactically correct.

There is also the possibility to clear history inside the application (see Image 12).

(25)

23

Image 12 – 4 screenshots of French-English recognized sentence, its new recording, Czech example and history

To see if the recording would work better from French to Czech, we tried to say in French quand avez-vous bu pour la dernière fois ?, which was translated to Czech as kdy jsi naposledy pil? [when did you drink last time?]. This is correct, but in Czech it is an informal way to address a man, whereas we wanted a formal way to address a woman. The sentence was corrected to kdy jste naposledy pila? and was then orally re-recorded. It was also perfectly recognized, probably because it was pronounced by a native speaker (see Image 13).

Image 13 – 3 screenshots of French-Czech example, correction and recording rated as correct

(26)

24 In conclusion, MedTranslate is different from the three above applications in the fact that it does not use pre-translated sentences, but is based on a combination of a speech recognition tool coupled to a translation tool. It offers 35 source and target languages. There is the possibility to correct by hand and re-record a sentence if it was not well translated. It can recognize the new right or wrong pronunciation during a new trial. There is history option to see the last used sentences and a simple way to clear it. The usage of correcting and re- recording a sentence for the target language is rather questionable, because one needs to be almost native in that language to correct a mistake, and re-record the right sentence with a recognizable accent.

1.2 Presentation of BabelDr

The BabelDr web application (Bouillon et al., 2016) is being developed by Pr. Pierrette Bouillon and her team at the FTI in UNIGE, in collaboration with Dr. Hervé Spechbach and his team from the Geneva HUG. The project was mostly financed by the Fondation privée des HUG [Private Foundation of the HUG] and received the price called “Prix Innogap” from Unitec. The application is already being used at the HUG where each doctor has his own account linked to medical files of his patients. Other users are currently not obliged to create an account and can use the username “babeldr2” and password “babeldr” (see Image 14). The doctor selects the patient’s language and gender, and chooses the domain which to begin with.

Image 14 – 4 screenshots of login, main page, choice of domains and choice of patient’s language

(27)

25 When the domain has been selected, the doctor can choose either to use the speech recognition tool, or to find the desired sentence by himself in the list underneath (see Image 15). When the speech recognition tool is clicked upon (see the well visible microphone icon in Image 15), it records the speech input of the doctor. If a sentence is recognized, it appears within the first white rectangle. The doctor then decides if the recognized sentence resembles enough the sentence he spoke out, and if not he can select among other proposed sentences.

After validating the sentence with a click, the software leads to a separate page where the translation appears in big characters together with all answer options (see Image 15). There are three possible answers, namely “✓”, “✗” and an image with “?”, or a rectangle where numbers or others types of answers can be entered (see Image 15).

Image 15 – 4 screenshots of speech recognition tool running and identifying the sentence, and answers examples

In the list of clickable sentences, some items have a blue plus d’options… [more options…]

possibility, which displays additional similar sentences that could be considered (see Image 16).

The doctor can also see the last translated sentence (dernière phrase traduite) and the patient’s answer.

(28)

26 Selecting voir tout l’historique leads to the whole history of selected sentences and their associated answers. From there, it is possible to click on télécharger PDF to download a PDF file with the same selected sentences and answers (see Image 16). This document can be printed out for various purposes such as an archive for the patient’s medical folder. The doctor has a precise overview of the consultation and does not have to write everything down, as opposed to a classical consultation.

Image 16 – 4 screenshots of plus d’options… [more options…] type of questions, history and PDF report

BabelDr is not the first medical translation application using the speech recognition tool. This feature has been also used in the previously described MedTranslate application. However, MedTranslate limits itself to the tool and if this tool does not work, the application is simply unusable until the problem is solved. This can occur for example if there are connection problems, or if there is some update or reboot of the system, or even if the doctor is not a native speaker of the used language.

In general, speech recognition tools “learn” how the words and sentences are pronounced by native speakers, because these are the people who best reflect the language pronunciation. So when a speaker has non-native pronunciation or makes linguistic mistakes, the tool will have difficulties to find a corresponding word or sentence. It will either recognize the vocal input as something different from what was said, or it will simply fail to recognize anything.

(29)

27 That is why such tools should theoretically be used only by native speakers to best match the words and sentences they were designed to recognize. The advantage of BabelDr is that the doctor can select a domain and search himself for the sentences if speech recognition fails to work, and thus carry on the consultation.

It is however not straightforward to find a desired sentence, since there are no subdomains and many questions do not directly correspond to the domain. For instance, all administrative sentences are present in all domains, such as avez-vous un numéro de téléphone ? [do you have a phone number?], which has in fact nothing to do with domains like ORL, Habits, etc..

Nonetheless, it seems that BabelDr was designed primarily for a desktop computer rather than for mobile devices. In particular, on those smaller devices, the sentences appear too close together, making it likely to click on a wrong sentence. In addition, the area showing the sentences is small, so one can only see a few sentences at the same time. These are the major challenges encountered by using BabelDr.

1.3 Advantages and disadvantages of all discussed applications

As seen so far, the four chosen mobile applications described in addition to the BabelDr web application are all quite different from one another. Three of these applications offer a large number of languages and most of them subdivide the sentences into their respective domains, sometimes also in subdomain branches.

The mobile applications are mostly (or sometimes just only) available on the Apple iPhones rather than on Android phones. It would obviously be useful to target both in order to accommodate more users.

Almost all the applications provide pre-translated sentences, making the translation more accurate compared to live machine translation systems.

The Graphical User Interface (GUI) of most applications is often not so user-friendly, as there are no icons or pictures to help. This is also the case with BabelDr, which lacks subdomains and has no images representing the domains.

(30)

28 Some applications use acronyms in order to make some medical expressions shorter, which can be useful for a doctor who knows them well. However, if there is only one source language and the doctor did not study medicine in that language, navigating among acronyms would certainly be difficult.

Not all applications allow the creation of a history report of the already used sentences. The absence of this feature could lead the doctor to lose track of the consultation and forget what he has already asked (or not asked yet) the patient. This increases the risk of making mistakes if no separate notes are taken. As designed in BabelDr, it is certainly a big advantage to have a history generated automatically for the benefit of the doctor. It could be useful to add a button to clear the history in a faster way.

Concerning the audio output, some application’s audio quality can be disputable. In contrast, BabelDr only uses validated audio, produced either by text to speech for languages where it is available (Arabic, Dari, Farsi, Spanish) or by human recordings (Albanian, Tigrinya).

Currently a custom synthetic voice is being developed for the Albanian language. A study shows that 60 translations (prompts) asserted by 12 Albanian native speakers increased in quality thanks to tools like Tacotron 2 (Tsourakis, Troque, Gerlach, Bouillon, & Spechbach, 2020).

Nevertheless, the most important functionality of any of those applications, and also the purpose these applications were designed for in the first place (apart from the linguistic aspects) is to enable the doctor to find a sentence quickly. When testing all these applications, one of the first observations was that it is a challenge to find a sentence. Most applications have domains, but either no or too many subdomains. Even a doctor, in contrast to a random user, might have difficulties to understand quickly these classification branches.

In emergency services especially, doctors do not have time to waste searching for a question to ask. Hence, the doctor needs a tool that is simple enough to be used, yet sufficiently extensive to be useful. This leads us to assert that the classification of domains, subdomains and sentences should follow logic guidelines similar to practical medical standard procedures the doctor is used to. In particular, since desired sentences have to be found quickly, there should not be too many sentences within a subdomain.

(31)

29 Table 1 compares the main aspects of all applications discussed before. However, to simplify this document, features of the new MobileDr were already added, Chapter 2 will later present the MobileDr web application.

Care to Translate

Canopy Speak

MediBabble Translator

Med-

Translate BabelDr MobileDr

Available on Android ✓ ✓ ✗ ✗ ✓ ✓

Available on iOS ✓ ✓ ✓ ✓ ✓ ✓

Unrequired profile creation ✓ ✗ ^✓ ^✓ ^✓ ^✓

Free full version ✗ ^✓ ^✓ ^✓ ^✓ ^✓

Change source language ✓ ✗ ✓ ✓ ✗(✓) ✗

Doctor/main user interface ✓ ✓ ✓ ✓ ✓ ✓

Independent patient interface ✓ ✗ ✗ ✗ ✗ ✗

Speech recognition tool ✗ ✗ ✗ ✓ ✓ ✗

Domains ✓ ✓ ✓ ✗ ✓ ✓

Subdomains ✗ ✓ ✓ ✗ ✗ ✓

Icons/pictures of domains ✓ ✗ ✗ ✗ ✗ ✓

Subtitles for sentences ✗ ✗ ^✓ ✗ ✗ ✗

Pre-translated sentences ✓ ✓ ✓ ✗ ^✓ ^✓

Acronyms of medical words ✗ ^✓ ^✓ ✗ ✗ ^✓

Separate translation page ✗ ✗ ^✓ ✗ ^✓ ^✓

Human-like audio records ✓ ✗ ✓ ✓ ✓ ✓

Selectable patient’s answers ✗ ✓ ✗ ✗ ✓ ✓

History of used sentences ✗ ✗ ✓ ✓ ✓ ✓

Clear history easily ✗ ✗ ✗ ✓ ✓ ✓

Words search tool ✓ ✓ ✓ ✗ ✓ ✗(✓)

Table 1 – 20 aspects compared in the 6 applications. Signs legend: ✓stands for “has this aspect”,✗ for “lacks of this aspect” and✗(✓)suggests the aspect was “planned to be added later”.

It is worth mentioning that indications about a future addition of English as a source language are present in the BabelDr source code. This means the “change source language” aspect is currently under development. In MobileDr, the wish to implement a “word search tool” has been expressed, but the deadline to finish the Master thesis did not allow for its development to start. That is why it is referred to as “planned to be added later”.

(32)

30 To summarize, the following characteristics are compared in Table 2 for all 6 chosen applications: number of domains, subdomains, languages each application offers, and most notably the languages present uniquely in each application (see Table 2).

Care to Translate

Canopy Speak

MediBabble Translator

Med-

Translate BabelDr MobileDr

Nb of domains 30 10 9 0 13 16

Nb of subdomains 0 71 155 0 0 98

Nb of languages 34 15 2 35 7 7

Unique languages 5 3 0 5 1 1

Table 2 – Other compared aspects in all six applications, as additional information in numbers.

Three of the applications offer a large number of languages. Here are listed all languages proposed in each application. Languages which are present only in one application (unique languages) and not in the others are in blue:

Care to Translate: Albanian, Arabic, Bengali (Bangladesh), Bosnian/Croatian/Serbian, Bulgarian, Chinese (Mandarin), Danish, Dari, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Kurmanji, Lule Sami, Norwegian, Pashto, Persian/Farsi, Polish, Portuguese (Brazil), Portuguese (Portugal), Romanian, Russian, Somali (Somalia), Sorani, Spanish (Latin America), Spanish (Spain), Swedish, Thai, Tigrinya, Turkish

Canopy Speak: Arabic, Bengali, Chinese (Cantonese), Chinese (Mandarin), Filipino, French, Haitian Creole, Hindi, Japanese, Korean, Malay, Portuguese, Russian, Spanish, Vietnamese

MediBabble Translator: English, Spanish (others are somehow not downloadable)

MedTranslate: Arabic, Chinese (simplified), Chinese (traditional), Croatian, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (Brazil, Portugal), Romanian, Russian, Slovak, Spanish, Swedish, Thai, Turkish, Ukrainian, Vietnamese

BabelDr + MobileDr: Albanian, Arabic, Dari, Farsi, Spanish, Swiss sign language, Tigrinya

(33)

31 Care to Translate and MedTranslate are offering the most languages and are also the ones having the most of unique languages. BabelDr is unique in its capability to allow the communication with deaf people, which are about 10’000 in Switzerland (SGB-FSS, 2016). A paper describing the translation methodology (into videos with a person doing the signs) and translation issues with medical speech with the Swiss sign language has been published (Strasly et al., 2018).

1.4 Conclusion

In this chapter, we have given an overview of four existing medical translation applications and of the original BabelDr application. Testing these different applications has allowed us to identify several useful features, as well as a number of usability issues. We will see in the next chapter how we have applied these observations to the development of the new MobileDr application, which uses the existing BabelDr resources (pre-translated sentences, audio) but aims at creating an improved interface for mobile devices.

(34)

32

2 MOBILEDR

This chapter describes in detail the MobileDr web application, whose layout was specially intended for smartphones. Section 2.1 outlines the main development guidelines. Section 2.2 describes the selecting and restructuring of the BabelDr content.

Section 2.3 presents the hierarchical structure of the MobileDr content, its domains placed under categories and the subdomains of each domain. Section 2.4 shows the different answers available for the patient depending on the sentence type. Section 2.5 shows the history and PDF layout. Finally, Section 2.6 illustrates a clinical case.

2.1 Main development guidelines

There was no pretention about developing a better web application than BabelDr, but rather to create a different version which would try to satisfy as much as possible requirements of Table 1 presented in Section 1.3. Indeed, Table 1 summarizes what we believe are the most important aspects a medical translation application ought to have in order to achieve the best possible doctor-patient communication in the absence of common language. This, of course, also automatically includes pre-translated sentences by a professional translator instead of live machine translation, since it is currently the only way to have no doubts about the target language outcome.

It seems that BabelDr was developed mainly for use on a desktop with a big screen, and with a main focus on speech interaction. It is very powerful for such a configuration, but the layout and the classification of sentences need to be adapted for smartphones and other small screen devices. Not being able to be at his desktop all the time, the HUG doctors collaborating in on the project expressed their wish to have an application designed for a device they could carry around with them at any time. A smartphone appeared to be a natural device to use for such purpose. On such a device, and with all the practical requirements described above, the following guidelines can be formulated. MobileDr has to be intuitive, effective and visually attractive. This consequently necessitates restructuring the data (sentences) in a new layout

(35)

33 more adapted to the practical usage. The following chapters describe in detail how these guidelines have been implemented in MobileDr.

2.2 Selecting and restructuring the BabelDr content

BabelDr contained more than 11’000 different sentences in its system February 2020. These sentences can be found through a combination of three methods: list search in a chosen domain, key word search or speech recognition. It is difficult to navigate inside such a big pool of sentences on a smartphone, where lists can become cumbersome and long to search.

The doctor might not find a needed sentence quickly enough and may lose time during a consultation. Especially in emergency services, it is even more important to find a necessary sentence quickly.

Thus, a decision was made to restructure the sentences according to domains, subdomains, and drop down menus which match usual consultation practices. In doing so, numbers of sentences, considered redundant for this new logic, have been removed and many variants of the same concepts have been replaced by drop down menus, simplifying noticeably the navigation inside the application. These changes will be discussed in detail below.

The sentences have been selected with respect to our guidelines, the simplicity of usage and the effectiveness, without forgetting that they have to fill every stage of a classical consultation. In a nutshell, the application of the above guidelines imposed finding a minimum necessary set of questions that satisfy all medical requirements. From there, it has been necessary to restructure the layout and search process in an intuitive and familiar way (from the doctors’ point of view) to reduce time and effort invested in navigating, translating and searching, and allow doctors to focus on their medical practice.

We will now take a closer look at the selection of sentences.

Naturally, the sentence classification was done according to the given available sentences. It could have been constructed differently. In all cases, it is hard to say if a perfect solution does exist. Here are shown examples of some removed sentences as an illustration of sentence number reduction.

(36)

34 Sentences introducing a subjective or vague dimension:

 avez-vous plutôt mal au milieu du dos ?

[do you rather have pain in the middle of the back?]

 vos douleurs durent-elles depuis seulement une heure ? [are you in pain since only an hour?]

 avez-vous arrêté un traitement récemment ? [have you stopped a treatment recently?]

Sentences with unlikely or odd information¹:

 vous devez faire une injection une fois par jour le matin et le soir

[you need to do an injection once a day in the morning and in the evening]

 avez-vous perdu l’appétit depuis plusieurs secondes ? [have you lost your appetite since several seconds?]

 votre visage est-il tordu ? [is your face crooked?]

 prenez-vous un traitement pour le diabète oral depuis plusieurs secondes ? [have you been taking oral diabetes treatment for several seconds?]

Sentences with unspecific or incomplete information:

 vous devez prendre trois suppositoires entre dix et quatorze jours [you should take three suppositories between ten and fourteen days]

 avez-vous mal au ventre dans le flanc ? [do you have a stomach ache in your flank? ]

 avez-vous mal au dos depuis longtemps ? [have you had back pain for a long time?]

2.3 Presentation of the web application

This section presents MobileDr and its different features. Its categories, domains and subdomains will be presented in detail. Our tests of MobileDr were made on recent Samsung phones and iPhones under recent versions of Chrome, Safari and Samsung Internet browsers.

It is not guaranteed that the application will work on other smartphones or browsers. While reading this section, it is recommended to test MobileDr on a smartphone. At the time of writing, it is available under https://regulus.unige.ch/textbabeldr/phone.

1 These sentences are the result of the compositional construction of the BabelDr resources, which are produced from a grammar with sentences including variables. Not all combinations of sentences and variables are likely.

(37)

35 2.3.1 Login and gender/language page

When going on the application’s website, the login page appears. It is very similar to BabelDr (see Image 17). The user either has a special doctor’s login username and password, or he can use respectively “babeldr2” and “babeldr”. After being logged in, we arrive to the choice of the patient’s gender and language (see Image 17). The patient is supposed to look at the screen and recognize his language written is his alphabet. There is currently only French as source language and the whole application’s pages and menus are in French as well. This is also the case for BabelDr, even though English as the source language is said to be under development.

If we forget to select the gender on the gender and language selection page, a warning message appears in red as a reminder: sélectionnez le sexe, svp [please select gender] (see Image 17). We cannot continue until both patient’s gender and language have been selected.

This is important for languages where men and women are addressed differently. Transgender patients could be addressed by the gender the feel they belong to.

Image 17 – 4 screenshots of login, change password page, gender and language selection, warning message

(38)

36 2.3.2 Domains page

After the selection of the patient’s gender and language, the domains page is displayed (see Image 18). There are 16 domains grouped in 3 categories, and a special isolated domain called COVID-19 is located on the top right corner. It is visually separated from the rest as it is a current pandemic, so that the doctor can find it immediately when facing such specific emergency. It could be replaced by some other urgent illness if such appears. On the top left corner there is a Précédent [Back] button which leads back to the gender/language selection page.

The 3 categories are Accueil [Welcome], Spécialités [Specialties] and Finalisation [Completion] (see Image 18). They are aimed to facilitate the doctor’s orientation among the domains and resemble a typical consultation in progress (Rey-Bellet et al., 2008; Bagheri, Ibrahim &

Habil, 2015). Accueil and Finalisation both contain 4 domains, and Specialties contains 8 domains. Every single domain contains its respective subdomains. Both

Image 18 – Domains page domains and subdomains have their own picture which

visually helps to remember and locate them.

2.3.3 Separate COVID-19 domain

COVID-19 (coronavirus disease 2019) has been declared a pandemic by the World Health Organization (WHO) in March 2020 (WHO, 2020). It is an infectious disease mostly transmitted by contact, droplets, air and contaminated surfaces. Symptoms are mostly fever, dry cough and tiredness. Serious symptoms are difficulty breathing, chest pain and loss of speech or movement. Loss of taste or smell, headache, sore throat, diarrhea, aches and pains,

(39)

37 conjunctivitis and skin rash can also manifest (OFSP, n.d). COVID-19 is such a complicated and emerging pandemic and even though vaccines start being available, not everything is known about it yet. It is therefore important to keep it in a separate domain.

COVID-19 domain contains 4 subdomains (see Image 19).

Bonjour/Administratif [Hello/Administration] contains only the most important sentences about doctor’s self- presentation, explanation about how the system works, asking about the patient’s details, etc. In Symptômes [Symptoms] are questions about most symptoms explained above. Patient à risque [Patient at risk] contains questions about vulnerable people, for ex. seniors, people with chronical disease, cancer, diabetes, people travelling, etc.

Frottis/Au revoir [Smear/Goodbye] has sentences explaining how the smear will be conducted, which measures the patient should take, contact with other people, and consultation ending.

Image 19 – COVID-19 subdomains

2.3.4 Accueil category: domains and subdomains

The Accueil category contains the 4 following domains (see Image 20):

 Administratif/Ananmnèse sociale [Administration/Social anamnesis]

 Anamnèse médicale [Medical anamnesis]

 Habitudes [Habits]

 Examen physique [Physical examination]

As mentioned before, the medical consultation has a fairly standard procedure (Rey-Bellet et al., 2008; Bagheri, Ibrahim & Habil, 2015), except in a vital emergency. The doctor needs to

(40)

38 collect minimum information about the patient, at first glance not related to the reason for consultation. This includes: personal information (spoken language, age, origin), social conditions (place of living, insurance or means of payment, health conditions) and professional situation (type of job, employment status). Contact details of a significant other or a close friend are often requested in case of a problem with the patient such as a serious complication or even death.

Certain indicators might lead the doctor to think directly about a specialized field, but it should not be forgotten that symptoms can have various causes. For example, a food-related problem can influence or cause a skin disease and it would therefore be wrong to ignore this cause. Alternatively, a patient can have metastasis in the brain while having lung cancer. For this reason, the doctor is in the habit of keeping a system anamnesis to check if the patient’s biological systems may be dysfunctional and thus may influence the course of the disease.

When welcoming a patient with information, a certain respect and kindness is required from the medical staff to build mutual trust. It is common sense to say hello, explain what to do, ask for permission to touch the body, clearly inform about the illness, etc.

Image 20 – Accueil category: 4 screenshots of each domain and its subdomains

The following section will describe the subdomains of these four domains.

MobileDr : adapting the BabelDr medical translator for improved usability on mobile devices

Master

Reference