Care to Translate

We will now introduce the four medical translation applications (Care to Translate, Canopy Speak, MediBabble Translator, and MedTranslate) to give an overview of what already exists.

Images in this chapter are screenshots of all these applications displayed on a smartphone.

1.1.1 Care to Translate

The application Care to Translate originated as a non-profit student project in Sweden and was developed as a downloadable application with a payed extension. The user does not need to create a profile to use the application as such. Another version of this application, called CTT Clinic, was probably developed only for professional healthcare providers. Indeed, the user must login with an already existing account which unfortunately cannot be directly created, so it was not possible to analyze it in this study. In what follows, the focus is only on the Care to Translate application.

When opening the application, the doctor can choose among 34 languages on both the doctor and the patient side (see Image 1).

Image 1 – 3 screenshots of possible settings, choice of type of user (doctor or patient) and choice of languages

12 On the doctor side, the sentences are classified in domains. Each domain consists of a domain name and a pictogram (see Image 2), which can help the doctor both linguistically and visually to situate himself within the structure. Since there are 30 domains, they cannot be all displayed on one single phone screen. This implies a scrolling up and down to see all available domains. When clicking on a domain, its sentences appear (see Image 2).

Image 2 – Doctor side: 4 screenshots of all 30 domains, sentences in a given domain and the warning sign

No order in the sentences has been found when going through the sentences of different domains. In domains like Presentation, Family, Front desk, etc., a disordered structure might not necessarily negatively affect a consultation. However, a chaotic order of sentences in domains of medical specialties (Psychiatry, Gynecology, etc.) could impact the consultation process with a doctor who might lose their train of thought (see Image 2).

When analyzing the sentences of a more evident specialty like Dental care, the lack of sentence order is clearly noticeable: the sentences range from a disease announcement (you have a gum disease) to naming body elements (the inside of your cheek), passing by a therapeutic intervention (I need to extract a tooth) (see Image 2). Evidently, all these sentences concern dental specialty. However, it is not so obvious to know in advance which

13 sentences can be found in this domain. The only clear aspect is that all of these sentences concern teeth.

From a medical perspective, these aforementioned three sentences could also be found in other domains than just the Dental Care domain: you have a gum disease in Diagnosis and information domain, the inside of your cheek in The body domain and I need to extract a tooth in Surgery domain. It is not clear whether every doctor would think the same way about the classification of sentences. In particular, during an emergency, the doctor will not have the time to navigate within the domains when searching for each question.

From a sentence formulation perspective, some available sentences are only single words or clauses. It is unclear when exactly a doctor would need to use the gums or a clause such as the inside of the cheek without turning it into a question or an affirmation. Indeed, it looks more like an answer to a question a patient might have asked. This shows that full sentences are not separated from clauses and words which seem to bear a resemblance to an answer.

It is worth mentioning that some sentences have a “warning sign” next to them, for example the sentence are you allergic to antibiotics? in the Medication domain (see Image 2). When clicking on such sentences, a special menu appears with the message “make sure that both question and answer are understood correctly”.

This might be for two reasons. Either the sentence contains a medical word which might be difficult to understand for a layperson, or the sentence could be wrongly understood depending on the context. Typically, confusion in the meaning can be created if a word exists in different homonym forms as homographs (words with same spelling but different meaning) or homophones (words with same pronunciation but different meaning) in a given language.

From this hypothesis, it can be concluded that a doctor will be warned (by the “warning sign”

and message) that the sentence he is about to use could possibly be problematic in its meaning. However, no concrete solution is proposed to him; even when asking a patient do you understand? and receiving a “yes” response, the doctor has no assurance that the patient understood the sentence, especially a problematic sentence.

From the patient side, a different set of 21 domains with corresponding pictograms is displayed, enabling him to express more than nodding and gesturing (see Image 3). Although

14 the possibilities are limited compared to the doctor side, the patient is able to ask for Orientation, express his Feelings, Ask for help, etc. With the number of domains, the patient also has to scroll down and up to see the whole domain set. It is unclear whether an elderly, suffering or confused patient would be able to confront this amount of information, given that he could only be familiarized with the application at the moment he enters the hospital.

From a sentence formulation perspective, there are single words (see Image 3) which could this time enhance confusion on the doctor side. For example, if the patient wants to express the fact that his grandfather had died from cancer and he does not find that sentence, he might just select grandfather and the doctor would probably not understand the intended meaning.

Image 3 – Patient side: 4 screenshots of 21 domains and examples of sentences in two domains.

In conclusion, the application Care to Translate has a doctor side and an independent patient side, which are independent, and provides 34 languages on both. Sentences are classified in domains consisting of a domain name and a corresponding pictogram. Domains can be of different types: specialties, diagnosis, posology, intervention and many others. In addition of full sentences, one can also find individual words and clauses. Potential usability issues related to the classification of sentences and numerous domains have been detected.

15 1.1.2 Canopy Speak

The digital health company called Canopy Innovations was founded in 2010 by Bill Tan and is head-quartered in New York, USA. One of its services is the Canopy Speak smartphone application, already used in some hospitals and healthcare systems. Its only source language so far is English. This implies it can be used efficiently only by healthcare providers who studied a medical field in English.

In order to use the application, one must create a profile and provide details such as personal information, work institution and the held occupation (see Image 4). It is also possible to contact directly an interpreter. This functionality has not been tested. The application consists of only the full doctor side with 15 target languages (see Image 4).

Image 4 – 3 screenshots of profile creation, language selection and domains

Canopy Speak has 10 domains (see Image 4) and up to 3 subdomain branches in a domain. To illustrate this branch structure, the way to reach the GI (gastrointestinal system) subdomain, one has to go through the path: Internal Medicine domain > History > HPI (history of the present illness) > GI (gastrointestinal system) (see Image 5). The only visibly traceable part of the path is the domain (here Internal Medicine) and first subdomain History (visible on the

16 blue toolbar in Image 5). For the subsequent subdomains, it is impossible to know the rest of the path retrospectively.

Image 5 – 3 screenshots of domain Internal Medicine, then subdomain branches History > HPI > GI

Selecting Abdominal Pain in the GI subdomain branch leads to a set of 56 sentences. At least three problematic issues were found at the very bottom of the list (see Image 6). When these questions are selected, their respective audios are played. The audios have clearly not been recorded by human voices, but rather by a text to speech engine with a slow unnatural pace, and it can be assumed that long sentences are constructed from short record segments.

Examples of problematic pronunciation for French as a target language follow:

Image 6 – 3 screenshots of Internal Medicine > History > HPI > GI > Abdominal Pain: wrong pronunciation

17 In the first example, the voice makes mistakes if there is a typography error in the translation.

For instance, in the sentence do you have a history of any sexually transmitted infections?

translated as avez-vous déjà eu des maladies sexuellement trnsmissibles ? (see Image 6), the underlined misspelled adjective “trnsmissibles “ comes out letter by letter as “t-r-n-missibles”

instead of “transmissibles” making it confusing.

Another incorrect pronunciation can be found in the sentence have you had an increased appetite?. It is translated as votre appétit a-t-il augmenté ? (see Image 6). The spelling of the underlined set of words is correct, but the voice pronounces distinctly almost every part of it:

“a tiret t il” (“tiret” means “hyphen”) comes out, yet it should be spoken out all at once as

“atil”.

As a third example, consider the sentence have you recently had abdominal surgery or any scope procedures?, translated as avez-vous récemment subit une chirurgie abdominale ou tout autre type d’intervention sur cette partie du corps ? (see Image 6). The noun “intervention” is spelled correctly, but the audio pauses while pronouncing (“interven-tion”). This shows that a word could be cut in the middle, even if spelled correctly, as if it was not known to the system. Furthermore, “tion” is pronounced with “t” whereas in this case, “tion” should be pronounced with “s” (“sion”). On the contrary, a misspelled word can be pronounced correctly. Indeed, in the same sentence, the conjugated verb “subit” is incorrectly spelled (it should be “subi”), but is still pronounced with accuracy. Based on our observations, the quality of the translations and audios has possibly not been validated.

In the same branching example Internal Medicine > History > HPI > GI > Abdominal Pain, it appears that the doctor has to click at least 5 times to select the desired sentence. Even when going through so many subdomains, he arrives to a final set of 56 sentences.

On top of that, it is worth mentioning there are also medical abbreviations in the application.

Hence, if the doctor doesn’t have a solid English medical background, it might be challenging to navigate through the subdomains without having to look for the acronyms. Moreover, as mentioned above, it is visually difficult to know in which subdomain we are situated at a given moment except for the domain and first subdomain (dark blue toolbar in Image 5).

18 There is a limited patient interface opened by selecting the three dots menu, which appears on the right-hand side of the selected sentence (see Image 7). In this menu, selecting Patient Response gives the patient a handful of sentences to choose from. The patient can say what he feels, needs or that he wishes to ask a question. Each of those options has its own set of subdomains, and the doctor can ultimately react to the answers at the end of the path. The options for the doctor to react are, however, very limited. Finally, it seems very easy for the doctor to overlook these options as the layout is not really user-friendly.

Image 7 – 4 screenshots of choosing to add patient’s response, response possibilities and doctor’s follow-up

In conclusion, Canopy Speak proposes 15 target languages and a subdivision in 10 domains.

The subdomain branches are numerous. Medical words are presented as acronyms. The recorded voice is clearly synthetic, with pronunciation mistakes. There is a very limited interface for the patient to express himself with restricted choices. The doctor has only a few options for the response.

19 1.1.3 MediBabble Translator

MediBabble Translator is only available on iOS and was created by NiteFloat Inc. It offers a limited choice of only English and Spanish as both target and source languages (see Image 8).

Even though there should be a possibility to download other languages, they failed to load when testing the application. The main menu presents 9 domains and each domain has a different number of subdomains (see Image 8).

There is an implicit separation into four categories. Three of them are explicit: Introductions

& Explanations, Physical Exam and Follow-up Questions. The remaining domains fall under the final category representing the medical history (see Image 8). On the right side of the screen the acronyms of their respective domains are presented when available (see Image 8).

Image 8 – 3 screenshots of the choice of languages, the domains, and an example of subdomains

When selecting one of these medical history domains, the History menu appears where the domains in the form of acronyms are visible on the top navigation bar (see Image 9). When selecting the different subdomains of History, some subdomains lead directly to a selection of sentences, while others require navigation through another branch of subdomains to reach the sentences. Some sentences have subtitles (for example Tobacco use) and some do not (see Image 9). The navigation is not very intuitive.

20

Image 9 – 3 screenshots of sentences examples with and without subtitles, and option to rotate the screen

When selecting a sentence, there is a possibility to rotate the phone of 90° and to obtain an enlarged version of the translation, probably to show it to the patient in case the audio record was insufficient to comprehend (see Image 10). In particular, it could be useful for an elderly or visually impaired person. In such cases however, it could be inconvenient for the doctor since he would have to rotate the phone for each sentence during the consultation.

On top of that, there is a history of recently played sentences (see Image 10), that probably is meant to remind which sentences were already used. It cannot be cleared inside the application. One has to reinstall the application to clear the history. This seems rather impractical.

To improve usability, several changes might be considered. For example, to create a separate page for each translation, write it in big characters, keep the phone straight and allow history clearing in the application.

21 Image 10 – 3 screenshots of selected question, screen rotation to get the translation and history

In conclusion, MediBabble Translator offers so far only English and Spanish as source and target languages. It consists of 9 domains, each of them subdivided in a different number of subdomain branches. Some sentences have describing subtitles. It is possible to rotate the screen and see the translated sentence in a bigger size. There is also a history of the last translated sentences, but there is no way to clear it in a simple way. MediBabble Translator works only on iOS.

1.1.4 MedTranslate

The last tested application is particularly interesting because, to our knowledge, it is the only medical translation application (except BabelDr) using speech recognition and direct translation. It was developed by BYU Creative Works and is only available on iOS. Its instructions (like Hold to Record Voice) are only in English, but the application offers 35 source and target languages (see Image 11). There are neither domains, nor choice of sentences, but rather only the speech recognition and automatic translation tools.

22

Image 11 – 3 screenshots of main menu with speech recognition tool, languages and recognized sentence

A random medical question in French was tried and perfectly recognized (see Image 11):

avez-vous mal quelque part? [do you feel pain somewhere?]. The speech recognition tool worked remarkably well, but the translation tool converted the previous sentence into do you hurt somewhere?.

One of the buttons appearing below on the screen enables to correct the translation by typing the right one on the virtual keyboard and then it is possible to record the new translation (see Image 12). We corrected the sentence to are you hurt somewhere?. However, the tool recognized “heard” instead of “hurt” and rated the record as incorrect, displaying a red

“unlike” icon, probably because it was not pronounced by a native English speaker (see Image 12). This was a try from French to English, which is a quite common combination.

To test the speech recognition and the translation tools with a less common source language, we used the Czech language. The sentence in Czech was pronounced by a native speaker and was perfectly recognized (see Image 12): bolí vás něco? [literally: is something hurting you?].

Its translation was: does it hurt somewhere?, which is grammatically and syntactically correct.

There is also the possibility to clear history inside the application (see Image 12).

23

Image 12 – 4 screenshots of French-English recognized sentence, its new recording, Czech example and history

To see if the recording would work better from French to Czech, we tried to say in French quand avez-vous bu pour la dernière fois ?, which was translated to Czech as kdy jsi naposledy pil? [when did you drink last time?]. This is correct, but in Czech it is an informal way to address a man, whereas we wanted a formal way to address a woman. The sentence was corrected to kdy jste naposledy pila? and was then orally re-recorded. It was also perfectly recognized, probably because it was pronounced by a native speaker (see Image 13).

Image 13 – 3 screenshots of French-Czech example, correction and recording rated as correct

24 In conclusion, MedTranslate is different from the three above applications in the fact that it does not use pre-translated sentences, but is based on a combination of a speech recognition tool coupled to a translation tool. It offers 35 source and target languages. There is the possibility to correct by hand and re-record a sentence if it was not well translated. It can recognize the new right or wrong pronunciation during a new trial. There is history option to see the last used sentences and a simple way to clear it. The usage of correcting and re-recording a sentence for the target language is rather questionable, because one needs to be almost native in that language to correct a mistake, and re-record the right sentence with a recognizable accent.

1.2 Presentation of BabelDr

The BabelDr web application (Bouillon et al., 2016) is being developed by Pr. Pierrette Bouillon and her team at the FTI in UNIGE, in collaboration with Dr. Hervé Spechbach and his team from the Geneva HUG. The project was mostly financed by the Fondation privée des HUG [Private Foundation of the HUG] and received the price called “Prix Innogap” from Unitec. The application is already being used at the HUG where each doctor has his own account linked to medical files of his patients. Other users are currently not obliged to create an account and can use the username “babeldr2” and password “babeldr” (see Image 14). The doctor selects the patient’s language and gender, and chooses the domain which to begin with.

Image 14 – 4 screenshots of login, main page, choice of domains and choice of patient’s language

25 When the domain has been selected, the doctor can choose either to use the speech recognition tool, or to find the desired sentence by himself in the list underneath (see Image 15). When the speech recognition tool is clicked upon (see the well visible microphone icon in Image 15), it records the speech input of the doctor. If a sentence is recognized, it appears within the first white rectangle. The doctor then decides if the recognized sentence resembles enough the sentence he spoke out, and if not he can select among other proposed sentences.

After validating the sentence with a click, the software leads to a separate page where the translation appears in big characters together with all answer options (see Image 15). There are three possible answers, namely “✓”, “✗” and an image with “?”, or a rectangle where

After validating the sentence with a click, the software leads to a separate page where the translation appears in big characters together with all answer options (see Image 15). There are three possible answers, namely “✓”, “✗” and an image with “?”, or a rectangle where