It makes no sense to calculate any values for Table 8, as there is no other table to be compared with (since the questions are different from Table 5 and Table 6). This table only exists to indicate on which good aspects (green scores) the doctors agree and which aspects most need improvements (red and blue scores) in MobileDr application.
The ten sentences of the customized SUS are the following:
1. Arrangement of domains in Welcome, Specialties, Completion is suitable for consultation 2. Images of domains and subdomains do not stimulate the doctor’s visual memory
3. The icon(s) at the top right representing the domain and subdomain help to locate myself 4. Subdomains are unrelated to their respective domains
5. The sentences correspond to their respective subdomain
6. It is easy to guess which choices are contained in a sentence drop-down menu
7. The presented order of sentences does not follow the order of importance of the questions 8. It is useful to see in the list of sentences the selected answer of the asked question
9. The icons and caption in the PDF give a poor overview of the consultation
10. The subdomains and their sentences did not influence the flow of the consultation
88 Table 8 presents the scores given according to each precise aspect of MobileDr:
D1 D2 D3 D4 D5 D6 D7 D8 tot /32
Table 8 – MobileDr customized SUS scores obtained by all eight doctors (D1-D8) in all ten questions (Q1-Q10)
Here follows a summary of the best aspects as well as the aspects to improve. Please see appendix B (and appendix D for its French version) to have a reminder about the customized SUS questionnaire.
Best aspects:
Question 2: the images stimulate visual memory (30/32)
Question 4: the subdomains are related to their domains (30/32)
Question 8: it helps to see the selected answer in the questions list (30/32) Question 9: the PDF is good for a consultations overview (29/32)
Question 5: the sentences correlate with the subdomains (28/32) Question 6: the drop-down menus content is clear (28/32)
Aspects to improve:
Question 1: the categories reflect a classical consultation (26/32) Question 3: the top right icons help to get situated (25/32)
Question 10: the layout did not influence the consultation flow (25/32) Question 7: the sentences are in correct order (20/32)
89 The scores give an indication of the overall navigation experience of each doctor.
Unfortunately, no doctor had the time to look at every domain, and subdomain and analyze every presented sentence, but a general feeling is still useful to identify which aspects gave the best impression and which aspects could be improved on.
Q2, Q4 and Q8 received the best scores (30/32), indicating that the doctors liked the following newly introduced features: images of domains and subdomains, the creation of subdomains which seem well related to their domain and the possibility to see the answer in the list of sentences.
Other good aspects are also reflected by Q9, Q5 and Q6, indicating that the PDF layout was improved, that the sentences correlate well with the subdomains and that the choices within the drop-down menus are easily guessable. One doctor in each of these last questions seemed unsure about the answer. However, these are isolated cases.
D4 and D5 (Interns) did not find some important sentences in some subdomains (Q7, grades 1) D2 and D6 (specialists) are not particularly sure of some subdomains order (grades 2). It is possible that depending on the specialization, doctors would prefer to choose the sentences in different orders or that the sentences they wanted were not in the available pool of sentences of BabelDr.
An interesting discussion is raised with Q10 about whether MobileDr influences the consultation flow. It seems that the scores cannot really be rated as positive or negative, since the influence of the application on the doctor’s decision can have both positive and less positive effects. In general, doctors agreed that the structure enables them to navigate and choose the sentences how they want, and that it is a good point to see other relevant sentences they might not have necessarily immediately thought of.
The aim of this evaluation was not to choose the doctors for a statistical value. These doctors were chosen to have an expert opinion about MobileDr and BabelDr on a mobile device. It has been shown that MobileDr suits them better; they gave considerably higher scores to all MobileDr aspects and these score were more homogeneous. Aspects to be improved were also identified. The individual opinion of the doctors will now be shown.
90 4.7 Doctors’ comments and suggestions about BabelDr and MobileDr
After they filled the questionnaires, a discussion took place with every one of them according to the time he or she could spend with us. Overall, all doctors were enthusiastic about the MobileDr concept and liked the new integrated features and layout of the application. Here are the most important improvements they suggested. As mentioned previously, for the sake of readability and participant anonymization, in the following discussion the masculine form is used to refer to participants independently of their actual gender.
SMPR doctors are already using BabelDr during their consultations. This means that they did not discover both applications (original and new) at the same time and have already frequently used the original version. Their favorite part is the speech recognition component, mainly because it allows them to normalize the interaction with the patient, since both doctor and patient express themselves through the device playing the role of an intermediate. They would like this component to be integrated in the new version. They appreciated the sentence classification in domains and subdomains, and the idea that the main menu arrangement follows a classical consultation structure, enabling to find a question in case the speech recognition component does not work or cannot find the desired sentence. For the same reason, they said the reduction of the number of sentences in the new version was also a good point.
In the first half of the main menu, Doctor 1 would prefer to have following order:
Administratif/Anamnèse sociale, Motif de consultation (which is a subdomain), all Spécialités category, and only then Habitudes and Examen physique, and leave Finalisation category as it is. This is because after the medical secretary takes the identity, contact details and insurance, the first thing a doctor will want to know is why the patient came to the hospital, then to talk about his specific problem, and only then to ask about medical history and family background, and finally to do a physical examination before entering the finalization section.
He also suggested putting an Examen neurologique subdomain in Examen physique domain instead of dividing the sentences into Tête, MS and MI.
Doctor 2 had a similar opinion as Doctor 1 concerning the main menu. In addition, he looked more precisely into the COVID-19 domain and recommended to divide the subdomains according to the usual procedure of suspected Covid-19 patient cases. He provided the paper
91 used in SMPR called “Formulaire pour la prise en soin d’un patient stable suspect d’une infection à SARS-CoV-2” [Form for the care of a stable patient suspected of infection with SARS-CoV-2]. He also suggested to move the question avez-vous de la peine à respirer ? [do you have trouble breathing?] into Symptômes instead of Patient à risque. He looked randomly at some dropdown menus and thought their choice content is easy to guess.
Doctor 3 raised concerns about some missing important questions in particular in subdomain Posologie of Ordonnance domain; for the time being, there are no questions about adverse drug reactions (headache, nausea, diarrhea, vertigos, insomnia, sedation, sweating, decreased libido, etc.) (Resplandy, 2014) which are among the most important warnings a patient should know when taking medication. He also agreed that it is unnecessary to have a list of medication names since they will be shown to the patient. In case a patient was taking a treatment before the consultation, he should come perhaps for the next consultation with a prescription or the relevant treatment, or write down its name if possible. He also said that putting fewer sentences in the application is more helpful, and that visual images are certainly convenient, even though fewer colors and details might be even better. He also pointed out the absence of instructions about removing metallic objects (earrings, necklace, etc.) before doing an XR or IRM and requested completing the lists of specialists a patient has visited, for instance avez-vous un cardiologue ? [do you have a cardiologist?]. Anyway, he prefers calling an interpreter rather than using any application, so that he can be completely concentrated on the patient and it corresponds better to a normal consultation.
An interesting opinion was provided by Doctor 4 due to his bilingual French and Albanian language skills: he could understand the translations and say that the Albanian provided is the one of Albany, but the people from Kosovo have good chances to understand it. For sentences which have a dropdown menu choice of countries as an answer, he noticed that Kosovo was missing. Same as Doctor 2, he found that important questions were missing. In domain Neuro/Endocrino under Thyroide subdomain, he would like to see more questions about thyroid problem symptoms, giving as an example that in the Balkans, people often develop a hypothyroid problem. In subdomain Vertiges of ORL domain, there should be such differentiation between les vertiges sont-ils comme sur un bateau ? [are vertigos like on a boat?] and les vertiges sont-ils comme sur un carrousel ? [are vertigos like on a carousel?]. In Psy domain under Troubles anxiodépressifs subdomain, questions about suicide should be completed by the question if a concrete project (where, when, how) to commit suicide exists,
92 and if it is possible to make a contract of non-suicide or not. Moreover, some questions to detect PTSD (Post-Traumatic Stress Disorder) are missing (reviviscences of a traumatic situation, anhedonia, etc.).
Doctor 5 liked the history and the PDF log display very much, saying it is really helpful to have an overview of the used sentences at any time. He liked the fact that the icons representing the patient’s answers are explained in the PDF caption, so when it is printed out, there can be no confusion during a later reading of the document. The answers located on the left of the history and PDF log seemed a bit unusual to him, although he thinks he can get used to it. He stated also that it is very important to have some sentences which convey empathy, caring and reassurance, such as ne vous inquiétez pas [don’t worry], êtes-vous d’accord ? [do you agree?], cela ne fera pas mal [it won’t hurt] which are already present and wanted to add more of these. As a family doctor, he confirmed how important it is to create a trustful and positive environment from both sides. He was asked if he has ever come across a situation where he had to call an interpreter. He responded that luckily such situation never appeared, but that he already had some German, Italian and Spanish speakers with whom speaking English was difficult. He showed us the book he used for these consultations (Guardiola & Gruber, 1985) which enabled him to properly question and inform the patients, although the sentences are unfortunately not so well structured, and that domains and subdomains subdivision similar to MobileDr would have helped a lot to find the questions in the book.
Doctor 6 was especially interested in the domains linked to his specialty and liked their subdomains division. He especially did not feel comfortable by using the speech recognition tool in the original version. He preferred selecting the sentences himself without waiting for the tool to find some sentence which did not really correspond to his initial speech input. He added that if various sentences are proposed, it still takes time to choose the right one (if it exists), so there might be no time saving. Likewise, he asserted the screen touch property makes the navigating between domains, subdomains and other features very easy, unlike classical desktops installed in hospitals and offices which are hardly ever tactile. He said that a layout with images and colors appears more playful and attractive both for the patient and the doctor. However, he found that closed questions are indeed recommended for beginner doctors, but not so much for a motivational interview, which is a method of communication
93 with his patients’ profile (Miller & Rollnick, 2013). He would prefer open questions where the patient can verbally speak to answer.
Concerning the layout, Doctor 7 asserted that it is comfortable to have more space between the questions to click on without making repetitive mistakes. He described MobileDr concept as very useful for PSM [Migrants health program] and CAMSCO [Mobile ambulatory consultation of community care], both being HUG programs. He proposed a different way of sentence classification depending on the answer given: if the answer to a question is “yes”, certain questions will appear; if the answer is no, some other questions will appear. He thought it could be useful to add a button leading directly to the main menu from the pages containing the list of sentences, as well as some missing questions in Traumato domain under Accidents subdomain about losing consciousness, losing sensitivity, feeling of tingling and a pain scale. Regarding the available languages, he advised to add Tibetan and Mongolian because not so long ago, there were many people speaking these languages admitted to the hospital.
Doctor 8 was very interested in using the application on a tablet, because the screen is larger than the one of a smartphone. This way, he would not need to wear glasses. Unlike Doctor 5, he is sometimes confronted to a situation where the patient does not speak French at all. The speech recognition tool from the original application did not work much with him, because he is not a native French speaker and thus has a foreign accent. The only way for him was to find the questions by himself, which he considered as easy on the new application thanks to the questions grouping. His center has not budgeted to finance interpreters; neither can his patients afford it. Therefore, the patients’ relatives are responsible for translating the whole consultation, which proved to be difficult because of confidentiality reasons and lack of a neutral position. Situations like domestic violence, personal problems and announcement of bad news are particularly delicate. They generate emotions not only from the patient, but also from the translator who is close (familywise or culturally) to the patient. MobileDr, he said, could largely facilitate the communication and would enable the patient to be alone with the doctor, as it should be. He did not seem to find the clicking process too long before finding a question and estimated that a preliminary study to get familiarized with the new application’s structure would help doctors overcome the many subdomains formatting. Medical doctors are used to study a large number of methods and cases, so it should not burden them much, he said. He asked to possibly add more questions regarding his medical field and also joined the
94 idea of Doctor 6 who would like to use the motivational interview as a questioning method, which is commonly used by specialists.
4.8 Conclusion
In this section, the results of three SUS questionnaires for both BabelDr and MobileDr were analyzed. The main motivation was to shine a light on the two main research questions of this thesis.
It has been found qualitatively that on a mobile device, MobileDr is generally considered easier to use, as well as easier to learn compared to BabelDr. The organization and layout of MobileDr, designed to support the common medical practices, have been appreciated by the participants. Small differences of opinions regarding each doctor individually, or already existing experience with BabelDr have been discussed. A quantitative analysis of mean values and standard deviations for both BabelDr and MobileDr tends to confirm the qualitative observations, but are limited due to small sample size and non-random participant selection.
A breakdown of the wishes and the suggestions made by the doctors interviewed has been presented. It has allowed to identify questions that could be added to the already existing question pool.
95
5 POSSIBLE FUTURE IMPROVEMENTS
This chapter presents several ideas for future possible improvements of MobileDr. In Section 5.1 new sentences are proposed. Section 5.2 contains propositions for new domains and subdomains, in Section 5.3 a possible link to the patient’s file is discussed, in Section 5.4 keyword search tool is mentioned and in Section 5.5 new languages are proposed.
5.1 New sentences
First and foremost, the application could be improved according to the doctors’ wishes. Since they tested the application in our presence and answered questionnaires, we have the chance of having their spontaneous and in real time opinions.
As already explained, the application’s structure and sentences classification was defined against the pool of already existing pre-translated sentences. However, even after having carefully checked all 11’000+ sentences one by one, it clearly appeared that important sentences are still missing in that pool. This statement was indeed confirmed by the eight doctors who evaluated MobileDr. Here are some propositions of formulated sentences according to the doctors’ suggestions for some domains and subdomains.
In Ordonnance > Posologie (suggested by Doctor 3):
– ce médicament pourrait causer des effets secondaires comme | [dropdown menu] | des maux de tête – des nausées – une diarrhée – des vertiges – une insomnie – une somnolence – des sueurs – une diminution de libido
[this drug could trigger side effects like | [dropdown menu] | headache – nausea – diarrhea – vertigos – insomnia – sedation – sweating – decreased libido]
In Endocrinologie > Thyroïde/Parathyroïdes (suggested by Doctor 4):
– avez-vous souvent chaud ? [do you often feel hot?]
– avez-vous souvent froid ? [do you often feel cold?]
– votre taux de cholestérol est-il élevé ? [is your cholesterol level high?]
96
In ORL > Vertiges (suggested by Doctor 4):
– les vertiges sont-ils comme sur un bateau ? [are vertigos like on a boat?]
– les vertiges sont-ils comme sur un carrousel ? [are vertigos like on a carousel?]
In Psy > Troubles anxiodépressifs (suggested by Doctor 4):
– avez-vous élaboré un projet de suicide ? [have you developed a suicide project?]
– accepteriez-vous de revenir me voir demain sans vous faire du mal ?
[would you agree to come back to see me tomorrow without hurting yourself?]
In Psy > new possible subdomain “PTSD” (suggested by Doctor 4):
– revivez-vous votre traumatisme pendant la nuit ? [do you relive your traumatism overnight?]
– avez-vous des flashbacks ? [do you have flashbacks?]
– êtes-vous constamment en alerte ? [are you constantly on alert?]
In new possible domain Empathie/Rassurance (suggested by Doctor 5):
– je comprends ce que vous ressentez [I understand how you feel]
– je comprends votre situation [I understand your situation]
– je me soucie de votre état [I care about your condition]
– faites-moi confiance, s’il vous plaît [trust me, please]
– rassurez-vous, je suis là pour vous aider [do not worry, I am here to help]
Sometimes and depending on the situation, physicians, especially specialists, like to proceed during their consultations by what is called “diagnostic scores”. It is a set of specific questions. Depending on how many questions triggered a “yes” answer the risk of a particular disease will be either probable or improbable (RMS, n.d.-a). We present an example taken from the Revue médicale Suisse which is a reference continuing education journal for practicing physicians. It is called score CAGE-DETA (Ewing, 1984; RMS, n.d.-b) which stands for: Cutting down (the alcohol consumption), Annoyance by criticism (of surrounding people), Guilty feeling (caused by excessive consumption), Eye-opener (about an exaggerated behavior) (in English) – Diminuer, Entourage, Trop, Alcool (in French). This score is used to identify an alcoholic disease. If two or more questions have “yes” answers a problem with alcohol can be suspected.
97 In Habitudes > Alcool (suggested by Doctor 8):
Score CAGE-DETA
– avez-vous déjà ressenti le besoin de diminuer votre consommation de boissons alcoolisées ? [have you ever felt the need to cut back on alcoholic beverages?]
– votre entourage vous a-t-il déjà fait des remarques au sujet de votre consommation ? [have people around you ever made comments about your consumption?]
– avez-vous déjà eu l’impression que vous buviez trop ? [have you ever felt like you were drinking too much?]
– avez-vous déjà eu besoin d’alcool dès le matin pour vous sentir en forme ? [have you ever needed alcohol first thing in the morning to feel good?]
The scores names could be put as “subtitles” among the sentences to make them more visible,
The scores names could be put as “subtitles” among the sentences to make them more visible,