User Case Study

In order to measure the efficiency of our collaborative system in terms of usability, we ran a controlled experiment over 10 participants (5 males and 5 females, with ages between 20 to 36 years old). The participants consisted of 2 experts and 8 students, where an expert was considered to be a user experienced with knowledge in medical image analysis.

5.6.3.1 Experiment Design

Based on the two scenarios, repeated measures experiments were conducted. Only one indepen-dent user feedback variable, the type of collaborative editing algorithm, was manipulated. Two dependent variables were measured as follows.

• Task Completion Time: time taken to complete the segmentation on a single image.

• Segmentation Error: errors between segmented and the reference segmentation.

We also measured user satisfaction using questionnaire as follows.

• User Satisfaction: which type of algorithm fulfilled the best requirements for the specific scenario? We devised a questionnaire based on a well-known IBM Post-Study question-naire [116]. 10 questions corresponding to the system usefulness was designed where each question used 5 point graphic scales, anchored at the end points with the terms ’Strongly agree’ for 1 and ’Strongly disagree’ for 5. Some space was left at the end of the question-naire for comments.

5.6.3.2 Tasks and Procedures

Participants were asked to learn our system with a simple introductory manual which exempli-fies the usage of the system. All participants had no prior exposure to the application. After participants were given only a basic introduction and a few minutes to play with the application, they were further instructed on the use of the application by participating in the ’student-expert’

scenario. Participants were given two tasks: each participant had to collaborate with another user in a segmentation task on an image using firstly the strict locking algorithm and then our proposed collaborative mechanism. The segmentation consisted in segmenting the 4 bones on a MRI (single slice image) as accurately as possible within the shortest time possible. Bone mod-els were initialized sufficiently close to the structures to be segmented so that the users could drive the meshes towards the structures through applying the three types of CPs (as described in section 3.9.4). The ground truth segmentation was a priori computed by experts for use as a benchmark in calculating the segmentation error. For a given mesh contour resulting from a user

segmentation, an error e1 was computed as the Euclidean distance between each point of the contour and its closer point on the reference contour. The same was done by inverting roles of user and reference contours to get an errore2. The final segmentation error for a mesh was thus the sum e1+e2. By averaging the errors for all bones, we got the so-called average symmetric distance error in mm. Pairs of participants were randomly chosen. After completing the tasks, participants filled in a questionnaire.

5.6.4 Results

We calculated discrete statistics for every dependent variable and user satisfaction.

• Task Completion Time

Figure 5.12a shows the average time taken by non-expert participants to complete their task. Using the proposed system, participants finished their task earlier compared to the strict locking based system because participants could update their inputs without waiting the other user to release the lock. Another reason is that each participant could work on different model contours at the same time. In some cases, participant did not release the lock preventing the other participant to improve the segmentation even though she/he had a better idea to complete the task.

00:00

Figure 5.12:Completion time and segmentation error results.

• Segmentation Error The obtained results revealed that participants’ segmentation error us-ing strict lockus-ing was 4 percent greater on average than usus-ing the proposed system, as shown in figure 5.12b. From our observation, this was attributed to the fact that the par-ticipants were able to correct each other’s errors, where the agreements from both the participants were more accurate than from an individual user.

• User Satisfaction From the questionnaire, a qualitative user satisfaction was derived. Most people were strongly or moderately satisfied with the collaborative algorithm as shown in figure 5.13. No participant strongly disapproved the use of both the proposed algorithm and the strict locking algorithm. Similarly, participants reported that they could effec-tively complete their tasks both with the proposed algorithm and strict locking algorithm.

However, participants felt more comfortable to collaborate with others with the proposed algorithm because more freedom was given and it was not necessary to wait for the lock to

be released.

Figure 5.13:Average of the selected questions in questionnaire.

Participants were asked for comments or suggestions in the questionnaire. Two participants reported that strict locking algorithm needed a floor control mechanism requesting a lock or at least other communication facilities, such as chat, to negotiate their turn because they felt uncomfortable to wait until lock is released. Two participants reported that they felt a little-bit uncomfortable to interact with others because the simulation responsiveness was relatively low in both cases because their inputs (constraint points) progressively affected the segmentation instead of having an instantaneous effect. As mentioned in section 5.6.2, a publisher run in a more powerful workstation or the use of segmentation optimizations would yield a faster simulation. However, as explained in section 3.9.3, this progressive change of the segmentation evolution is essential to be able to detect and correct errors sufficiently early. Finally, in the context of our specific physically-based segmentation, cre-ating large “brutal” changes may create instabilities in the simulation and thus ultimately affect the quality of the segmentation.

5.6.5 Concluding Remarks

In this experiment we tested a collaborative telemedicine system for real-time and interactive seg-mentation of volumetric medical images and demonstrated its usages using two typical case sce-narios: teacher-students and expert-expert collaboration. User evaluation was conducted which measured the enhancements with our approach in comparison to the conventional strict locking collaborative system that is often found in medical systems. Our system performance results indicated that, prior to code optimization, it can support large number of users for online collab-orative editing of 3D volumetric medical images. View generation speed was heavily dependent on the computational resources (CPU, memory size, video card capability) of the producer side rather than the subscriber side or network condition because subscribers used less thanone per-cent of CPU consumption. Our user study results revealed that our proposed system can be useful in teleradiology context and has many potential clinical applications. As our system is designed to be modular, the system can be integrated to other, more complete, medical image viewers, such as the popular open source image viewers built using Insight Segmentation and

Registration Toolkit (ITK)². Our system is also not restricted to the use of the deformable model segmentation algorithm as presented in this study. This algorithm was chosen for its inherent characteristics to evolve from rough to optimal segmentation results via an iterative and intuitive visual feedback. The advantage of visual feedback is in the ability for the users to understand the segmentation process as it iterates to the final result. We conducted some preliminary tests on a mobile device (an UMPC) over a wireless network, and similar performance was observed.

Future work will mostly focus on dynamic polymorphic presentation (view and interface) adap-tation and context-aware network adapadap-tation according to the current device capability in order to support nomadic users to collaborate with each other exploiting diverse devices.

5.7 Collaborative Services with shared Data Models

We look at two aspects of the implementation which are thecollaborative pattern designerand the GPU based cloth simulation.