Preliminary Evaluation of the Pattern Cutting and the Ligating Loop Virtual Laparoscopic Trainers

HAL Id: hal-01030809

https://hal.archives-ouvertes.fr/hal-01030809

Submitted on 1 Sep 2014

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Ligating Loop Virtual Laparoscopic Trainers

Amine Chellali, Woojin Ahn, Ganesh Sankaranarayanan, J. T. Flinn, Steven D. Schwaitzberg, Daniel B. Jones, Suvranu De, Caroline G. L. Cao

To cite this version:

Amine Chellali, Woojin Ahn, Ganesh Sankaranarayanan, J. T. Flinn, Steven D. Schwaitzberg, et al..

Preliminary Evaluation of the Pattern Cutting and the Ligating Loop Virtual Laparoscopic Trainers.

Surgical Endoscopy, Springer Verlag (Germany), 2015, 29 (4), pp.815–821. �10.1007/s00464-014-3764-

7�. �hal-01030809�

laparoscopic trainers

Running head: Chellali et al.: Preliminary Evaluation of the VBLaST-PT

and VBLaST-LL

A. Chellali

, W. Ahn

, G. Sankaranarayanan

, J. T. Flinn

, S. D. Schwaitzberg

, D.B. Jones

, Suvranu De

, C.G.L. Cao

1

Department of Surgery, Cambridge Health Alliance, Harvard Medical School, Cambridge, MA, USA

2

Department of Computing Engineering, University of Evry, IBISC Laboratory, Evry, France

3

Center for Modeling, Simulation and Imaging in Medicine, Rensselaer Polytechnic Institute

4

Department of Biomedical, Industrial and Human Factors Engineering, Wright State University, Dayton, OH, USA

5

Department of Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA

Presented as an oral presentation at the SAGES 2014 annual meeting.

Funding information: This work was supported by NIBIB/NIH Grant # R01EB010037.

Corresponding Author:

Caroline Cao

Wright State University 207 Russ Engineering Center 3640 Colonel Glenn Hwy Dayton, OH 45435

Email: caroline.cao@wright.edu Phone: (937) 775-5044

Fax: (937) 775-7364

ABSTRACT

Introduction: The Fundamentals of Laparoscopic Surgery (FLS) trainer is currently the

standard for training and evaluating basic laparoscopic skills. However, its manual scoring system is time-consuming and subjective. The Virtual Basic Laparoscopic Skill Trainer (VBLaST

) is the virtual version of the FLS trainer which allows automatic and real time assessment of skill performance, as well as force feedback. In this study, the VBLaST

pattern cutting (VBLaST-PC

) and ligating loop (VBLaST-LL

) tasks were evaluated as part of a validation study. We hypothesized that performance would be similar on the FLS and VBLaST

trainers, and that subjects with more experience would perform better than those with less experience on both trainers.

Methods: Fifty-five subjects with varying surgical experience were recruited at the Learning

Center during the 2013 SAGES annual meeting and were divided into two groups: experts (PGY 5, surgical fellows and surgical attendings) and novices (PGY 1-4). They were asked to perform the pattern cutting or the ligating loop task on the FLS and the VBLaST

trainers.

Their performance scores for each trainer were calculated and compared.

Results: There were no significant differences between the FLS and VBLaST

scores for either the pattern cutting or the ligating loop task. Experts’ scores were significantly higher than the scores for novices on both trainers.

Conclusion: This study showed that the subjects’ performance on the VBLaST

trainer was similar to the FLS performance for both tasks. Both the VBLaST-PC

and the VBLaST-LL

tasks permitted discrimination between the novice and expert groups. Although concurrent and discriminant validity has been established, further studies to establish convergent and predictive validity are needed. Once validated as a training system for laparoscopic skills, the system is expected to overcome the current limitations of the FLS trainer.

Keywords: Surgical training, virtual reality (VR), Virtual Basic Laparoscopic Surgical

Trainer (VBLaST), Fundamentals of Laparoscopic Skills (FLS), Force feedback

INTRODUCTION

Laparoscopic surgery is the preferred technique for many general procedures from the

patients’ perspective (smaller incisions, less blood loss, decreased analgesic requirements, and quicker postoperative recovery) [1]. However, laparoscopic surgeons develop skills and face many challenges that are different from what they experience in open surgery. These skills include handling tissues with instruments that have limited degrees of freedom and that provide limited haptic feedback, or dealing with altered hand-eye coordination due to changes in depth perception and to indirect vision [2]. Therefore, there is a need to establish dedicated training curricula and assessment tools to help surgeons build an acceptable level of

laparoscopic skills and thus maintain safe surgical practices [3, 4].

In this context, surgical simulators have gained momentum in the last decade as the training environments of choice for laparoscopic surgery [5]. One of the currently most popular simulators is the Fundamentals of Laparoscopic Surgery (FLS) trainer. This physical-box trainer, based on the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) [6], was developed to teach and measure basic laparoscopic skills through five basic tasks: peg transfer, pattern cutting, ligating loop, suturing with intracorporeal knot, and suturing with extracorporeal knot [4].

The FLS trainer is currently used as the standard method for assessing the proficiency of laparoscopic surgical skills by the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) and the American College of Surgeons (ACS) [7], and the successful completion of the FLS examination (including a didactic component) has become a requirement for all general surgeons in the USA since 2009.

However, the FLS trainer is relatively expensive and resource intensive as it requires a large

supply of consumables, lacks objective methods for performance assessments, and is time-

consuming and labor intensive [6, 8, 9].

Virtual Reality (VR) technology provides a safe and versatile practice medium for teaching laparoscopic skills [10] and can overcome some limitations of the physical trainer box. For example, the VR-based systems can be used multiple times, with no need of consumables.

They provide objective and automated measurement of performance, which can improve the evaluation process [11]. Furthermore, they can be designed to provide haptic feedback, an essential component for training minimally invasive surgery skills [10, 12, 13, 14, 15, 16].

However, before introducing this technology into training curricula, there is a need to demonstrate its validity as a training tool and also as an instrument for assessing skill in laparoscopic surgery [17].

Lap-X (Epona Medical, Rotterdam, The Netherlands) [18] LAP Mentor (Simbionix USA, Cleveland, OH) [19], and LapVR (Immersion Medical, Gaithersburg, MD) [20] are examples of commercially available VR systems for laparoscopic surgery training. However, some of these systems lack realistic haptic feedback, are costly, or have not been validated for training of laparoscopic skills (e.g., Lap-X) [21, 22, 23, 24].

The Virtual Basic Laparoscopic Surgical Trainer (VBLaST

) is a new VR-based training system, which simulates the five basic laparoscopic tasks present in the FLS trainer box [9].

The main motivation of designing this system was to overcome some of the current limitations of the FLS trainer. As such, VBLaST

includes automated assessments of laparoscopic skills and repeated trials with no need to replenish materials. Moreover, the system provides haptic feedback to the user through a haptic device. The VBLaST

simulating the peg transfer task of the FLS trainer has been previously validated [24, 25]. In this study, two new tasks of the VBLaST

system, namely, the pattern cutting (PC) task and the ligating loop (LL) task, are evaluated.

This research is part of a larger study which aims to validate the VBLaST as a training system

for laparoscopic surgical skills. The objective of this study was to compare the current version

of the VBLaST

system, which includes the pattern cutting and ligating loop tasks, to the FLS trainer. It was hypothesized that subjects’ performance would be similar on VBLaST

and the FLS trainers (H1) for both tasks. In addition, subjects with more experience would perform better than those with less experience or no experience at all, when using either the FLS or the VBLaST

(H2) for both tasks.

METHODS

Participants

Fifty-five (55) subjects (25-56 years old) with varied experience in surgery were recruited in this Institutional Review Board (IRB) approved study at the Learning Center during the 2013 SAGES annual meeting. Three of the subjects were left-handed. Thirty-three subjects ( 27 males, 6 females, 2 left handed) performed the pattern cutting task while twenty-two subjects (20 males, 2 females, 1 left handed) performed the ligating loop task.

For each task, subjects were divided into two groups according to experience (Table 1):

experts (surgical fellows and practicing laparoscopic surgeons) and novices (PGY 1-4).

Table 1: Composition of the subject groups per task

Apparatus

The FLS and the VBLaST

(Figure 1) were used to perform the pattern cutting and ligating loop tasks. For the pattern cutting task, the work space of the FLS trainer consisted of a piece of gauze with a pre-marked black circle, placed in the center of the FLS box trainer and secured with clips. For the ligating loop task, the work space of the FLS trainer consisted of a

Groups Novices Experts

Expertise

PGY1 PGY2 PGY3 PGY4 PGY5 Fellows Practicing

surgeons

Pattern cutting task

5 4 7 1 0 6 11

Ligating loop task

2 4 4 1 0 4 7

red organ-shaped foam, with appendages, placed in a clip at the center of the FLS box trainer such that the appendages are free and visible in the field of view.

Figure 1: (A) the pattern cutting trainers, (B) the ligating loop trainers (Left trainer:

VBLaST©, Right trainer: FLS trainer)

The VBLaST

simulator consists of computational software to graphically simulate the FLS work space for both tasks (Figure 1), and a physical user interface to connect different

laparoscopic instruments (one Maryland dissector and one endoscopic scissors for the pattern cutting task; one pre-tied ligating loop, one grasper with locked handle and one endoscopic scissors for the ligating loop task) to two PHANTOM Omni haptic devices (Geomagic Inc., Boston, MA, USA). These devices allow force feedback to be transmitted to the users when interacting with the virtual environment. For the FLS tasks, one experimenter manually recorded the subjects’ task completion time with a stop watch. Penalty scores for the LL task were calculated manually by the same experimenter after each trial. PC penalty scores (based on the deviations from the circle cutting) were calculated using a custom-built image analysis tool that has been validated and reported elsewhere [26]. The method consists of digitally scanning the cut pieces of gauze and analyzing them by the image analysis software to compute the area of error, both inside and outside the prescribed circle. This automatic scoring method provides a quick, accurate, and consistent error measurement for PC scoring.

The performance measures for the VBLaST

(both time and penalties) were automatically

recorded by the system for both tasks.

Tasks

The pattern cutting task

For this task, the subjects were given one Maryland dissector and one endoscopic scissors, and were allowed to interchange the left and the right instruments only at the beginning of the task. One hand was used to provide traction on the gauze using the Maryland dissector and to place the gauze at the best possible angle to the cutting hand. The subjects were instructed to cut the gauze as accurately as possible along the marked circle. Timing started when the gauze was grasped and ended upon completion of cutting the marked circle. A penalty was assessed for any deviation from the line demarcating the circle.

The ligating loop task

For this task, the subjects were required to place a pre-tied ligating loop around the tubular foam appendage and secure the knot on a provided black mark. The subjects were given one pre-tied ligating loop, one grasper with a locking handle, and one endoscopic scissors. They were instructed to insert the ligating loop into the field of view and to position it properly. The knot had then to be secured on the mark near the base of the foam appendage by sliding the pusher rod down. Once the knot was secured, the string had to be cut using the scissors.

Timing started when the appendage was grasped and ended when the string was cut. A penalty was assessed for any deviation from the predetermined marking.

Procedure

Before the start of the experimental session, subjects were asked to fill out a questionnaire detailing the demographics and their previous laparoscopic surgery experience. They were then shown an instructional video describing the pattern cutting task or the ligating loop task.

After that the subjects were asked to perform one trial of the assigned task on both the FLS

and the VBLaST

trainers. To control the effect of simulators’ order on the performance, the

presentation order of the simulators (FLS vs VBLaST) was counterbalanced. That is, half the

subjects were randomly assigned to begin with the FLS, while the other half of the subjects began with the VBaST. At the end of the session, the subjects were asked to complete another questionnaire evaluating the features of the VBLaST

trainer compared with the FLS trainer, using a 5-point Likert scale (from very poor/not satisfactory to very good/very satisfactory).

The questions were related to visual appearance, haptic feedback, 3D perception, tool

movements and overall quality and reliability of the system as a training and assessment tool.

As a further step in data collection and validation, the subjects were given the opportunity to comment on their experience with the system after session completion.

Dependent measures

For the FLS trainer, the dependent measures consisted of a total raw score (ranging from 0 to 300) calculated using undisclosed formula for the FLS [6]. For each task, the raw score was calculated using a combination of the task completion time and the penalty score. These raw scores were then divided by a normalization factor that was calculated using the best expert FLS score obtained in the FLS condition [6, 24]. The FLS normalized scores ranged from 0 to 100. For the VBLaST

trainer, the raw score was automatically calculated by the system using the same formula. These raw scores were then divided by a normalization factor calculated using the best expert score obtained from the VBLaST

condition in the experiment. The VBLaST

normalized scores also ranged from 0 to 100.

Data analysis

Since the data were not normally distributed (tested using the Shapiro-Wilk test), non- parametric statistical tests were used to analyze the data. The Spearman correlation test, the Wilcoxon signed rank sum test, and the Mann-Whitney's U-test were used, wherever

appropriate. P values under 0.05 were considered significant. Descriptive statistics (the mean

subjective ratings and the standard errors) were used to analyze the questionnaire data. All

analyses were performed using SPSS v.21.0 (IBM Corp., Armonk, NY, USA).

RESULTS

The pattern cutting task

Correlation tests

The Spearman’s correlation test showed that the pattern cutting FLS PC and VBLaST-PC

mean normalized scores had a correlation of 0.47 (Spearman’s r(31) = 0.47, p = 0.006).

The effect of experience and simulator order

The Mann Whitney U-test showed that the FLS PC and the VBLaST-PC

scores (Figure 2) were significantly higher in the expert group than in the novice group (U = 39, Z = 3.49, p <

0.0001; U = 68, Z = 2.45, p = 0.01, respectively).

Finally, the Mann Whitney U-test showed no significant effect of the order of simulators either on the FLS scores or on VBLaST scores (U = 133, Z = -0.10, p > 0.05; U = 99, Z = 1.33, p > 0.05, respectively).

The effect of simulator

Figure 2: effect of experience for each simulator on the performance scores in the pattern

cutting task (error bars represent the standard error)

The Wilcoxon signed rank sum test showed no significant difference (Figure 2) between the FLS PC scores and the VBLaST-PC

scores (Z = 1.94, p > 0.05).

Subjective evaluation

Figure 3: Subjective evaluations of the VBLaST-PC

trainer (error bars represent the standard error)

The subjective ratings of the VBLaST-PC

features are shown in Figure 3. The average rating for “realism of objects” was the highest. “Usefulness of VBLaST for training laparoscopic skills” was rated second highest. “Realism of instrument handling”, “quality of force

feedback” and “trustworthiness in quantifying performance measures” had the lowest ratings.

Only 28% of the subjects preferred using the VBLaST-PC

over the FLS box for training laparoscopic surgery skills.

The ligating loop task

Correlation tests

The Spearman’s correlation test showed that the FLS LL and VBLaST-LL

mean normalized

scores had a correlation of 0.53 (Spearman’s r(20) = 0.53, p = 0.01).

The effect of experience and simulator order

The Mann Whitney U-test showed that the FLS LL and the VBLaST-LL

scores were significantly higher (Figure 4) in the expert group than in the novice group (U = 27.5, Z = 2.16, p = 0.03; U = 34, Z = 2.13, p = 0.03, respectively).

Finally, the Mann Whitney U-test showed no significant effect of the order of simulators on either the FLS scores or VBLaST scores (U = 52.5, Z = -0.52, p > 0.05; U = 43.5, Z = -1.11, p

> 0.05, respectively).

The effect of simulator

The Wilcoxon signed rank sum test showed no significant difference (Figure 4) between the FLS LL scores and the VBLaST-LL

scores (Z = 0.21, p > 0.05).

Figure 4: effect of experience for each simulator on the performance scores in the ligating loop task (error bars represent the standard error)

Subjective evaluation

The subjective ratings of the VBLaST-LL

features are shown in Figure 5. The average ratings for “realism of objects” and “usefulness of VBLaST for learning” were the highest.

“Realism of instrument handling” and “usefulness of force feedback” had the lowest ratings.

Only 30% of the subjects preferred using the VBLaST-LL

over the FLS for training laparoscopic surgery skills.

Figure 5: Subjects’ ratings of the VBLaST-LL© trainer (error bars represent the standard error)

DISCUSSION

VR is a promising technology that can overcome some limitations of the existing laparoscopic box simulators such as the FLS trainer. This study aimed to evaluate two tasks on the

VBLaST

system, the new VR-based trainer for basic laparoscopic technical skills. By comparing the VBLaST

to the validated FLS trainer, we showed that the subjects’

performance for the pattern cutting and ligating loop tasks was similar on both systems. This finding supports our first hypothesis and suggests that the VR simulation has faithfully reproduced the content of the FLS system.

The results also demonstrated the concurrent validity of the VBLaST

system, i.e., allowing

discrimination between the novice and expert groups. For both tasks, the experts performed

significantly better than novices.

Finally, the results showed that the VBLaST

scores and the FLS scores were significantly correlated for both tasks although the correlation coefficients were of medium strength (0.47 and 0.53). These results suggest that some aspects of the FLS system were correctly

reproduced in VBLaST while others require some improvements to acquire a higher correlation score. In this context, the subjective evaluation and the participants’ comments provide valuable information for future improvements of the VBLaST

user interface.

Although they were satisfied by the quality of the graphical user interface in both tasks, the subjects’ rating of the realism of instrument handling was low. This suggests that they had difficulty adapting to the use of real instruments with virtual objects. Moreover, the subjects rated the quality of the system’s force feedback as low. They commented that the haptic feedback was more important during the pattern cutting task than in the other FLS tasks. In fact, the main learning objective of the task was to be able to apply the correct tension/counter tension on the tissue to cut it properly. Although the quality of the system’s haptic feedback did not prevent the users’ from completing this task, improvements in the realism of haptic feedback will be necessary to achieve a higher quality training simulator.

For the ligating loop task, the subjects commented that the virtual loop behaved differently from the actual tool. For instance, the subjects felt that the virtual loop slides too easily on the appendage, preventing them from securing it accurately on the mark. While this did not prevent the task from being completed, it is necessary to improve the virtual physics to ensure the learning of correct laparoscopic skills in this task.

Finally, although the users agreed on the usefulness of VR technology for training

laparoscopic skills, most of them preferred to use the FLS trainer. Hence, improvements in the

manipulation of the tools and the realism of haptic feedback will be necessary to improve the

user’s willingness to adopt this new technology.

LIMITATIONS AND FUTURE WORK

The limited number of trials per subjects on each simulator (only one trial per system) may have influenced the users’ performance in our study. Nevertheless, it provided us with the much needed feedback at an early stage of the system development and served to inform our design modifications of the VBLaST

system. Our design approach for the VBLaST

system utilizes the user-centered design method (as prescribed by ISO 13407). This method is based on an iterative process and recommends involving the end users during the different design stages [27]. The study findings suggest that some components of the system were correctly reproduced while others will require another design iteration to be improved. Future developments will include improvement of the physical user interface and the virtual

environment physics with a focus on the learning objectives of each task. For instance, force sensors will be mounted on real instruments to measure the exact forces felt by the trainees when using the real FLS box. These measurements can then be used to improve the haptic feedback on the VBLaST system. Moreover, other measurements will also be used to improve the physical behavior of the virtual objects and enhance the realism of the VBLaST system.

Currently, only one study has been conducted to compare the automated assessment method of the pattern cutting task used in the VBLaST system with the manual scoring method currently in use showing advantages in speed, accuracy, and precision for the automated method [26]. More investigations are necessary to validate the automated scoring method for the other tasks and further demonstrate the value of the VBLaST system.

Although not addressed here, the issue of cost savings in consumables must be balanced by

the cost of the VR simulator itself. Further studies will be conducted in the future before

offering the VBLaST

as an alternate training standard for laparoscopic surgery.

AKNOWLEDGEMENTS

This project was supported by National Institutes of Health (NIH) Grant NIBIB R01 EB010037-01. This research was conducted in collaboration with the SAGES FLS Committee.

DISCLOSURE

Dr. Steven, D. Schwaitzberg is a consultant for Stryker and Olympus and member of the advisory board for NeatStitch, AcuityBio, MITI, Cambridge Endo and Surgiquest. Jeff Flinn and Drs. Amine Chellali, Woojin Ahn, Ganesh Sankaranarayanan, Daniel B. Jones, Suvranu De, and Caroline, G.L. Cao have no conflicts of interest or financial ties to disclose.

REFERENCES

[1] Maithel S, Sierra R, Korndorffer J, Neumann P, Dawson S, Callery M, Jones D, Scott D (2006) Construct and face validity of MIST-VR, Endotower, and CELTS: are we ready for skills assessment using simulators? Surg Endosc 20(1): 104-112

[2] Cao CGL, MacKenzie CL (1997) Direct, 2-D vs. 3-D endoscopic viewing & surgical task performance. A Symposium on Evolving Technologies: Surgeons' Performance of Surgical Tasks. J. Sport and Exercise Psychology 19

[3] Adrales GL, Chu UB, Witzke DB, Donnelly MB, Hoskins JD,MMJJ, Gandsas A, Park AE (2003) Evaluating minimally invasive surgery training using low-cost mechanical simulations. Surg Endosc 17(4): 580-585

[4] Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF, Schirmer B, Hoffman K, FLS

CS (2004) Development and validation of a comprehensive program of education and

assessment of the basic fundamentals of laparoscopic surgery. Surgery 135(1): 21-27

[5] Powers TW, Murayama KM, Toyama M, Murphy S, Denham EW, Derossis AM, Joehl

RJ (2002) House staff performance is improved by participation in a laparoscopic skills curriculum. Am J Surg 184: 626–629

[6] Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, & Fried GM (2003) valuating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 17(6): 964-967

[7] Fried GM (2008) FLS assessment of competency using simulated laparoscopic tasks. J Gastrointest Surg 12(2): 210-212

[8] Botden SMBI, Buzink SN, Schijven MPJJJ (2007) Augmented versus Virtual Reality Laparoscopic Simulation: What Is the Difference? World J Surg 31(4): 764–772 [9] Sankaranarayanan G, Lin H, Arikatla VS, Mulcare M, Zhang L, Derevianko A, Lim R,

Fobert D, Cao C, Schwaitzberg SD, Jones DB, De S (2010) Preliminary face and construct validation study of a virtual basic laparoscopic skill trainer. J Laparoendosc Adv Surg Tech A 20(2): 153-157

[10] Chellali A, Dumas C, Milleville-Pennel I (2012) Haptic communication to support biopsy procedures learning in virtual environments. Presence: Teleoperators and Virtual Environments 21(4): 470-489

[11] Grantcharov T, Rosenberg J, Pahle E, Fench P (2001) Virtual reality computer simulation. Surg Endosc 15: 242-244

[12] Reich O, Noll M, Gratzke C, Bachmann A, Waidelich R, Seitz M, Schlenker B, Baumgartner R, Hofstetter A, Stief CG (2006) High-level virtual reality simulator for endourologic procedures of lower urinary tract. Urology 67(6): 1144-1148

[13] Dang T, Annaswamy TM, Srinivasan MA (2001) Development and evaluation of an

epidural injection simulator with force feedback for medical training. Stud Health

Technol Inform 81: 97-102

[14] Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ (2009) The Role of Haptic Feedback in Laparoscopic Simulation Training. J Surg Res 156(2): 312-316 [15] Seymour NE, Gallagher AG, Roman SA, O’Bri BMK, Andersen DK, Satava RM (2002)

Virtual Reality Training Improves Operating Room Performance Results of a Randomized, Double-Blinded Study. Ann Surg 236(4): 458–464

[16] Ström P, Hedman L, Särnå L, Kjellin A, Wredmark T, L. FT (2006) Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc 20(9): 1383-1388 [17] Aggarwal R, Grantcharov TP, Eriksen JR, Blirup D, Kristiansen VB, Funch-Jensen P,

Darzi A (2006) An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 244(2): 310-314

[18] Epona Medical | LAP-X. www.lapx.eu. Available at: http://www.lapx.eu/en/lapx.html.

Accessed June 26, 2013

[19] Zhang A, Hünerbein M, Dai Y, Schlag PM, Beller S (2008) Construct validity testing of a laparoscopic surgery simulator (Lap Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc 22(6): 1440-1444

[20] Iwata N, Fujiwara M, Kodera Y, Tanaka C, Ohashi N, Nakayama G, Koike M, Nakao A (2011) Construct validity of the LapVR virtual-reality surgical simulator. Surg Endosc 25(2): 423-428

[21] Ayodeji ID, Schijven M, Jakimowicz J, Greve JW (2007) Face validation of the

Simbionix LAP Mentor virtual reality training module and its applicability in the surgical curriculum. Surg Endosc 21(9): 1641-1649

[22] Salkini MW, Doarn CR, Kiehl N, Broderick TJ, Donovan JF, Gaitonde K (2010) The

role of haptic feedback in laparoscopic training using the LapMentor II. J Endourol

24(1): 99-102

[23] Gallagher AG, O’Sullivan GC (2012) Human factors in acquiring medical skills; learning and skill acquisition in surgery. In: Gallagher A, O’Sullivan G (eds) Fundamentals of surgical simulation, Springer-Verlag, London, pp 98–118

[24] Arikatla VS, Sankaranarayanan G, Ahn W, Chellali A, De S, Cao CGL, Hwabejire J, DeMoya M, Schwaitzberg S, Jones DB (2013) Face and construct validation of a virtual peg transfer simulator. Surg Endosc 27(5): 1721-1729

[25] Zhang L, Sankaranarayanan G, Arikatla VS, Ahn W, Grosdemouge C, Rideout JM, Epstein SK, De S, Schwaitzberg SD, Jones DB, Cao, C.G.L. (2013) Characterizing the learning curve of the VBLaST-PT© (Virtual Basic Laparoscopic Skill Trainer). Surg Endosc 27(10): 3603-3615

[26] Flinn J, Wood D, Cao CGL (2013) Technology-based procedure for automatic and objective error measurement in FLS pattern cutting task. Proceedings of the Annual Meeting of the Society of Gastrointestinal and Endoscopic Surgeons (SAGES 2013), Baltimore, MD

[27] ISO (1999) Human-centred design processes for interactive systems (ISO 13407:1999).

International Organization for Standardization