Publisher’s version / Version de l'éditeur:
CHI '13 Extended Abstracts on Human Factors in Computing Systems, pp.
2383-2384, 2013-04-01
READ THESE TERMS AND CONDITIONS CAREFULLY BEFORE USING THIS WEBSITE.
https://nrc-publications.canada.ca/eng/copyright
Vous avez des questions? Nous pouvons vous aider. Pour communiquer directement avec un auteur, consultez la
première page de la revue dans laquelle son article a été publié afin de trouver ses coordonnées. Si vous n’arrivez pas à les repérer, communiquez avec nous à PublicationsArchive-ArchivesPublications@nrc-cnrc.gc.ca.
Questions? Contact the NRC Publications Archive team at
PublicationsArchive-ArchivesPublications@nrc-cnrc.gc.ca. If you wish to email the authors directly, please see the first page of the publication for their contact information.
NRC Publications Archive
Archives des publications du CNRC
This publication could be one of several versions: author’s original, accepted manuscript or the publisher’s version. / La version de cette publication peut être l’une des suivantes : la version prépublication de l’auteur, la version acceptée du manuscrit ou la version de l’éditeur.
For the publisher’s version, please access the DOI link below./ Pour consulter la version de l’éditeur, utilisez le lien DOI ci-dessous.
https://doi.org/10.1145/2468356.2468778
Access and use of this website and the material on it are subject to the Terms and Conditions set forth at
We'll take it from here : letting the users take charge of the evaluation
and why that turned out well
Munteanu, Cosmin; Fournier, Hélàne; Lapointe, Jean-François; Emond,
Bruno; Kondratova, Irina
https://publications-cnrc.canada.ca/fra/droits
L’accès à ce site Web et l’utilisation de son contenu sont assujettis aux conditions présentées dans le site LISEZ CES CONDITIONS ATTENTIVEMENT AVANT D’UTILISER CE SITE WEB.
NRC Publications Record / Notice d'Archives des publications de CNRC:
https://nrc-publications.canada.ca/eng/view/object/?id=4ebebddc-f888-4eb0-b980-fc2d2f721231
https://publications-cnrc.canada.ca/fra/voir/objet/?id=4ebebddc-f888-4eb0-b980-fc2d2f721231
We'll take it from here: letting the users
take charge of the evaluation and why
that turned out well
Abstract
The operational challenges faced by law enforcement and public safety personnel are constantly evolving, while the training and certification process has stayed the same. New technologies such as virtual reality, mixed reality, or game-based simulators are being researched as promising enhancements to traditional training methods. However, their widespread adoption, particularly by smaller units, faces barriers such as cost – due in no small part to the difficulties of developing and especially evaluating such large-scale interactive systems. In this case study, we present MINT – a low-cost mixed-reality Multimodal INteractive Training system, aimed at supporting the training of small- and medium-sized law enforcement and infantry units. We discuss the challenges and approaches taken in the participatory design of the training system, its agile-based development and implementation, and its qualitative evaluation with users and subject-matter experts.
Keywords
User studies; evaluation methodology; mixed-reality interaction; immersive gaming.
Cosmin Munteanu1) Cosmin.Munteanu@nrc-cnrc.gc.ca Hélène Fournier2) Helene.Fournier@nrc-cnrc.gc.ca Jean-François Lapointe3) Jean-Francois.Lapointe@nrc-cnrc.gc.ca Irina Kondratova1) Irina.Kondratova@nrc-cnrc.gc.ca Bruno Emond3) Bruno.Emond@nrc-cnrc.gc.ca
National Research Council Canada
1)
Fredericton, NB, E3B 9W4
2)
Moncton, NB, E1A 7R1
3)
Ottawa, ON, K1A 0R6 Canada
Copyright 2013 Her Majesty in Right of Canada.
CHI 2013 Extended Abstracts, April 27–May 2, 2013, Paris, France. ACM 978-1-4503-1952-2/13/04.
ACM Classification Keywords
H.5.2 [User interfaces]: Evaluation/methodology; K.3.1 [Computer Uses in Education]: Computer-assisted instruction.
General Terms
Experimentation; Human Factors.
Summary
The growing training and operational needs of law enforcement and public safety personnel can no longer be met efficiently and effectively through existing infrastructure and resources. While the demands of day-to-day operations are constantly changing, the training of law enforcement personnel and the
certification process has largely stayed the same. While several technological solutions exist for enhancing training, widespread adoption of current approaches and solutions, such as virtual training, is hindered by significant cost barriers and by lack of scalability and reach. This creates challenges for smaller
geographically disconnected units, which characterizes most rural police departments in North America. To address this challenges, we have developed MINT, a technological solution for a game-based, immersive, mixed-reality multimodal Course of Fire and Application of Judgement training, which allows its users to easily modify training scenarios, interact with game avatars through voice commands, and operate laser weapons and various other electronic artefacts. A complete description of the system's implementation can be found in [1], while details of its features to support law enforcement training are presented in [2].
In this case study we report on the evaluation of MINT with our target users – infantry soldiers and police
officers (illustrated in Figure 1). We describe our agile development, combined with a design-evaluation cycle that closely engaged users (a hybrid between
participatory and user-centred design), an approach that ensured the best balance between development needs and our stakeholders' constraints. We also review the challenges we met while conducting such
evaluations, mainly in the form of limited time and number of users that our partners could allocate to the system trials. We conclude with a discussion of our evaluation and qualitative data collection that we employed, and present the lessons learnt and implications for future evaluations under similar constraints.
Case Study
The detailed case study presentation is available at: http://www.academia.edu/2433801/Well_take_it_from _here_letting_the_users_take_charge_of_the_evaluatio n_and_why_that_turned_out_well
References
[1] Fournier H., Lapointe, J.-F., Munteanu C., Emond B., Kondratova, I. (2011) "A Multidisciplinary Approach to Enhancing Infantry Training through Immersive Technologies". In Proceedings of the Interservice/Industry Training, Simulation and Education Conference – I/ITSEC.
[2] Fournier H., Lapointe, J.-F., Kondratova, I, Emond B., Munteanu C. (2012) "Crossing the Barrier: A Scalable Simulator for Course of Fire Training". In Proceedings of the Interservice/Industry Training, Simulation and Education Conference – I/ITSEC.
Figure 1: Evaluating the MINT system with subject-matter experts.
