8.1 Achievements
A series of 21 user-based Participatory Design (PD) events have been successfully conducted involving teachers and students from nine countries across Europe in Go-Lab Year 2. Among these events, the Go-Lab Summer School 2014 was most extensive where three formal studies with a large and diverse group of teachers were conducted, as well as deep involvement with the teachers engaged in Go-Lab tasks throughout the week which resulted in a wide range of observational data, interview data and ethnographic style immersion and insights.
Furthermore, the HCI research team of WP3 have also carried out several heuristic evaluations, feeding relevant input on usability and user experience issues to other work packages, also covering system functionality, robustness, and other qualities. The usability findings from heuristic evaluations seem to have correlated closely with usability findings from teacher evaluations;
however the teacher evaluations have provided more insight into contextual and practical matters and also perceived usefulness.
Additionally, a core group of teachers has been established to provide rapid online feedback on specific Go-Lab-related questions. This proves to be a highly effective method for rapid
engagement.
All these PD activities are valuable sources of feedback on a wide range of Go-Lab system components, some based on non-interactive mock-ups and others on working interactive prototypes.
The findings have been communicated to the pedagogical and technical teams on an ongoing basis, and many have already been addressed in their respective design and development work.
8.2 Challenges
Clearly, there have been different challenges in implementing the PD events. The most critical one was the recruitment of teachers, students and schools. The recruitment process has been more successful in some countries but more challenging in others, due to cultural and political
differences, varied professional styles, and the workload and level of work-related stress
experienced by teachers in some countries. Even when a session is organised and agreed, in field studies there are many possibilities for unplanned and uncontrollable factors (e.g. attendance, timetabling issues, distractions, technology issues) to influence the outcomes. Good planning is beneficial but researchers still sometimes have to improvise.
Within Go-Lab, like many RTD projects involving public users, there is a concern for the reputation of the product, and for it to be shown in its best possible light, partly to create a good reputation for the product and partly so that teachers who have engaged in participatory design sessions would find them useful and would be willing to continue to be involved. This means that there are sometimes compromises and trade-offs to be made when considering asking teachers to help evaluate software in their classes, especially in this early phase of development when the system is – quite understandably – not yet market ready. The Heuristic Evaluations and the use of the Core Teacher Group have helped us considerably in meeting these challenges.
Go-Lab 317601 Page 77 of 130
8.3 Limitations
As has already been mentioned, recruitment of schools and particularly getting access to live classes with students has been limited in some countries, and we have therefore not been able to be as purposeful and systematic about when and where we conduct studies as we might have liked.
As a result, some of the system components – especially the more recently created ones – have been evaluated by a relative small number of participants, perhaps not including every perspective to be found in the target domain.
Feedback received from participants has sometimes been difficult to interpret, especially when they have not been responding in their native language, or when the material has been translated. Also in some cases different participants have very different usability needs, preferences and ideas so elements of feedback can contradict each other.
The workflow defined at Section 2.5.4 has been most effective when partners have been able to negotiate a PD session well in advance and give the WP lead partner plenty of notice to prepare or adapt a protocol. However, occasionally PD opportunities occur ad hoc, and protocols and materials have been prepared rather quickly and less optimally. We will therefore endeavour to prepare a toolbox of more flexible materials suitable for a wider variety of PD sessions with students of different ages, sessions of varied durations etc.
8.4 Outlook
In Year 3 and Phase B of Go-Lab, WP3 will be more ambitious to involve an even larger number of teachers and students to evaluate more mature interactive Go-Lab system components. With the ongoing support from WP7 (Pilot), we are optimistic that this rather challenging target number can be reached. Figure 8.1 illustrates that as the software becomes more functional and usable, and the in-house ILSs become more relevant to school curricula, teachers may find it more attractive, effective and productive to conduct a lesson in this way, and provide PD data. As the engaged teacher community grows we expect to have a reducing need for Heuristic Evaluations, but active involvement of Core Teacher Group in providing prompt feedback will continue.
Figure 8.1. Changing communities of engagement
Go-Lab 317601 Page 78 of 130 Furthermore, with prototypes of higher fidelity, more performance-based data, in addition to the perception-based, can be collected. One possible source of metrics which we will explore is the use of system-based measurements using web-analytic types of tools. This would require significant support from the development team, From the WP3 perspective, one intriguing research question is to understand the downstream utility of evaluation feedback to be collected with different
approaches and instruments. In other words, we aim to assess the actual impact of such feedback on the design and development work by tracking how users’ perceptions, attitudes, behaviours, performances, acceptance, and adoption will vary with the newer versions of the prototypes, which are redesigned based on the issues identified and associated improvement suggestions.
It is anticipated that WP3 will have even closer collaborations with WP1, WP4 and WP5 in the coming year, in order to prioritise studies and communicate findings even more effectively.
Go-Lab 317601 Page 79 of 130