HAL Id: hal-00190052
https://telearn.archives-ouvertes.fr/hal-00190052
Submitted on 23 Nov 2007
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Learning a New Game: Usability, Gender and Education
Hatice Sancar, Turkan Karakus, Kursat Cagiltay
To cite this version:
Hatice Sancar, Turkan Karakus, Kursat Cagiltay. Learning a New Game: Usability, Gender and Education. Young researchers furthering development of TEL research in Central and Eastern Europe, 2007, Sofia, Bulgaria. �hal-00190052�
LEARNING A NEW GAME: USABILITY, GENDER AND EDUCATION HATICE SANCAR
TURKAN KARAKUS KURSAT CAGILTAY
Department of Computer Education and Instructional Technology Middle East Technical University
Ankara Turkey
[email protected], [email protected], [email protected]
Abstract: This research study aims to investigate the usability issues of a computer game; regarding how people learn a new game. Because the user characteristic and the strategies they use while learning a new tool are important factors for usability, they were also taken into account in the study.
Both qualitative and quantitative methods were used in the study. 16 participants played the computer game. They were also observed and their eye-movements were recorded by an eye tracking device.
Results showed that participants learned a new game by trial and error and the insufficient instruction caused their placing numbers in wrong places as well as not realizing important tools that can help them to complete game.
Keywords: Computer games, usability, eye tracking.
1 INTRODUCTION
Computer games are one of the most popular media in today’s world. Each year, billions of hours are spent to play computer games by people around the world (Von Ahn, 2006; Brand, Knight & Majewski, 2003). Unique characteristics of the games that attract people are “rules”, “goals”, “objectives”, “outcome and feedback”, “conflict”, “competition, challenge, and opposition” and “interaction” (Prensky, 2001). According to Hogle (1996), the games including these characteristics are activities which the participants must use physical and mental skills in order to access a goal by following specific rules. All these issues are connected with the interaction between users and computer games and Hill, Wiley, Nelson and Han (2003) state that usability which is connected learner content interaction facilitates learning a course. Similar to other kinds of software, the interface of the games should be an efficient and effective means for the user to interact with the program (Federoff, 2002).
Although computer games are most promising tools that can meet the peoples’ need stemmed from the new digital technologies and media, the usability problem which comes from the interface design and rules of the game can prevent the effectiveness of the computer games. Effectiveness, efficiency and satisfaction are emphasized as the major aspects of games and these issues can be provided with fun elements in the games because users cares having a good time by playing (İnal, et. al., 2005). As Von Ahn (2006) states humans require some incentive to become part of a collective computation and online games are good incentives for encouraging people to participate in the process. However, it is important to know that fun elements are related to the ease of a game use including controls and interface (Barendregt et. al., 2003). According to Clanton (1998) game interface, game mechanics, and
game play are different usability issues of games. Actually, all these aspects are related to the
“playability” which includes “quality of the games” and “style of interaction”.
The interface gives chance to live or die for a game. If there are many usability problems and users cannot tackle with them, the second chance may not exist. The usability of computer games should be studied because of these reasons. Most usability studies involve measures like time to complete a task, percentage of participants succeeding, type and number of errors, and subjective ratings of ease of use (Nielsen & Mack, 1994) and these methods are based on human performance, in other words, observable behaviors. For that reason, some types of questions remain difficult to be answered by using these techniques. Of course, researchers can observe the users spending times to complete a given task or errors that they make. However, which particular parts of the interface or playability aspects extend the periods and enhance the errors remain unclear and the researcher may not understand what went wrong. Not knowing the answers of these questions may cause design by trial and error, and increase development time and cost (Keith et al, 1998).
Eye tracing method which records the movements of participants' eyes during task performance may offer detailed information that may help researcher answer usability questions and reduce trial and error in user interaction design. It is believed that using traditional and eye tracing methods as one of the new methods provide researcher to understand deeply the results of the research study and enable them give clues designers to design better. There are not many studies combine these methods for evaluating usability of a game. Therefore, this study aims to assess usability of a game in terms of “game interface”,
“game mechanics”, and “game play” by merging traditional usability and eye tracing methods. In addition, how users learn to play a new game is explored in this research.
The main and sub research questions of this study are:
* How do the users learn to play a new game?
- On which element(s) do the users spend much time?
- What types of errors do users make?
- On which task(s) do the users make errors?
- Is there any significant performance difference between girls and boys?
- Does education level make an impact on performance?
2. METHODOLOGY
Both qualitative and quantitative approaches were used to answer the research questions. The users were observed while playing games and wanted them to verbalize their thought processes in a "think aloud" protocol during the tasks. In addition, semi-structured interviews which include questions about the users opinion on the game were also conducted.
Eye tracing method used to get collect quantitative data to understand how much time they spent to complete the computer game. In addition, which element(s) the users make eye- fixations, how much time they spend on them and what types of errors they made were analyzed by using the eye tracking methods.
SUBJECTS AND PROCEDURE OF THE STUDY
Totally, 16 participants participated in this study. 8 of participants were undergraduate students (3 males, 5 females) and 8 of them were PhD students (3 males, 5 females). Ages of the participants ranged from 21 to 26 years old. All subjects played the game by using an eye
tracking machine which is located in the Human Computer Interaction laboratory. Each playing session lasted about 6-7 minutes. All of them participated to the study voluntarily and reported that their English was sufficient to play the game.
First step of the study was to choose a computer game which is not known by the participants. Several games were selected by taking expert and peer opinions. To make the appropriate selection, a pilot study was conducted. Some games were eliminated because they were so simple; some were eliminated because they were complicated. Finally, it was decided that “Addemup”, a game from the “BlueBug” (www.bluebuggames.com) website was the most appropriate one. All of the participants were asked whether they knew the game or not before playing it and all of them reported that they did not know the game.
Second step was to determine which procedure should have been followed to answer research question best. It was decided that instruction was not given to participants about the game and they were observed while playing the game. Participants were asked to use “think aloud protocol” to get information about their activities. Moreover, the sessions were videotaped and notes were taken.
In order to collect information about usability issues, eye tracking method was used and how much time they concentrated at which part of the game was also coded. Tobii 1750 eye tracker was used in the study and this equipment recorded the eye movements of the participants while they were playing the game.
Third step was to conduct semi-structured interviews. After the participants had played the game, a semi-structured interview was conducted to collect in depth information about learning process of the game. In addition, the participants’ actions were videotaped to add information which might not be gotten with the interview as well field notes.
COMPUTER GAME- ADDEMUP
The “Addemup” game which is an electronic computer game on the BlueBug web site aims to make players practice on addition. It consists of two parts, one of them is classical part which includes one level and has tools on it to help the user complete the game. The other one is a puzzle part which has five levels but has not any specific tools on it. The mechanism used in the two parts is the same, obtaining the number given in the right side by adding the number in the game area. If the total is equal the given number, the numbers disappear. The aim is to clear all numbers from the whole game area. We wanted the users play the classical part because this part has tools and move limits (Figure 1.). If the user cannot the clear the board in 50 moves, s/he looses the game. To make correct inferences about the how users learn to play a new game, the participants should have been observed while playing the same part of the game. For those reasons, the classical part was preferred.
FIGURE 1. THE SCREENSHOT FROM THE CLASSICAL PART OF THE ADDEMUP
DATA COLLECTION AND ANALYSIS
Three sets of data were collected in this study. The first data set consisted of eye tracking data produced by Clearview System on the Tobii eye tracker. The second set of data obtained during semi-structured interviews after each test session. And the third set acquired through observations. Both qualitative and quantitative approaches were used in the study.
Interview and observation results composed of the qualitative part. In addition, the fixations on the screen were used to support the findings of the interviews and observation results. The quantitative part included the dependent variables, the coordination of eye fixations on screen, duration of eye fixations and the time completion the games. These data were recorded to SPSS and analyzed.
The first, second and the last sub questions were answered by using quantitative approach. Eye tracing helped the researcher to get data for the quantitative analysis and the
“SPSS” used to analyze that data. Actually, eye tracking method was used to interpret qualitative data for supporting the results of the observation and interview in addition to quantitative data.
The participants were observed while playing games. Each participant was recorded and notes were taken in addition to records. They played the game on the eye tracking machine in separate times and individual valuation was taken in to account. While the users were playing the game, the researchers observed how they learn to play the new game and where they got difficulty. In addition, to get detailed information, "think aloud" protocol was used. In other words, they were asked to verbalize their ideas while playing games.
3. RESULTS
The results of the study are explained according to the gathered data. These are Demographics of subjects, Eye Tracking Results and Usability of the computer game.
DEMOGRAPHICS OF PARTICIPANTS
The demographics data were collected to see whether there was any other factor affect the results of the study. Participants who have similar characteristics were selected for the study. All the participants were picked from the same department. 10 participants were female, 6 of them were male. Their ages ranged from 21 to 27 and mean of the participants’
age was 24.6.
In addition to demographic characteristic of the participants, the information on their computer usage and computer game playing was collected. All of the participants reported that they had computers at home and at their offices in PhD students group. Also, all PhD participants had the Internet access at home. 3 PhD students (2 males and 1 female) said that they played games rarely. While 5 female and 1 male PhD reported that they preferred to play puzzle, cards and shodoku type games, 2 male PhD students reported that they prefer sports, adventure and action/strategy type games. In undergraduate student group, all participants reported that they had opportunity to access computer in their schools and dormitory. Also, they had regular Internet access. While two participants (1 female and 1 male) in undergraduate students said that they did not play game, others pointed that they played games ranging from 2 hours to 10 hours. While four female students reported that they preferred to play puzzle or cards type games, two males reported that they preferred to play adventure, fighting or sport type games.
EYE TRACKING RESULTS
The game does not have any time limitations. However, there is move limitation. The player has to achieve the goal within 50 moves. It was also interesting that none of the participants could complete the game. Despite these facts, it was important to get information on how much time they spend while playing the game and on which parts their eyes fixed. In this part fixation numbers, total duration analyzed by regarding gender and addition to these two dependent variables, fixation counts, gaze time and first fixation time for each Area Of Interests (AOI) were analyzed in terms of level of education (undergraduate and PhD).
Fixation number refers to number of points which eye focused in a unit time. Totally all participants’ fixation number is M=656.31 with SD=337.51, and total duration they spent on game is M=312.1 second with SD=124.9. Gaze time is the total time of all fixations in the respective AOIs. We described four areas of interests in this game as “upper menu”, “gaming area”, “number area” and “help menu”. For upper menu gaze time is M=3.43 second with SD=6.70, for gaming area gaze time is M=135.23 second with SD=82.43, for number area gaze time is M=14.12 second with SD=10.40 and for help menu gaze time is M=5.34 second with SD=2.70.
Means and standard deviations of fixation numbers and gaze time durations are shown in Table 1. PhD students have more fixation numbers and durations for area of interests however standard deviations for this group is very high. Fixation numbers for undergraduate group is M=640.25 second with SD=264.11. Total duration for undergraduates is M=301.77 second with SD=98.26 and for PhD’s it is M=322.40 second with SD=153.56.
TABLE 1. FIXATION NUMBERS, TOTAL DURATION AND GAZE TIME OF AOI’S IN TERMS OF EDUCATIONAL LEVEL
Undergraduates PhD Students
Mean SD Mean SD
Fixation Number 640.25 264.11 672.37 416.84
Total Duration* 301.77 98.26 322.40 153.56
Upper Area Gaze Time* 4.51 9.26 2.35 2.78
Gaming Area Gaze Time* 126.80 67.29 143.66 99.35
Numbers Area Gaze Time* 14.28 12.02 13.96 9.34
Help Menu Area Gaze Time* 4.43 3.42 6.24 1.58
* values are “second”
Comparison of means for males and females as shown Table 2 implies that males spend more time to learn games than females. Males’ fixation number is M=804.83 with SD=422.89 and females’ fixation number is M=567.20 with SD=258.79. Total duration which is parallel to fixation number is also higher for males. Total duration is M=368.65 s with SD=128.40 for males and M=278.15 with SD=116.05 for females.
TABLE 2. FIXATION NUMBERS, TOTAL DURATION AND GAZE TIME OF AOI’S IN TERMS OF GENDER EDUCATIONAL LEVEL
Males Females
Mean SD Mean SD
Fixation Number 804.83 422.89 567.20 258.79
Total Duration* 368.65 128.40 278.15 116.05
Upper Area Gaze Time* 2.08 3.13 4.23 8.20
Gaming Area Gaze Time* 162.62 114.86 118.80 57.64 Numbers Area Gaze Time* 16.32 11.245 12.80 10.24
Help Menu Area Gaze Time* 6.81 34.1 4.46 1.81
* values are “second”
Although descriptive data shows some differences, ANOVA results showed no significant differences for gender and educational levels for fixation numbers and total durations (Table 3). Gender has no significant effect on fixation number with F(1,14)=1.98 p=.18, and has no significant effect on total duration with F(1,14)=2.11 p=.17. Educational level also has no effect on fixation number with F(1,14)=.034, p=.85 and has no effect on total duration with F(1,14)=.102, p=.75. Interaction with gender and educational level also has no effect on fixation number with F(3,12)=1.05 p=.40 and total duration with F(3,12)=.67 p=.58.
TABLE 3. UNIVARIATE ANALYSES OF VARIANCE F VALUES FOR GENDER AND EDUCATIONAL LEVEL EFFECTS FOR FIXATION NUMBER AND TOTAL
DURATION
ANOVA Fixation Number Total Duration
F F
Gender 1.98 2.11
Educational level .034 .102
Gender * Educational L. 1.05 .67
*p<.05
Some comparisons with ANOVA were conducted to reveal whether there are differences between gaze times in areas of interests. For upper menu there is no significant difference between undergraduates and PhD students with F(1,14)=.40 p=.53. For other AOI’s also there is no significant results, for gaming area F(1,14)=.15 p=.69, for numbers area F(1,4)=.004, p=.95 and for help menu F(1,14)=1.90 p=.19.
There is no significant difference between the users who play the game frequently and occasionally in terms of the amount of the time they play the game. Actually, it was an unexpected result. Maybe, it was stemmed from the types of games they played because only 2 participants reported that they played mind games.
QUALITATIVE RESULTS CONTRIBUTING USABILITY OF THE GAME
It should be emphasized that 4 of the 8 PhD participants firstly entered the “How to Play” part of the game. On the other hand, 3 undergraduate participants entered the “How to Play” parts before playing the game. The students said that they had entered to “How to Play”
part of the game to get an information about the playing rules of the game. The graduate participants who did not enter “How to play” part reported that they preferred to learn playing a new game by trial and error approach. One of them said: “I know that he should enter the firstly “How to play part” but I do not like to do that. I prefer the game while playing”.
“How to play parts” of the Addemup game consists of two parts. In the former part, the first rule of the game is shown in detail (Figure 2.). In the latter part, the second rule of the
game, tool usage and reminder notes for the help usage (Figure 3.). If the participants started to play directly, again an instruction part about how the game can be played is emerged. This part is the same with the “How to Play” part. 4 PhD and 3 undergraduate participants who entered the “How to play” part said that the instruction which is given at the beginning of the game was not necessary. All participants pointed that they did not use tools which can be used to eliminate numbers because they did not realize their existence. When the researchers asked that the tools and their functions were explained in the second part of the instruction, all of them said that they did not realize them. All participants reported that they understood the logic of the game when they read the game but there were three instructions about the game in the second part of the instruction; for that reason, they did not pay attention this part and they did not realize tools and second rule of the game. All these factors caused the participants did not play easily as they guessed before playing game. Actually, the mean of the time they spent in the “How to Play” part is 71.38 seconds. And the mean of the rest time they spent to play game is 253.89 seconds minutes. In addition, they realized that fact coincidentally. On the other hand, the rules of the game were realized by the method of trial and error.
FIGURE 2. THE SCREENSHOT FROM ADDEMUP GAME FOR THE SECOND RULE.
FIGURE 3. THE SCREENSHOT FROM ADDEMUP GAME FOR THE FIRST RULE.
14 participants out of the 16 participants tried to place the number which is given to find its aggregation on the number in the game area instead of blank parts.
“Why I can’t the replace the number? What happened? I could not understand” said one of the participants who tried to replace the wanted number on the number in the game area. All the other participants said the similar things.
The participants could understand the first rule of the game. In other words, they realized that the total of the number which is given in the tile would be found. However, they confused about how to replace the numbers. They found the second rule of the game by the trial and error method and interestingly, none of them used tools.
“Oh! Can the numbers be replaced at the empty area? But it is not showed in the instruction part.” said one participant who realized where she could replace the number.
It was interesting that there is also an extra explanation about the rules of the game in the “play” part. 3 of 7 participants thought the play started when they saw the instruction.
Actually, the participant who entered the “Play” part directly also thought that the game started. The participants’ comments were from these three different perspectives as below.
“What? Did not the game start? But it is not necessary, anyway there are “How to play part” said the one of the participant who entered the “How to Play” part and thought the instruction given as the game.
“What will I do now? What? It is instruction, but it is not necessary if I want to see instructions, I would enter the “How to Play” part.
After the game play started, all of the participants who reported they had understand the game but 3 of them (1 PhD student and two undergraduate students) got stuck because they did not realize that the game area has a space which covers 9 squares and the total of the numbers in that space should be equal to number wanted. However, these 3 participants could not understand how the numbers eliminated because they could not realize the area consisting of 9 squares. One participant said:
“But the space which covers 9 square is not clearly seen. It is very dim. I did not realize it”
14 of the participants did not use any of the tools which help the players when they got stuck. Only 3 of them realized the “Hint” tool but one of them did not use it although he realized the hint tool. He said that:
“I realized “Hint” button but I did not use it because I focused on game. Also, I did not know its function.
The other two participants said that Hint button is twinkle and go out a sound as well as it gets out of this sound when they did not found the total number of the number given in the tile. Actually, the “Hint” usage was not done consciously because “Hint” tool could be used 5 times but the participant did not realize it. Other tools were not used by the any of the participants. After the game they were showed the tools and asked why they had not use them.
One participant said:
“I concentrated on the game. I did not look any other place in the game. I fixed just numbers” pointed out one of the participant who did not use any tool.
“I used only “Hint” because it is green and twinkle. The other buttons are the same color with the background of the game area.” said the participant who used only the “Hint”
tool.
The all participants firstly looked for if there is a time limitation of the game. One of them realized that if they did not complete the game with 50 moves, the game is over. They were wanted to stop playing game when they understood the first and second rule of the game.
Participants spent nearly all of the playing time on the game area. They did not realize the tools or the move limitation for that reason. The mean of the time spent in the game area is 135.23 second.
DISCUSSION AND CONCLUSION
The results of the study showed that the players learn to play a new game with trial and error. For that reason, the games should be designed to reduce the errors made by players.
To giving the instruction about the rules and game playing before the game starts may reduce the errors. Also, the participants of the study had the similar characteristics in terms of the computer age and computer usage. There were no gender differences in terms of the time they spent to learn the game and total fixation duration. Also, PhD and undergraduate students did not show significant difference for these two dependent variables. Maybe, these results stemmed from the similarity in their background.
All participants tried to solve the puzzle on the game area at first instead of the empty space. 3 out of 16 participant did not realized that the total of the wanted number has a space of consisting of 9 squares and the total number in that space should have to be equal the wanted number. All these confusion stemmed from the insufficiency of the instruction part.
The placement of the wanted number should be showed clearly and the space which consists of 9 squares should be more emphasized. The simulation in the instruction part shows that the number in tile placed between the numbers which has blank squares. However, the first screen of the game has not any blank square. In other words, the platforms in the instruction and game are different than each other. For that reason, participants tried to place number in the tile on the numbers in the game area. In the games, platforms used in the instruction and help parts should be similar to the game platform at the beginning.
The participants mainly concentrated on the game area while playing “Addemup”
game and did not realize the tools in the game. The interface design of the game and insufficiency of the instruction in “Addemup” caused it. For example, all the tools’ having the same colors with the background of the game prevented the participant to realize tools of the game. Also, there is not any alarm to take participant attention to tools such as “sound”, “a color change on tools” and so on. To use different colors in the tools design from play area and sound effects may ease the participants’ realizing the tools. The instruction about the tool usage is not showed clearly in the “how to play” part, just there is an alarm as “If you get stuck, you free to use more moves.” The usage of tools should be showed with a simulation in detail in the games.
3 of the 16 participants realized the “Hint” tools and just two of them used it. All participants looked for time limitation in the game and none of them realized that the game is over if they did not finish the game in 50 movements because there was no explanation about this issue. All participants looked for a time limitation to complete the game during game playing. To prevent this kind of concern of the player an explanation about the time or move limitation of the game should be emphasized in the games.
As literature advocates computer games are the promising tools that can be help the instructors make their instruction effective because they attract the learners’ attention.
However, the usability problem can be a big problem in front of that aim. In order to keep the attractiveness of computer games, more researches are need in HCI field. According to Leikas et al. (2000) Human-Centered Design approach where the computer game is designed and evaluated with end-users in every step of the iterative design process should be used to create an immersive, captivating and highly usable game. Jorgensen (2004) claims that the potential of usability methods for computer games are currently not clear in development of games and computer games should be easy to learn, but difficult to master contrasts conventional usability evidence easy to learn and easy to master. The fundamental challenge for the player and the fundamental utility for the user can be achieved with this approach according to him.
As a result games should provide simulations in learning phase of games, because people who have computer fluency prefer to use trial and error while they learning a new game. This might cause lack of motivation for intelligence games especially children. Before passing playing area, an orientation part might be given as compulsory phase to prevent them challenge with trial and error process.
REFERENCES
1. Barendregt, W., Bekker, M.M. and Speerstra, M. (2003). Emprical Evaluation of Usability and Fun in Computer Games for Children. Proceedings the conference on Interact’03. Human Computer Interaction. Published by IOS Press. Pp 705-708.
2. Brand, J.E., Knight, S.J. and Majewski, J. (2003, 4th-6th November). The Diverse Worlds of Computer Games: A Content Analysis of Spaces, Populations, Styles and Narratives. Paper presented at the first Level Up Digital Games Research Conference, University of Utrecht, The Netherlands.
3. Clanton, C. (1998). An Interpreted Demonstration of Computer Game Design.
Proceedings of the conference on CHI 98 summary: human factors in computing systems: Chi 98, 1-2
1. 4. Federoff, M.A. (2002). Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in Video Games. Published Master Theses, Indiana University.
4. Hogle, J. G. (1996). Considering Games as a Cognitive Tools: In Search of Effective
“Edutainment”. ERIC ED 425 737.
5. Hill, J. R., Wiley, D., Nelson, L. M. & Han, S. (2003). Exploring research on internet- based learning: from infrastructure to interactions in David H. Jonassen (Ed.). Handbook of Research for Educational Communications and Technology. NewYork: Macmillan.
pp.433-460.
6. Inal, Y., Cağiltay, K. and Sancar, H. (2005). Factors Effecting on Game Preferences of Children. Proceedings of the Conference TBD, Bilişim’05.
7. Jorgensen, A. H. (2004). Marrying HCI/Usability and Computer Games: A Preliminary Look. ACM International Conference Proceeding Series. Proceedings of the third Nordic conference on Human-computer interaction; Vol. 82, Pp.393 – 396.
8. Keith S. K., Ellis, S. and Cornell J. (1999) The Hunt for Usability: Tracking Eye Movements. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI'99), Pittsburgh, PA.
9. Leikas, J., Väätänen, A., Räty V.P., (2000). Virtual Space Computer Games with a Floor Sensor Control Human Centred Approach in the Design Process. Lecture Notes In Computer Science. Proceedings of the First International Workshop on Haptic Human-Computer Interaction table of contents. Vol. 2058, pp.199-204
10. Nielsen, J., & Mack, R. L. (Eds.) (1994). Usability inspection methods. New York, NY:
John Wiley & Sons
11. Prensky, M. (2001). Digital Game Based Learning. McGraw-Hill Press. New York.
12. Von Ahn, L.( 2006). Games with a Purpose. Computer, 39 (6), pp. 92-94, (AN 21306024).
