Influence of user background

Finally, one of the factors that we felt was important to look at was whether the users’

personal background played a role in their overall performance with the system. In particular, we wanted to look at whether there were differences between native and non-native speakers of English, and between male and female participants, and whether experience with speech recognition systems and the amount of time that users spent with a computer on a daily basis had an influence on their success in finding answers. To do this we looked at the proportions that each of these factors held in both the top 20 and bottom 20 individual users for both the success score and the number of questions answered.

Native vs. non-native English speakers

Out of the 80 users in the experiment, only 12 (15%) were native English speakers. Out of those 12 users, 9 (75%) had top 20 success scores, and 8

(67%) had a top 20 score for the number of answered questions.

Moreover, only 1 of the native English speakers had a bottom 20 success score, and the same person was also in the bottom 20 for the number of questions answered. This suggests that native English speakers have an advantage over non-native speakers when using the Archivus system.

Unfortunately, we could not control for the distribution of native English speakers in the data set, so while some conditions have 2 native speakers (M, P, V, VK, and PVK), others have only 1 (MV and PV) and some have none at all (MK, MVK and PK).

Male vs. female performance

Out of the 80 users in the experiment, 33 (41%) were female. Only 6 of them (18%) had top 20 success scores, as opposed to 14 (30%) out of the 47 males, whereas 10 (30%) had success scores in the bottom 20, as opposed to 10 (21%) out of the 47 males. Of the top 20 question answerers, only 5 were female (15%) as opposed to the 15 males (32% of the male population), and from the bottom 20, 9 (27%) were female as opposed to the 11 males (23% of the male population). These figures suggest that women do a bit worse than men when using the system.

However, since the female users were distributed fairly evenly among the different conditions – most conditions (M, P, MK, PV, MV and PVK) had 3 females each, one condition (MVK) had only 2 females, and three conditions (V, VK and PK) had 4 females – we don’t think that this had a significant impact on the results⁷.

Previous experience with speech recognition systems

32 out of our 80 users (40%) had controlled a computer with their voice before. Out of those, 9 (28%) had a top 20 success score, and 8 (25%) had a top 20 questions answered score, while 7 (22%) had a bottom 20 success score and 7 (22%) had a bottom 20 questions answered score. The differences between these numbers are not large enough to conclude that previous experience with speech recognition systems played a role in user performance.

7 Note: we did not have data as to the educational background and professional experience of the users, so a more thorough analysis was not possible.

Amount of time spent daily with a computer

Out of our 80 users, 28 (35%) spent more than 7 hours per day using a computer. From those, 9 (32%) had success scores in the top 20, and 8 (29%) had scores in the top 20 for the number of questions answered. But, 7 (25%) also had success scores in the bottom 20, and 8 (29%) were in the bottom 20 for the number of questions answered. Conversely, 33 of the users (41%) spent less than 4 hours per day with a computer. From those, 7 (21%) had success scores in the top 20, and 6 (18%) were in the top 20 for the number of questions answered, while 9 (27%) scored in the bottom 20, and 6 (18%) in the bottom 20 for the number of questions answered.

While there is some variation in the number of users for each case, the differences are not large enough to say with certainty that experience with a computer was a significant factor in how successful users were with Archivus.

Overall, we can conclude that while native English speakers did have an advantage during the experiments none of the other factors played a role in the performance of the individuals involved in using the system. This is shown by the relatively low percentages, which indicate that the users who did place in the top and bottom 20 did not constitute a significant proportion of the population in question.

6.8.6 Conclusions

In order to consider efficiency in a holistic sense, we need to take into account all 3 factors for task completion – the number of questions answered, the success score and the correctness score. Given that in all of the conditions users had access to all of the modalities in the second phase of the experiment it is difficult to determine which particular modality combinations would have been most successful if additional modalities had not been added. Based on performance in phase one, we can hypothesize that MK, P, V and VK would have been the most effective, since each appears in the top 4 for at least two of the three factors.

We can also use this data to hypothesize about how the modalities and the system should be introduced to the user by looking at the overall results for the period of time including both phases. These results show that both the MK and MV conditions have high scores and high numbers of questions answered, and their correctness scores are 2.56 and 2.54 respectively, while V has a high correctness score and a high success score. This makes these conditions the most effective to learn to interact with Archivus in order to achieve the best results in the long term. In this conclusion we can also discount the influence that

the presence of native English speakers has since each of the conditions had 0 (MK), 1 (MV) or 2 (V) native English speakers.

Finally, we found MVK and PVK to be the least effective modality combinations. They consistently placed towards the lower figures in all three factors in both phases, and since there were no modalities added over time, we can conclude that providing all three of the modalities at once is not very helpful to the user. An interesting question that poses itself here is whether the problem lies in learning with all three modalities or the general availability of all three modalities. If it is a question of the availability of all three modalities, then we can draw the further conclusion that in fact, the MK and V conditions are the most useful overall for interacting with Archivus, since they are the only modalities that appear to be strong both in P1 and overall. We can also go further and claim that in this case, the MV condition which was also strong overall would be effective since in fact the performance in that condition appears to get better over time while data shows that the addition of keyboard has little impact on the interaction (keyboard is used only 5% of the time in phase two). However, in their work Le Bigot, Jamet, and Rouet [53] did not find that the use of voice lead to an increase in performance, which leads us to conclude that a more detailed analysis is necessary.