Evolution of a single modality

We first wanted to see whether the use of a single modality changed over time, and whether it did so in a similar manner across all of the various conditions. If this was the case, it would suggest that the use of that modality was not dependent on other modalities.

Each graph in this section (Figures 5-8) shows the change in interaction over time for a single modality. Each curve in the graph represents one experiment condition (labeled according to the P1 condition in the legend) over the full 40 minutes of interaction with the Archivus system. Only conditions in which the modality in question was used in both phases of the experiment are considered in the graph. The x-axis shows the 5 minute intervals into which the 40 minutes were divided, with a vertical line marking the point at which P2 begins. The y-axis indicates how much (as a percentage of all interactions) the modality being examined was used.

Figure 5, below, shows the change over time of voice use over the 40 minutes of the experiment, broken down into 5 minute intervals. In the graph, we see the general trend

Figure 5: Change in voice use over time

for voice use to decrease over time, which contradicts results found by Rudnicky [27]

which showed that voice was a preferred input modality even during extended use with a system. This difference is results might be attributed to the nature of the tasks being undertaken in the two studies.

We can also see that the V and VK conditions experience a rather sharp decrease in voice use while in the other conditions the decreases are much more gradual. The steep

decreases in the V and VK conditions are not surprising however, since they occur at the shift between phases where the user was also given access to a pointing device. What is interesting to note is the difference in behaviour between the MV and PV conditions after the first five minutes of interaction in P2. In the MV condition we see that there is a slight rise in voice use in the first five minutes which later decreases quite sharply, while in the PV condition there is a decrease in voice use, which then rises quite sharply, only to decrease again. In fact, we had expected the opposite to be the case due to the preference for keyboard interaction when the traditional MK paradigm is present (as shown in section 6.4) and the overall preference for voice interaction, but in particular in combination with the pen.

Keyboard use also clearly decreases over the 40 minutes in the conditions where it was present in both phases - MK, PK, VK, MVK and PVK. Figure 6, shows the change over time of keyboard use across the 40 minutes, broken down into 5 minute intervals.

Figure 6: Change in keyboard use over time

However keyboard use tends to fluctuate quite a lot in the MVK condition, so it is not clear whether it would continue to drop off if the experiment had been carried on for a longer period of time. The other interesting point to note here is that keyboard use drops off much more steeply in P2 in those conditions where voice is introduced, which is not

surprising since voice has already been shown to be the stronger/preferred of the two language modalities.

Contrary to the general trend of language use decreasing over time, Figures 7 and 8 indicate that pointing use increases over time. Looking at Figure 7, which shows the change over time of mouse use across the 40 minutes, broken down into 5 minute intervals, we can see that the nature of the rise is independent of the conditions. In the MV and MVK conditions the rise is gradual and quite steady, but in the two conditions where voice is introduced in the second phase (M and MK), we see a similar amount of drop in mouse use, and then a fairly similar pattern in the subsequent rise in mouse use during P2, which differs from the rise in mouse use found in the conditions that had voice use throughout the 40 minutes (MV and MVK). This is likely due to the limited effect that the introduction of the keyboard as an additional modality has on interaction.

Figure 7: Change in mouse use over time

Figure 8 shows the change over time of pen use across the 40 minutes, broken down into 5 minute intervals. Here we see a very similar pattern in change in pen use for the conditions in which voice was added during the second phase (P and PK) as we saw for change in voice use for M and MK. However, we can also see that overall there is much less pen use in the conditions that have voice access throughout (PV and PVK) than there was mouse use in the parallel mouse-based conditions. Furthermore, in those conditions

the rise in pen use only becomes marked in the second phase of the experiment, while its use was quite steady in the first phase. This result is surprising because there are no additional modalities added in the PVK condition, and in the PV condition, only the keyboard is added, which has been shown to have little impact on proportions of modality use in general, and when combined with pen in particular (section 6.4).

Figure 8: Change in pen use over time