The evaluation of the exploratory search task design protocol 83

In the evaluation of the protocol exposed in Table 5.1, we want to verify if users can design exploratory search tasks with the given indications. We want to evaluate the following hypothesis:

H1: with our protocol for the design of exploratory search tasks, the users design effectively exploratory search tasks.

5.3.5.1 Method

We selected twenty participants, all of them are computer scientist and did not know anything about exploratory search before the test. Ten participants have to design exploratory search tasks for Discovery Hub, and the ten others for 3cixty. Indeed, the elaboration of exploratory search tasks takes place in the context of an evaluation of a specific exploratory search system, and more precisely in the evaluation phase of a functional prototype or system. We wanted to test our protocol with different exploratory search systems.

The evaluation follows this procedure:

1. Presentation of the test’s goal

2. Presentation of the system (Discovery Hub or 3cixty) with discovery scenarios exposed in Table 3.7 in Chapter 3.

3. Elaboration of two exploratory search tasks for one of the two systems by using the protocol exposed in Table 5.1

4. End of the test and acknowledgment

5.3.5.2 Results

As mentioned previously, the designer of the protocol evaluates the participants’

tasks with the list of exploratory search tasks’ characteristics exposed at the end of Section 5.2. Indeed for each task we verify if it checks all the characteristics.

In Table 5.2 we give an example of the analysis with one of the participants’ tasks exposed in Table 5.3. For all the participants, 100% of the tasks which where designed with our protocol are effectively exploratory search tasks. We also noticed that the tasks were designed without notable difficulties.

Then the hypothesis H1 is validated: our protocol for the design of exploratory search task is validated.

5.3.5.3 Conclusion and perspectives

The evaluation shows that the protocol we designed allows the elaboration of exploratory search tasks. In this protocol we propose to design a non customizable task. Depending on the system’s database, the users of the method (i.e. the designers of the systems or evaluators) can also design with our protocol for the design of exploratory search taskstemplates of taskswhich can be customized in the last minute with the participants. This solution can imply a more important engagement from the participant because it may suit its personal interest, and excite her curiosity.

For example, Discovery Hub offers a large range of possible subject to explore. The task "you heard on the radio a song of XX, you go on Discovery Hub in order to know more about her/him" can suit the participant’s musical taste. The same task with an imposed artist would not be as attractive to the participant. Further evaluations are necessary to find out whether our protocol allows the design of such customizable template tasks.

5.4 A protocol for the model-based video analysis

In Section 5.2, we present a procedure of customizable users tests. We present in Section 5.3 a way to design suitable and ecological exploratory search task with a given protocol.

In this section we present a video analysis grid for the evaluation of recorded exploratory search sessions. It corresponds to step 3 of the procedure presented in Section 5.2: the records analysis. In this evaluation step, the evaluators analyze records of exploratory search session: screencasts and users’ comments (they explain what they do and why). As the heuristics of exploratory search, our video analysis grid is based on our model of exploratory search process presented in Section 3.

Indeed, the evaluators identify in the video records and users’ comments which model’s feature the user is doing. The video analysis grid we propose aims to facilitate this identification.

With such an analysis, the evaluators see if the users achieve all the features without difficulties, and if it is not the case, they can identify the source of the issues (e.g.

missing features, usability issues. . . ).

Our task design criteria Our evaluation of the task:

"you planned a trip in Corsica with a limited budget, you go on 3cixty in order to plan and organize this trip"

The task does not provide a clear crite-ria on:

• how to begin the search

• when to end the search: she does not specify limits in terms of time or number of results.

This is indeed the case: the task is not limited to the hostels or places of inter-est for example. Furthermore, it does not indicate any limit in term of time or number of results.

The task suggests a knowledge acquisi-tion, comparison, or discovery task

This is indeed the case: the task sug-gests learning, comparing and discover-ing activities. Indeed, planndiscover-ing a trip may imply to discover the places of in-terest, compare different hostels,etc.

The task provides a low level of speci-ficity about:

• the information necessary for their search

• how to find the required informa-tion

• how to recognize the required in-formation

The task provides effectively a low level of specificity about required informa-tion. The only indication which will be used as a framework of the search is the limited budget. Apart from that, the task does not specify what kind of infor-mation/results the user has to search.

The task must be ecological:

• a situation which the test persons can relate to and in which they can identify themselves

• a situation that the test persons find topically interesting

• a situation that provides enough imaginative context in order for the test persons to be able to re-late and apply the situation

The task gives a situation providing ef-fectively an imaginative context. The user can assume ownership of the task and the way they want to explore the topic. The situation of planning a trip is a common situation and generally in-teresting for every body. For all these reasons, we can say that the task is eco-logical.

Tab. 5.2: The evaluation with our task design criteria of the task proposed by Participant 1 on 3cixty.

Discovery Hub 3cixty Participant 1: You organize a birthday

party and you go on Discovery Hub to prepare a playlist

Participant 1:You planned a trip in Cor-sica with a limited budget, you go on 3cixty in order to plan and organize this trip

Participant 2: You want to establish a list of movies in order to prepare a movie night with friends and you go on Discov-ery Hub

Participant 8: You are going to make a business travel in Paris. You want to occupy your freetime between two meet-ings and you go on 3cixty to explore the museums’ exhibitions in that period

Participant 6: You are interested in the field of Psychology and you go on Discov-ery Hub in order to compare, discover the differences and similitudes between the subdomains (cognitive psychology, clini-cal psychology. . .)

Participant 11: You planned a trip in Barcelona and you want to know more about the city (customs, traditions. . . )

Tab. 5.3: Examples of tasks designed by using our protocol for the elaboration of exploratory search tasks

The element of method we propose here is a video analysis grid that helps the user in the identification of the model’s features in a user’s exploratory search session and comments. Such an analysis highlights the difficulties encountered by the participants and gives clues of improvements.

We first explain how the grid was designed and, in a second time, present the first results of the evaluation of the protocol.