Future prospects

As demonstrated in this thesis, physiological signals can be used to assess emotions. However, there are still some research issues that have to be investigated in order to improve the accuracy of the assessment and apply it to concrete HCI applications.

Emotions can be elicited in several contexts influencing the emotional expression. It is thus important that the methods developed to assess emotions take into account those contextual elements. Since new human-computer interfaces will more and more involve all of the human senses it can be interesting to analyze the performance of emotion assessment algorithms according to the sense used for the emotion elicitation. Analyzing the combinations of such stimuli can also be of interest. The emotional model proposed by Ortony [57] is a possible direction to follow in order to better determine an emotional state according to the course of events that elicited the emotion. Emotion assessment from physiological signals (or from other sources) can then be added to this model to add personal emotional information. The mood of the user and the persistence of the precedent emotional state are also context related issues that can influence the elicitation of an emotion and thus should be taken into account.

All the experiments conducted in this thesis were done in an “ideal” environment where the participants were instructed to accomplish given tasks and to avoid movements. Switching from this type of experiment to the real environments gives rise to several issues. Physiological signals are very sensitive to movements, for instance if a user stands up his / her blood pressure will change. Moreover, the user can be disturbed by external events that are not related to the

application of interest. It is thus very important to develop algorithms that are able to detect signals changes that are not related to emotional processes.

While this work focused on the identification of emotional classes defined as areas of the valence-arousal space, going further toward the identification of a point in this space is of high interest to determine emotional states with higher precision. This could be useful to infer the intensity of the emotion, generally defined as the distance of the point to the center of the space.

Having enough resolution in this space is also mandatory to map a valence-arousal estimation to a given emotional label. Assessing the dominance (or control) dimension of emotions can be useful to well differentiate emotional states like fear and anger. Continuous estimations of emotional points in those spaces can be done by using supervised regression algorithm. In that case, since the valence and arousal variables seem to be dependent, the regression should be performed accordingly. Going forward to other continuous representations of emotions, like the SEC proposed in Scherer’s [10] model is still an opportunity but requires the evaluation of many continuous variables.

As demonstrated in this thesis, EEG signals can be used for emotion assessment. However, a lot of effort should be put on the design of new EEG caps that are less obtrusive to go toward applications. For parts of this thesis, 64 electrodes were employed to monitor brain activity. This high number of electrodes is problematic regarding prices aspects and leads to high dimensional feature spaces. Developing algorithms that are able to select the electrode positions of interest for emotion assessment is of major importance to solve those issues. A possible solution to this problem could be to use the MI computed between pairs of electrodes (as proposed in Chapter 3) to regroup electrodes that recorded similar information and choose one of them as the representative of the group.

Increasing the accuracy obtained from EEG features is of major importance for practical use of this device. For this purpose, new features should be investigated possibly inspired from the BCI community like features based on common spatial patterns. The MI feature set used with success in this study encourages the investigation of interactions between brain areas during emotional processes. The synchronization of brain processes could be used to determine new features. Some of the brain structures involved in emotional processes lie deep in the brain and it is thus difficult to assess their activity from surface EEG signals. Solving the inverse problem (i.e. finding the brain sources corresponding to a given EEG) to estimate deep sources and using this information as new features for emotion assessment could be promising.

Fusion with other sensors and sources of emotional information could lead to improvements of the emotion assessment accuracy. Several sensors can be used to acquire signals originating from the same sources. For instance, EEG measurements can be coupled with fNIRS measurements to

better estimate brain activity. However, since emotions are multi-modal processes that involve several component of the organism it is certainly most valuable to perform the fusion of different sources of information. For instance, this could be achieved by combining facial expression and speech identification with physiological measurements of emotions. Those fusions would necessitate more studies, especially concerning time aspects. The time resolution of different sensors is not the same and the different components involved in emotional processes do not have the same reaction time.

Most of the studies concerning emotion assessment from physiological signals (including this thesis) are done on emotional data that are not available to the whole research community. As a consequence it is difficult to compare the methods used for emotion assessment since their performances strongly depend on the protocol used for data acquisition. There is thus an important need for databases that are freely accessible. Such databases should ideally be multimodal, include contextual information and meet the constraints imposed by the law / ethical rules. A freely available multimodal database¹⁰ of emotionally driven brain and peripheral signals was constructed in collaboration with partners of the European Network of excellence Similar.

Unfortunately, this data was not analyzed in this thesis but we strongly encourage the use of this database for further research on the topic. A similar effort is now underway in the context of the EU project Petamedia.

Taken together, the above suggestions should lead to the development of a robust emotion assessment system. Once such a system is developed the next step would be to determine how the machine should adapt to the user’s emotional state. Some propositions were given in Chapter 7 for computer games but this strategy is highly dependent on the gaming application. Finally, the investigation of how the user perceives the complete system (i.e. emotion assessment and adaptation) is mandatory to control how it would be received by the general public.

10 available at http://enterface.tel.fer.hr/index.php?frame=results (retrieved on 19 May 2009)