Emotions Detection on an Ambient Intelligent System Using Wearable Devices
Angelo Costa1, Jaime A. Rincon2, Carlos Carrascosa2, Vicente Julian2, and Paulo Novais1
1 Centro ALGORITMI, Escola de Engenharia, Universidade do Minho, Guimarães acosta@di.uminho.pt, pjon@di.uminho.pt
2 Universitat Politècnica de València. D. Sistemas Informáticos y Computación jrincon@dsic.upv.es, carrasco@dsic.upv.es, vinglada@dsic.upv.es
Abstract. In this paper we present an Ambient Intelligent System, the iGenda, and the integration of a wearable device. The aim is to detect emotional states through the wearable device and ultimately represent and manage the social emotion of a group of entities. The advantage of this action is that its usability is in line with retirement homes and similar places, where the community is extended and an harmonious environment is imperative. The iGenda serves has the visual interface and the information centre, receiving the information from the wearable device and managing the community emotion by sending information to the care-receivers, caregivers, or changing home parameters (like music or lighting) to achieve an specific emotion (such as calm or excitement).
Thus the goal is to provide an affective system that directly interacts with humans by discreetly improving their lifestyle.
Keywords: Aging, Emotion, Simulation, Wearables
1 Introduction
Ambient Assisted Living (AAL) is an booming area where a large number of projects are being developed with the aim of providing assistance to elderly and disabled people [1,2,3]. What we can observe is that most of them have as a goal to be simple and provide the least interaction with the users as possible. But a common issue that they possess is that they rely solely on automatisms and static user profiles. While they are effective to a certain degree (simple tasks and basic likes) they do not encompass punctual changes that are part of the complex human states (like boredom), which forces changes to the profile [4].
These changes are usually associated to certain emotional states [4,5,6] that affect the human procedures, and asking every time the user for its consent to every action/decision defeats the purpose of being discrete and ubiquitous [6].
A solution would be the adoption of an adaptive system that is able to per- ceive these emotional changes and adjust to them. Furthermore, with sufficient information, emotional profiles could be created in order to mimic the user com- mon response and pre-emptively use this information to respond to decisions.
They way that humans perceive the world influences their emotional state and that has a repercussion on the physical level. While most can hide facial expressions and body movements (like hiding a state of surprise) there are low- level signals that the human body sends inadvertently like skin/muscle tension- ing, pupil dilatation and micro-movements. These most of these signals have a corresponding bio-electrical impulse, thus they can be captured by sensors.
We propose the usage of a wearable device in form of a wristband, named Emotional Smart Wristband(ESW) that feed information to the iGenda in order to schedule new tasks according the emotional status.
The paper is structure as follows: section 2 presents the ESW architecture, the emotional model and the virtual actor concept; section 3 presents the conclusion and future developments.
2 Emotional Smart Wristband
The ESW monitors the Galvanic Skin Response and the Photoplethysmogram, processes that information locally and sends that information to other systems like the iGenda [7,8,9]. The platform that supports the ESW ontology is able to receive the information of multiple ESW’s and process them individually or in group. The group emotion detection is very useful to determine the general emotion status of an environment that is composed of multiple people, like an nursing home. In the AAL concept the individual is the most important factor in the system, but that is applicable only when there is only one individual in a smart home; when we consider multiple individuals we find that the best way to maintain an harmonious ambient is when most of the people tend to an overall emotion (considering varying degrees).
For instance, it is well known that there is typically one or a few more lead- ers in elderly communities [10]. One issue with these leaders (specially in older adults) is that they establish the likability of certain activities and the rest of the group follows (even if they are internally in disagreement). This tends to result in a very biased set of activities that the leader enjoy and the rest tolerate. By using the ESW the users can respond unbiased of what their feelings are towards an activity. Tracking the community responses results in the overall likeness and in the social aspect identify if the leader is a good one or not.
2.1 Architecture
The ESW platform is composed of agents that calculate the group emotion (or Social Emotion) of the participants, through the Social Emotion Agent (SEtA) and resorting to machine learning techniques and makes it available for con- sumption [11,12]. The voltage outputted from the sensors is captured locally (in the wristband) and pre-processed there, thus the information fed to the agents is already a high-level one. Additionally, the multi-agent system has dedicated agents that grab that information and verify against the adopted models to ex- tract the current emotion, joining it to the rest of the agents community that
represent other users. The overarching result is the availability of the immediate emotion of each user and the global emotion of all users and the emotional trends and evolution. The trend of the group emotion is important as it is easier to find the culprits of the emotional changes.
For instance, consider a bar, it contains a group of friends and has music playing. Now consider that some of these friends have different musical tastes.
If a music is playing that is enjoyable to 90% of the group we are able to no- tice a emotional progression towards "satisfied", while if it is the opposite (90%
dislikes) we can observe movement in the opposite direction. One of the possi- ble aim of this analysis is obtaining a playlist that is enjoyable to most of the group with a specific alignment of songs that cater to each individual like. This specificity is used to avoid reaching a tension point where one or more person has reached a saturation point and refuses any further change or abandons the environment.
The task of the iGenda is to consume this information a use its scheduling feature to change the events of each participant, based on his/her profile, to achieve a specific emotion, thus guiding the group to a common emotional state.
The emotional states are measured and placed in the PAD [13,14] model, out- putting a representation of the emotional state, easing the task of the iGenda on choosing new events. We have adopted the PAD model due to the simplicity and the datasets available. Furthermore, we are able to project the information captured into a 3D space, which facilitates the way the information is perceived.
2.2 The PAD model
The model seen in Figure 1 is a granular visual representation of the PAD model where the Valence replaces the Pleasure as in the psychological area is more representative and has a larger range of values [15]. The model has 12 sub- quadrants that are 30 degrees from each other. The emotion is represented by the vector #»
E(Ag) = [Arousal, V alence]and the representation of emotions uses polar coordinates, constituted by the angle and the magnitude of the vector.
We use a fuzzy logic process that transforms quantitative values to qualita- tive values that are able to be placed in the model showed in Figure 1. Thereon, a neural network is trained using the DEAPdataset3 as to achieve a real-world result from real humans. The correlation between the data captured and the dataset is established in the neural network meaning that each processed elec- trical input can be translated into an emotion.
2.3 Virtual Actors
A new development is the inclusion of virtual actors in the system that emulate the real participants. This feature aims to enhance of the decision-making process of the system, detecting in advance the possible emotional states and prepar- ing changes to the participants surrounding environment to accommodate these
3 http://www.eecs.qmul.ac.uk/mmv/datasets/deap/
Fig. 1.Emotions circle [16].
changes and proactively shift them to other states if the expected state is not desirable [17,18].
In this case specifically, when related to the elderly there is little margin for experimentation, as they have a large number of conditionings and are very fragile. The emotional profile helps to preview the possible reactions to changes on their daily routines and their daily emotion when performing specific activ- ities. The group emotion is very useful in environments like nursing homes and residential elderly communities. In this type of environment is very common to perform group activities and generate teams of elderly people. One common is- sue is the user verbal response and the real feeling, e.g., a person may tell that they are enjoying an activity but that be only a "kind" response due to peer pressure or to social engagement with the caregivers [19,20]. If the caregiver is able to receive unbiased information about how one feels about an activity it is easier to schedule similar or different activities or group that person with other people [21].
Another benefit from using virtual actors is the possibility of introducing Immersive Virtual Environments (IVE). In this specific case, the IVE’s would be used to project the environments that the users reside and simulate all of the possible components, meaning that every sensor/actuator is mapped as an agent and has agency, thus it is able to directly interact with the human agents. This forwards our research by having an safe environment were there is the ability of testing multiple outcomes of real interactions in fractions of seconds and project the optimal actions to achieve a certain outcome.
3 Conclusions
The main goal of this project is to create cohesive environments in which the users are happy and feel accomplished. This is an arduous task as people is com- posed of individuals that most of the time are very different to the rest. Further- more, there are several different applications to the ESW ecosystem apart from the healthcare area, like: crowd control, visual interfaces management, stress and fatigue monitoring, virtual environments societies, among others.
4 Acknowledgements
This work is partially supported by the MINECO/FEDER TIN2015-65515-C4- 1-R and the FPI grant AP2013-01276 awarded to Jaime-Andres Rincon. This work is supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT - Fundação para a Ciência e Tecnología within the projects UID/CEC/00319/2013 and Post-Doc scholarship SFRH/BPD/102696/2014 (A. Costa)
References
1. Bravo, J., Cook, D., Riva, G.: Ambient intelligence for health environments. Jour- nal of Biomedical Informatics64(December 2016) 207–210
2. Ramos, C.: Ambient Intelligence – A State of the Art from Artificial Intelligence Perspective. In Neves, J., Santos, M.F., Machado, J.M., eds.: Progress in Arti- ficial Intelligence. Volume 4874 of Lecture Notes in Computer Science., Berlin, Heidelberg, Springer Berlin Heidelberg (2007) 285–295
3. Cook, D.J., Augusto, J.C., Jakkula, V.R.: Ambient intelligence: Technologies, applications, and opportunities. Pervasive and Mobile Computing 5(4) (2009) 277–298
4. Gratch, J., Marsella, S.: Tears and fears. In: Proceedings of the fifth international conference on Autonomous agents - AGENTS’01, Association for Computing Ma- chinery (ACM) (2001)
5. Maaoui, C., Pruski, A.: Emotion recognition through physiological signals for human-machine communication. In: Cutting Edge Robotics 2010. InTech (Septem- ber 2010)
6. van Hoof, J., Kort, H., Rutten, P., Duijnstee, M.: Ageing-in-place with the use of ambient intelligence technology: Perspectives of older users. International Journal of Medical Informatics80(5) (May 2011) 310–331
7. Rincon, J.A., Costa, A., Novais, P., Julián, V., Carrascosa, C.: A dynamic emo- tional model for agent societies. In Demazeau, Y., Ito, T., Bajo, J., Escalona, M.J., eds.: Advances in Practical Applications of Scalable Multi-agent Systems.
The PAAMS Collection: 14th International Conference, PAAMS 2016, Sevilla, Spain, June 1-3, 2016, Proceedings. Volume 9662. Springer International Pub- lishing (2016) 169–182
8. Costa, A., Novais, P., Simoes, R.: A caregiver support platform within the scope of an ambient assisted living ecosystem. Sensors (Basel, Switzerland)14(3) (January 2014) 5654–5676
9. Costa, A., Julián, V., Novais, P.: Advances and trends for the development of ambient-assisted living platforms. Expert Systems (2016) n/a–n/a EXSY-Mar-15- 075.R1.
10. Estabrooks, P.A., Munroe, K.J., Fox, E.H., Gyurcsik, N.C., Hill, J.L., Lyon, R., Rosenkranz, S., Shannon, V.R.: Leadership in physical activity groups for older adults: A qualitative analysis. Journal of Aging and Physical Activity12(3) (jul 2004) 232–245
11. Rincon, J.A., Julián, V., Carrascosa, C.: Representing social emotions in MAS.
In Demazeau, Y., Decker, K.S., Pérez, J.B., de la Prieta, F., eds.: Advances in Practical Applications of Agents, Multi-Agent Systems, and Sustainability: The PAAMS Collection - 13th International Conference, PAAMS 2015, Salamanca, Spain, June 3-4, 2015, Proceedings. Volume 9086 of Lecture Notes in Computer Science., Springer (2015) 308–311
12. Rincon, J.A., Julián, V., Carrascosa, C.: An emotional-based hybrid application for human-agent societies. In Herrero, Á., Sedano, J., Baruque, B., Quintián, H., Corchado, E., eds.: 10th International Conference on Soft Computing Models in Industrial and Environmental Applications, SOCO 2015, Burgos, Spain, June 2015.
Volume 368 of Advances in Intelligent Systems and Computing., Springer (2015) 203–213
13. Mehrabian, A.: Analysis of affiliation-related traits in terms of the PAD tempera- ment model. The Journal of Psychology131(1) (January 1997) 101–117
14. Nanty, A., Gelin, R.: Fuzzy controlled PAD emotional state of a NAO robot.
In: 2013 Conference on Technologies and Applications of Artificial Intelligence, Institute of Electrical and Electronics Engineers (IEEE) (dec 2013)
15. Gupta, R., Hur, Y.J., Lavie, N.: Distracted by pleasure: Effects of positive versus negative valence on emotional capture under load. Emotion16(3) (2016) 328–337 16. Thayer, R.E.: The biopsychology of mood and arousal. Oxford University Press
(1989)
17. Minhat, H.S., Mohd Amin, R.: Sociodemographic Determinants of Leisure Partic- ipation Among Elderly in Malaysia. Journal of Community Health 37(4) (2011) 840–847
18. Hartgerink, J.M., Cramm, J.M., Bakker, T., van Eijsden, A.M., Mackenbach, J.P., Nieboer, A.P.: The importance of multidisciplinary teamwork and team climate for relational coordination among teams delivering care to older patients. Journal of Advanced Nursing70(4) (2014) 791–799
19. Fernández-Mayoralas, G., Rojo-Pérez, F., Martínez-Martín, P., Prieto-Flores, M.E., Rodríguez-Blázquez, C., Martín-García, S., Rojo-Abuín, J.M., Forjaz, M.J.:
Active ageing and quality of life: factors associated with participation in leisure activities among institutionalized older adults, with and without dementia. Aging
& Mental Health19(11) (2015) 1031–1041
20. Muñoz, D., Gutierrez, F., Ochoa, S.F., Baloian, N.: Enhancing social interaction between older adults and their families. In: International Workshop on Ambient Assisted Living, Springer (2013) 47–54
21. Chang, P.J., Wray, L., Lin, Y.: Social relationships, leisure activity, and health in older adults. Health Psychol33(46) (2014) 516–523
