Computational modeling of appraisal theory of emotion

(1)

Thesis

Reference

Computational modeling of appraisal theory of emotion

MEULEMAN, Ben

Abstract

Appraisal theories of emotion have proposed detailed—and causal—hypotheses about the connection between situations and emotional responding, and between the components that constitute emotional responding. Many of these hypotheses present computational challenges to scientific research, in that they require the analysis of numerous mental and bodily changes simultaneously over time. In this thesis, I applied statistical models of machine learning to address these challenges, and to investigate hypotheses concerning interaction effects, curvilinear associations, feedback among emotion components, synchronization of components, and the felt experience of synchronized changes (e.g., feeling angry). Results of the four studies generally supported the algorithmic complexity that underlies emotion unfolding, and that modelling these complexities is necessary to differentiate patterns of emotional responding quantitatively and qualitatively. Using a novel measure for emotional synchronization, I showed experimentally that changes in motivation, physiology, and expression responses synchronized following a manipulation of the [...]

MEULEMAN, Ben. Computational modeling of appraisal theory of emotion. Thèse de doctorat : Univ. Genève, 2015, no. FPSE 621

URN : urn:nbn:ch:unige-836386

DOI : 10.13097/archive-ouverte/unige:83638

Available at:

http://archive-ouverte.unige.ch/unige:83638

Disclaimer: layout of this document may differ from the published version.

1 / 1

(2)

Section de Psychologie

Sous la direction de Prof. Klaus Scherer, Prof. Olivier Renaud & Prof. Agnes Moors

COMPUTATIONAL MODELING OF APPRAISAL THEORY OF EMOTION

T

HESE

Présentée à la

Faculté de psychologie et des sciences de l’éducation de l’Université de Genève

pour obtenir le grade de Docteur en Psychologie

par

Ben MEULEMAN

de Gent, Belgique

Numéro d’étudiant : 11-335-007

Thèse No 621

GENEVE, Octobre, 2015

(3)

i

(4)

ii

Acknowledgments

This manuscript would not be complete without an acknowledgment of the people that contributed to my work and life over the past five years. Working in science has been extremely rewarding and I am grateful for the many opportunities that the university of Geneva provided to me. This project enabled me to learn new skills, visit new places, and above all meet new people. Finishing this Ph.D. would have been impossible without the support of my wonderful colleagues, friends, and family, all of whom made my life enjoyable during the past five years and bearable when the work was tough. I am aware how privileged this environment has been to work in.

In the first place, I would like to thank my three supervisors, who supported this project financially and scientifically. I could not have wished for a more impressive team of experts to advise me. Foremost, I thank Klaus Scherer, who enabled me to pursue this Ph.D. at the Swiss Center for Affective Sciences (CISA) and whose own scientific work formed the basis for my studies. Klaus has been one of the most progressive thinkers in affective science. His theory of emotion was first formulated more than three decades ago but to this day remains unparalleled in its scope and integration. His ideas on bodily synchronization and emergent feelings were ahead of their time, especially when considering that the computational tools to study these ideas were virtually absent when he proposed them. My appreciation for Klaus’s theory has only deepened in my attempts to operationalize it. Without this blueprint my work could not have existed and I hope I have been able to make a meaningful contribution. On a more practical note, I also want to thank Klaus for his availability for feedback and discussions. It is common for doctoral students to complain about their absent supervisors but I can happily say that this has not been my experience at all.

Agnes Moors I have known longer than any other of my advisors. I followed her classes on cognition and emotion back in 2006 and later wrote my master thesis under her supervision—also on appraisal theory. Without Agnes I never would have heard of the CISA in the first place. Moving to Geneva has probably been the best decision of my life and I am glad that she encouraged me to do it.

This paragraph would not be long enough to acknowledge how much I am indebted to her. On a professional note, I simply want to thank Agnes for her critical mind. The depth of her feedback has been intimidating, at times, but without her scientific rigour I know I would be a lazy scientist today.

Agnes’s feedback always pushed me to set the bar high, both in my writing and in the development of experiments. As a researcher, Agnes represents a type of meta-theorist that is sadly rare in affective science. That is, a theorist who reviews, compares, and integrates, and who reaches across disciplines.

Agnes’s papers helped me enormously to navigate the labyrinth of emotion theories and also taught me to appreciate the value of philosophy to research. The current academic climate makes it hard for pure theorists to thrive but I think science badly needs the kind of expertise that Agnes exemplifies. I can only hope for her to become even more influential and important in affective science. Finally, on a

(5)

iii

more personal note, I want to thank Agnes for her mental support, especially during the last years. I know that she cared for my well-being and I was glad to be able to talk to her when I was down.

Olivier Renaud I want to thank for the many insightful—and enjoyable—discussions on data analysis. His involvement has been less visible perhaps when compared to Klaus or Agnes but his contributions to the project were no less fundamental. Without his help the ELSA model would have remained a theoretical abstraction. The final version of ELSA incorporates many of his suggestions and improvements, for which he deserves much credit. I have been extremely lucky to have had such a complete team of supervisors. I am proud to be associated with all of them.

***

The last word in this section I want to reserve for my colleagues, friends, and family. An exhaustive list would be endless—and I fear I would miss out many names—so I will first extend a general note of thanks and appreciation to all the people at the CISA. It was a joy to work with all of you and, in many cases, share our lives. I know that, when I look back on the past five years, it is not the thesis that comes to mind first but the faces of all the people that I knew. Thank you all.

Special thanks goes to my close friends in Geneva. Katja I thank for being my first and oldest friend in Geneva, for her outgoing personality, and for the many fun evenings out that we spent in bars, in restaurants (especially Trulli), and at conferences. Hopefully many more will follow in the future. Vanessa I thank for making me laugh more than any other person at CISA, for her social intelligence, and for basically acting as my private therapist these past years. Eva I thank for her cheerfulness and optimism, and for all the fun moments that we shared together with Vanessa. I will be sad to see her leave Geneva and wish her all the best for California. Tommaso I thank for deep philosophical discussions on many life topics (politics, art, nerd hobbies, etc.) and for making our office at the old CISA a cool place to hang out. Géraldine I thank for her sharp mind, for her left- field—in the best sense—personality, and for the many late-night skype sessions during her time at Yale. Cristina I thank for her warm and caring personality. Kornelia I thank for being my dearest office neighbour at the Biotech Campus. Again, I apologize to all my colleagues and friends that I cannot thank personally in this section but I enjoyed the time we shared and appreciated your support.

Finally, a note of thanks to my Belgian friends and my family. I have been very happy living in Geneva for the past five years but I know that my Belgian social life has suffered as a result. I sincerely regret that I have not been able to be in touch with those friends as much as I wanted to be, especially during the last half year. I want to apologize for my absence and give a warm thanks to all the people who supported me at a distance. First I want to thank my best friends Tim and Evelien, who have stood by me since a long time and who always made me feel welcome to be back in Belgium.

Evelien I also thank for her scientific support and for being a type of researcher that I always aspired to be. I thank my two brothers, Jeroen and Rob, for all the fun times together (past and present), for

(6)

iv

visiting me in Geneva, and for hosting me during the Christmas holidays in Belgium. I thank my parents for their unwavering support (emotional, financial, technical, etc.) and for caring about what I do. I know they are very proud of me achieving this doctorate. Lastly, I want to give special thanks to my girlfriend Alexandra, who suffered with me through these final months but supported me throughout and never stopped believing in me.

Sincere thanks to all, Ben

(7)

v

Abstract

Appraisal theory is an important framework in affective science for the study of emotion processes.

Theories belonging to this framework all share as their primary assumption that emotions are caused by appraisal, that is, the subjective evaluation of an event or stimulus with respect to its significance to personal concerns. In addition to this causal hypothesis, appraisal theories have also made assumptions about the contents of emotion episodes, properties of these contents, relations between the contents, and their time course. The contents of an emotion episode are often defined by five components, which are appraisal, motivation, physiology, expression, and feeling. Changes in appraisal are assumed to cause changes in the other components, leading to a coherent and adaptive pattern of responses in the five components. This pattern is assumed to define an emotion episode. The relative level of detail makes appraisal theory an appealing framework for empirical modeling but also poses computational challenges. Three significant challenges are (C1) the large number of interacting mental and bodily components involved in emotion, (C2) the possibility of non-linear relations between the components, and (C3) the time-varying nature of emotion episodes. Traditional approaches to empirical modeling of appraisal theory—which includes classic computational approaches of engineering and classic statistical approaches of psychology—have not dealt with these challenges systematically. This has deterred important hypotheses about emotion to be tested.

For this thesis, I proposed a new computational approach to empirical modeling of emotion that operationalizes theoretical assumptions of appraisal theory using flexible statistical models from the field of machine learning. In four studies, I applied this approach to test five important hypotheses of appraisal theories, which are (H1) that appraisal variables interact to elicit and differentiate changes in other components of emotion, (H2) that there are curvilinear relations between appraisal and the other components of emotion, (H3) that there are feedback relations between emotion components, (H4) that components of emotion become relatively more synchronised during emotion episodes than during non-emotion episodes, and (H5) that the feeling component of emotion represents an integrated awareness of changes in the other emotion components. All these hypotheses address the algorithmic processes that underlie emotion unfolding, especially the mechanisms that link changes in appraisal to changes in motivation, physiology, expression, and that finally integrate into a cohesive emotional feeling.

The four studies that I conducted modeled emotion processes at increasing levels of complexity, encompassing different methodologies (observational, experimental, and simulation), different types of emotion data (recalled experiences, knowledge, online responding), large datasets, different statistical models (e.g., random forests, multivariate adaptive regression splines, liquid state machines, penalized regression), and confirmatory analyses via cross-validation. In the third study, I proposed a new computational model of emotion—based on Scherer’s theoretical Component Process Model—

that integrates time-series data of emotion components in a nonlinear dynamic system. This model

(8)

vi

handled all three computational challenges to empirical modeling of appraisal theory and allowed the testing of the five hypotheses. For this model, I also developed a new statistic to measure emotional synchronization. In the fourth study, I applied the model to data of an experiment that collected time series measurements of motivation, physiology, expression, and feeling, in response to manipulated appraisal of power and fairness.

Results of the four studies generally supported the hypotheses of appraisal theories. Empirical support was found for nonlinear relations between emotion components (H1–H3), such as interactions between appraisal variables (especially involving appraisal of causal agency and norm compatibility), threshold effects, and cross-component feedback. Accounting for such effects was found necessary to differentiate patterns of componential responses corresponding to important qualitative categories of emotion (e.g., joy responses, anger response, fear responses). Results also suggested that nonlinear effects in emotion data were only moderate. Interaction effects did not exceed the third degree and were often specific to individual subcomponents (e.g., the degree of tears responding as a function of appraisal variables), suggesting that heavily nonlinear predictions of many appraisal theories need to be revised. Results of modelling in the fourth study showed, for the first time, that there was a significant temporary increase in synchronization (H4) between motivation, physiology, and expression responses, following a manipulation of appraisal. Moreover, the degree of synchronization was associated with the intensity of self-reported feelings of surprise and irritation (H5). These results bear major significance to affective science, in that they supported a widely adopted view that emotions are characterized by temporary and synchronised responding in multiple mental and bodily components simultaneously. Models of machine learning successfully handled the computational challenges that emotion theories posed and therefore should become a standard tool of analysis for future research. In addition, the success of the new computational model of emotion that I proposed highlighted the need for componential time series measurement in experimentally controlled settings.

This methodology should be encouraged for future studies to achieve a more objective measurement of emotion.

(9)

vii

1 General introduction

1.1 Introduction and summary of the thesis

Emotions are complex phenomena that involve potentially the entire organism, from subtle mental changes (e.g., a focussing of attention) to overt behavior (e.g., aggressive behavior). Such patterns of reactions stand out not only by their globality (Freeman, 2005) but by their subjective feeling that what goes on is cohesive and qualitatively distinct (Scherer, 2004). Hence, emotions are often perceived of as categorical phenomena and denoted by labels such as anger, fear, or joy. An important challenge in affective science is to account precisely for how distributed mental and bodily changes integrate into a unified emotional experience. A theoretical framework that has been foremost in addressing this challenge is appraisal theory. Appraisal theory represents a collective of individual appraisal theories (e.g., Arnold, 1960; Frijda, 1986; Lazarus, 1991; Ortony, Clore, & Collins, 1988;

Roseman, 2001; Scherer, 1984) that all share as their central tenet the idea that the link between events and emotional responding is causally mediated by a process called appraisal. The appraisal process is defined as a subjective evaluation of importance with respect to personal goals and desires. In other words, appraisal theory assumes that a situation provokes an emotion only when that situation is appraised as being personally relevant.

When compared to other theoretical frameworks of emotion, appraisal theories have been relatively detailed in describing emotion processes. Models of appraisal have attempted not only to explain emotion causation but have also attempted to specify the contents of emotional responding (e.g., which mental and bodily changes should be involved), the relation between appraisal and emotion categories (e.g., how anger and fear patterns can be differentiated), and various process aspects about emotion involving temporal dynamics and sequencing. This level of detail has made appraisal theory an appealing framework for empirical modeling, which has been a part of experimental, observational, and simulation-based research. At the same time, empirical modeling of appraisal theory has faced computational challenges that have deterred the testing of important hypotheses. In this thesis I identified three such challenges and five such hypotheses (Figure 1.1 and Table 1.1). The challenges are (C1) the high dimensionality of the set of mental and bodily changes to be modeled, (C2) the possibility of nonlinear associations, and (C3) the need to incorporate time.

With respect to challenge C1, many appraisal theories have subscribed to the view that emotions are componential. This term refers to the idea that the mental and bodily changes of an emotion episode can be grouped into components that correspond to major organismic subsystems. Frequently cited components of emotion are appraisal, motivation, physiology, expression, and feeling. Each of these components is assumed to encompass numerous subcomponents, for example, changes in heart rate, breathing rate, or skin conductance, for the physiology component. Collectively, the five

(14)

2

components of emotion cover most of the human mind and body and, as such, empirical modeling is tasked with the difficult problem of managing a large number of interdependent subcomponents simultaneously (Wehrle & Scherer, 2001). From a statistical point of view, the set of variables would be called high dimensional. This problem is relevant in particular for hypotheses that involve all components of emotion simultaneously, such as the hypothesis of emotional synchronization (H4) and the hypothesis of feeling integration (H5). The synchronization hypothesis claims—concisely stated—

that components of emotion should be more coherently organized during an emotion episode than during a non-emotion episode (e.g., Freeman, 2005; Gross, 2010; Scherer, 2001; 2009a). The feeling integration hypothesis states that the contents of the feeling component of emotion represent an integrated awareness of changes in the other components (Grandjean, Sander, & Scherer, 2008;

Moors, 2009; Scherer, 2004).

Figure 1.1. Schematic overview of computational challenges in modeling emotion and its related hypotheses. Absent arrows indicate that testing the hypothesis in question does not (necessarily) depend on the computational challenge to be solved.

With respect to challenge C2, many appraisal theorists have hypothesized that associations between appraisal and other emotion components are nonlinear. This hypothesis has been expressed in three prominent areas, which are (H1) interaction effects of appraisals on other emotion components (Ortony et al., 1988; Lazarus, 2001; Roseman, 2001; Scherer, 2001), (H2) curvilinear associations between appraisal criteria and other emotion components (Kappas, 2001; Tong, Ellsworth & Bishop, 2009), and (H3) time-dependent feedback processes between emotion components (Lewis, 2005;

Scherer, 2009a). Such hypotheses demand further complexity of empirical models of emotion and necessitate the use of nonlinear methods, which have not been widely used in affective science (Lewis,

(15)

3

2005). As a consequence, nonlinear hypotheses have not been researched systematically in empirical literature, despite being a core aspect of many appraisal theories.

Table 1.1. Overview of computational challenges and hypotheses.

Hypotheses of appraisal theories and related computational challenges

Hypothesis H1 Appraisal

interactions

Major theories of appraisal assume that appraisal criteria interact to elicit and differentiate responding in the other emotion components (Ortony et al., 1988; Lazarus, 2001; Roseman, 2001; Scherer, 2001).

H2 Curvilinear associations

Several appraisal theorists assume that there should be curvilinear associations between the appraisal component and other components of emotion, such as sigmoid or polynomial associations (Kappas, 2001; Tong & Tay, 2011; Tong et al., 2005; 2009).

H3 Component feedback

Temporal theories of emotion assume that components of emotion cross-influence each other over time, through feedforward and feedback relations (Lewis, 2005; Scherer, 2001, 2009a).

H4 Component synchronization

Multiple theories of emotion assume that components of emotion are organised during an emotion episode in a manner that is more coherent than during non-emotion episodes (e.g., Bulteel et al., 2013; Ekman, 1972, 1992; Freeman, 2005; Gross, 2010; Lazarus, 1991; Levenson, 1994, 2003; Matsumoto, Nezlek, & Koopmann, 2007; Scherer, 1984; 2001; 2009a; Tomkins, 1962)

H5 Feeling integration

The feeling component of emotion is often assumed to be an integrated awareness of some or all changes in the other emotion components (Moors 2009; 2013; Scherer, 2000; 2004; 2009a).

Challenge C1 High-

dimensional features

Componential definitions of emotion claim that emotions consist of components (i.e., appraisal, motivation, physiology, expression, feeling; Frijda, 2008; Frijda & Scherer, 2009; Moors, 2009;

Scherer, 2005). These components consist of numerous subcomponents and, collectively, represent a substantial part of the mind and body. Computationally, this implies high- dimensional datasets of emotion features and requires multivariate models for analyzing emotion data.

C2 Nonlinear associations

Appraisal theories have posited that the relation between the appraisal component and the other emotion components are nonlinear (Kappas, 2001; Lazarus, 2001; Lewis, 2005; Ortony et al., 1988; Roseman, 2001; Scherer, 2000; 2001; 2009a). Computationally, this requires nonlinear models for analyzing emotion data.

C3 Time- dependence

Emotion theorists assume that emotions are time-varying phenomena, or episodes, with temporal properties such as onset, dynamics, and offset (Davidson, 1998; 2015; Kuppens &

Verduyn, 2010). Computationally, this requires emotion data to be modeled over time (e.g., in time-series models).

With respect to challenge C3, emotions are widely considered to be time-varying phenomena that are characterized by temporal properties such as onset, dynamics, offset, and duration (Davidson, 1998; 2015; Kuppens & Verduyn, 2010). Hypotheses have been formulated about all these properties, with some authors appealing to mathematical frameworks such as dynamic systems theory to explain

(16)

4

emotion unfolding (Lewis & Granic, 2000). In particular, the dynamic systems approach has been used to elaborate on the synchronization hypothesis (H4) and the feeling integration hypothesis (H5).

Scherer (2000; 2001; 2009a;b) has proposed that emotional synchronization is a product of the time- dynamic interaction between appraisal, motivation, physiology, and expression, and that the feeling component is emergent upon the degree of synchronization. However, time-dependent modeling of emotion processes has emerged only recently as a noteworthy branch of empirical research.

Traditional approaches to modeling appraisal theory have difficulty coping with the three challenges that I listed (Figure 1.1). These traditional approaches include classic computational models (Marsella, Gratch, & Petta, 2010), which directly operationalize theoretical appraisal models as computer programs, and linear statistical models, which estimate hypothesized relations from empirical data (e.g., linear regression, linear discriminant analysis). For this thesis, I adopted a hybrid computational approach that integrates the two traditional approaches by combining structural assumptions of appraisal theories with nonlinear statistical methods from the field of machine learning. Machine learning offers a wide range of models (e.g., artificial neural networks, support vector machines) that are capable of extracting complex patterns from empirical data, including data that are high dimensional or time varying (see Bishop, 2006, Hastie, Tibshirani, & Friedman, 2009, and Ripley, 1996, for overviews). These methods are well-suited to the computational demands of emotion theories—such as appraisal theories—and for testing complex hypotheses such as those given in Figure 1.1 and Table 1.1.

The primary goal of this thesis was to use machine learning to deal with the three computational challenges I identified and to test empirically the five hypotheses of appraisal theories that are affected by these challenges. To achieve this, I conducted four major studies with large datasets. In these studies, I modeled appraisal theory with an increasing degree of complexity, concluding with a full operationalization of an appraisal theory that incorporated all components of emotion, nonlinear processes, and time in a single statistical model.

This thesis is organized as follows. The remainder of this chapter is first dedicated to discussing the theoretical background of the studies in detail. In those sections, I elaborate further upon the issues that I raised in this introduction (e.g., definition of emotion, appraisal theory), I give an overview of computational challenges for modeling appraisal theory and I present the solution to these challenges for this thesis. Chapters 2 through 5 cover the empirical studies that I conducted for the thesis.

Chapters 2 and 3 focus on the modeling of interaction effects in appraisal theory for large databases of recalled emotion experiences (Chapter 2) and of cross-cultural emotion knowledge in all components of emotion (Chapter 3). Chapters 4 and 5 focus on the development and application of a practical computational model for emotion simulation, called Emergent Liquid State Affect (ELSA). This model operationalizes a concrete appraisal theory to allow the integration of time-varying data on all components of emotion and the analysis of emotional synchronization. Chapter 4 provides the theoretical background of the model and presents two data simulations, one using synthetically

(17)

5

generated data and one using real data from an appraisal experiment that manipulated appraisal criteria and measured electro-encephalogram (EEG) and electro-myogram (EMG) activity. Chapter 5 applies the ELSA model to a new study that was tailored to collect time-varying data on all emotion components simultaneously, in response to manipulated appraisal criteria. Chapter 6, finally, is devoted to a general discussion of the research findings presented in the four empirical studies.

1.2 Theoretical background

1.2.1 Definition of emotion

Before discussing appraisal theory in detail it is important to provide a general frame of reference on emotion. This requires foremost a clear definition of the term emotion. Unfortunately, defining emotion has been a source of ongoing debate in affective science. Disagreements have focused, among others, on the contents of emotion, the relations between the contents, and process aspects of these contents. At the same time, an agreement on many of these aspects has also emerged by a kind of majority consensus. In psychological literature, this consensus has converged upon the componential definition of emotion (Frijda, 2008; Frijda & Scherer, 2009; Moors, 2009; Scherer, 2005). This definition states simply that emotions consist of components—as introduced in the previous section. In this thesis I used the following version of the componential definition:

An emotion is an episode of concurrent changes in appraisal, motivation, physiology, expression, and feeling, as an adaptive response to a change in the environment.

This definition is also depicted schematically in Figure 1.2. The componential part of the definition is represented by the inclusion of five emotion components, that is, appraisal, motivation, physiology, expression, and feeling. Each of the five components is assumed to (a) represent a distinct subset of emotional responding, (b) originate from a corresponding mental or bodily subsystem (e.g., a motivation system, a physiology system), and (c) play a functional role during an emotion episode (Scherer, 2005). In the following sections I summarize these aspects for each of the components (see Fontaine, Scherer, & Soriano, 2013, for a book-length treatment of componential emotions).

The appraisal component is often referred to as the cognitive part of emotion. The function of this component is to appraise situations with respect to their personal significance (Arnold, 1960;

Frijda, 1986; Lazarus, 1991; Scherer, 1988) and, as such, to determine which situations merit an emotional response and which do not. It is assumed that a person continuously appraises her environment in this manner, and that this process occurs largely automatically (Frijda & Zeelenberg, 2001; Moors, 2009; Oatley & Jenkins, 1996; Roseman & Smith, 2001; Scherer, 2001). The content of

(18)

6

the personal significance that is being evaluated is typically restricted to a limited number of dimensions or criteria, such as goal relevance, compatibility with goals and desires, compatibility with expectations, causal agency, potential to cope with or control the situation, and compatibility with norms and values (see Ellsworth & Scherer, 2003, for lists of common criteria). Such criteria can be considered as communalities that are latent to emotion-eliciting situations (e.g., goal blocking is a communality of anger-eliciting situations). Appraisal theories assume not only that the appraisal component elicits changes in the other emotion components but also that specific patterns of appraisals will differentiate patterns of emotional responding qualitatively (e.g., a guilt or a shame episode).

Because appraisal theories are a central focus of this thesis, I present a further discussion of their assumptions and predictions in Section 1.2.1.

The motivation component of emotion involves the preparation of behaviour through changes in action tendencies, such as the tendency to approach, to avoid, or to attack (Frijda, 1986, 2009; Haidt, 2003; Prinz, 2010; Roseman, 2001; Scherer, 2005; Smith & Lazarus, 1990). Action tendencies help the organism adapt to a relevant situation by prioritizing goals and solutions that are appropriate for dealing with that situation (Frijda, 1986, 2010; Frijda & Mesquita, 1998). The tendency is expected to facilitate the execution of concrete actions but need not necessarily be acted upon. As such, action tendencies serve as a kind of latent mediator between appraisal and overt emotional behavior. This latent state is expected to be terminated when the desired goal is achieved or is no longer relevant (Austin & Vancouver, 1996). Due to its adaptive function, the motivation component has been particularly prominent in evolutionary accounts of emotion (Ledoux, 2012; Plutchik, 1980; 2001;

2003).

The physiology component is also referred to as the somatic component of emotion and involves bodily reactions produced by the central and peripheral nervous systems, as well as hormonal changes produced by the endocrine system (e.g., changes in heart rate, blood pressure, skin conductance, or cortisol levels). Like motivation, the role of the physiology component is to support adaptive behaviour during an emotion episode (Stemmler, 2004; 2009). Physiological responding has featured in the earliest studies of emotion (e.g., Cannon, 1915; James, 1884) and has received extensive attention in contemporary empirical research (see Kreibig, 2010, for a an overview of theoretical debates and a qualitative review of research).

The expression component is also referred to as the motor component. The function of this component is to execute or communicate actions prepared by motivational tendencies and physiological activity (Mortillaro & Scherer, 2009). This can involve gross behavior (e.g., escaping, attacking), changes in facial expression (e.g., frowning), or changes in vocal expression (e.g., screaming). The scientific study of emotional expression—especially facial changes—dates back to Darwin (1965) and later emerged as a fundamental area of research in discrete emotion theories (Tomkins, 1962; Ekman, 1972, 1992; Izard, 1971, 1993).

(19)

7

Figure 1.2. Schematic definition of emotion. An appraised event elicits changes in appraisal, motivation, physiology, and expression. These changes lead to an integrated feeling component that can be labeled or categorized with an emotion word.

Bidirectional arrows between components imply time-dependent feedback processes.

The feeling component represents the experiential part of emotion. It is the conscious awareness of the ongoing changes in the other emotion components, represented as a kind of integrated Gestalt (Moors, 2009). These qualities make the feeling component highly salient subjectively during an emotion episode. In contrast to the changes in the other emotion components, which are distributed across the entire body and mind, feeling provides the sensation that what goes on is cohesive and qualitatively distinct from not being in an emotional state. The function of the feeling component is to allow the person to monitor their internal state and act on it by conscious regulation. This affords a certain degree of control over the responses that are promoted or executed by the motivational, physiological, and expression components. The contents of feeling can include subjective experiences of intensity, duration, dynamics, and also quality. The latter is often verbalized by emotion terms (e.g.,

“anger”, “fear”, “joy”). As Figure 1.2 shows, however, the act of labeling is considered to be separate and optional from any feeling changes, and therefore should not be conflated (Scherer, 2004).¹ Nevertheless, feeling labels represent highly salient and unified descriptors of emotion. For this reason, many theoretical models of emotion have put a strong emphasis on explaining variation in feeling categories (e.g., Ortony et al., 1988; Roseman, 2001; Scherer, 2001). In empirical research, the majority of studies have used ratings of feeling labels as the principal measure of emotions. Some of these data have motivated psychological constructivist emotion theories (e.g., Barrett, 2006; Russell, 2003) to challenge the notion that emotion categories are natural kinds. Instead, these theories assume

1 Note that, in practice, emotion words are often used interchangeably to describe feeling states and emotional episodes as a whole. Although these two uses are expected to be overlapping (i.e., an anger feeling implies that the person experiences an anger episode, and vice-versa), the former is a more narrow use than the latter.

(20)

8

a dimensional feeling space that underlies all emotion terms, spanning valence (feeling good versus bad) and arousal (feeling calm versus excited).

The definition of emotion that I presented in this section focused mainly on componentiality.

Two additional aspects cited in the definition require further comment. First, I added that component changes are considered as an adaptive response to a relevant event (e.g., Lazarus, 1991; Smith &

Kirby, 2001). This implies that the collective of changes have a clear and directed function and that emotions are not random or “irrational”, as has sometimes been claimed (see Dixon, 2003, and De Sousa, 1987, for a discussion on historic connotations of the word “emotion”). In the previous section I highlighted how the motivation and physiology components in particular promote behavior that is adaptive to an environmental change, when triggered by the appraised relevance of the situation.

Second, I added that emotions are episodes. The word “episode” makes clear that emotions are considered as transient and time-varying phenomena. Davidson (1998; 2015) coined the term

“affective chronometry” to refer to the study of emotion and its temporal parameters (e.g., onset, dynamics, offset, duration). These parameters are widely acknowledged but have not been elaborated upon in many emotion theories. In Figure 1.2, time dependence is assumed through the bi-directional arrows between emotion components. These arrows indicate feedback relations between components, which implies time-dependence.

1.2.2 Appraisal theories

Emotion theories are often categorized into families such as discrete theories, constructivist theories, and appraisal theories. The existence of these families reflects a history of scientific debate on fundamental aspects of emotion, such as its definition, but sometimes merely a difference in focus on one emotion component versus another (e.g., basic emotion theories have emphasized the expression component in research, whereas appraisal theories have emphasized the appraisal component).

Appraisal theories have primarily distinguished themselves from competing theory families by the comprehensiveness with which they model emotion processes, by addressing virtually all steps that lead from an event to an emotion episode. Two important phases in these steps are (I) appraisal derivation from an event to appraisal values and (II) component derivation from appraisal values to values in all other components of emotion (Marsella et al., 2010; Moors & Scherer, 2013). With respect to appraisal derivation, appraisal theorists have specified the content of appraisal (e.g., values on criteria of relevance, goal compatibility), the underlying mechanisms that lead to this content (e.g., dual or triple route), the format of the representations that the mechanisms operate on (e.g., perceptual, symbolic), and the degree of automaticity (e.g., consciousness, efficiency, speed, control; Moors, 2010). With respect to component derivation, appraisal theorists have specified the contents of component patterning, the underlying algorithms (e.g., linear versus nonlinear), the underlying organismic subsystems (e.g., the central nervous system, the endocrine system), temporal sequencing,

(21)

9

and predictions for concrete sets of component patterning (e.g., fear component patterning, joy component patterning; Moors et al., 2013; Roseman & Smith, 2001).

A full discussion of the two phases and their aspects in appraisal theory would be outside the scope of this thesis (see Moors et al., 2013 for an overview). I focused on the operationalization of appraisal models for component derivation. In Figure 1.2, this phase is represented by all arrows within the dotted emotion episode box. Appraisal theories have been strong advocates of componentiality (Moors et al., 2013) and have assumed strong links between appraisal and changes in motivation (e.g., Ellsworth & Tong, 2006; Frijda, 1987; Frijda, Kuipers, & ter Schure, 1989; Lazarus, 1991; Roseman, 2011), physiology (Aue & Scherer, 2007; Scherer, 1993, 2001, 2009b), expression (e.g., Laird & Bresler, 1992; Kaiser & Wehrle, 2001; Scherer, 2009b; Scherer & Ellgring, 2007), and feeling (e.g., Lazarus, 2001; Ortony et al., 1988; Roseman, 2001; Scherer, 2001). Appraisal theorists argue for specificity and differentiation in component changes for major emotion categories, and sometimes even for individual appraisal changes (Scherer, 2001; 2009b).

Componentiality in appraisal theory is also reflected by hypotheses that involve more than two components of emotion simultaneously. The feeling integration hypothesis (H5) posits that the content of feelings represent an integration of changes of other components that have reached consciousness (Ellsworth, 1991; Frijda, 2007; Scherer, 2009a). The emotion coherence hypothesis (H4) posits that, in general, component changes mutually coordinate during an emotion episode to form coherent patterns (e.g., Bulteel et al., 2013; Ekman, 1972, 1992; Freeman, 2005; Gross, 2010; Lazarus, 1991; Levenson, 1994, 2003; Matsumoto, Nezlek, & Koopmann, 2007; Scherer, 1984; 2001; 2009a; Tomkins, 1962).

Scherer (1984, 2001, 2009a) refers to this type of coherence as synchronization. His appraisal theory, called the Component Process Model (CPM; 1984, 2001, 2009b) assumes that there is time-dependent cross-influencing between emotion components (i.e., feedforward and feedback), and that the degree of synchronization between appraisal, motivation, physiology, and expression generates an integrated feeling as an emergent product.

Finally, appraisal theories have proposed hypotheses about component changes and their relation to discrete categories of emotion (represented by verbal labels). Indeed, predictions of this kind are the most common type of hypothesis in appraisal theory. Lazarus (1991, 2001), Ortony et al.

(1988), Roseman (2001, 2011, 2013), and Scherer (2001, 2009b) have all put forward tables or diagrams that link specific appraisal configurations to discrete emotion categories (e.g., “anger”, “joy”,

“guilt”). An example from the CPM model is given in Table 1.2. For some of the appraisal theories just cited, variation in labeled emotion categories is the primary to-be-explained phenomenon (Lazarus, 1991; 2001; Ortony et al., 1988; Roseman, 2001; 2011; 2013). Other theories seek to explain variation in all components, without necessarily linking the patterns to feeling labels (e.g., Scherer, 2001, 2009a). For these theories, the space of component patterning is considered potentially infinite.

These two classes of theories have been identified by Moors as two flavours of appraisal theories (Moors, 2014).

(22)

10

Table 1.2. Predicted appraisal profiles for selected emotion categories, proposed by Scherer (2001). Table adapted from (Scherer & Meuleman, 2011).

Appraisal criterion Joy Rage Fear Sadness Relevance

Novelty

Suddenness High High High Low

Familiarity Open Low Low Low

Predictability Low Low Low Open

Intrinsic pleasantness Open Open Low Open

Goal/need relevance High High High High

Implication

Cause: agent Open Other Other Open Cause: motive Open Intentional Open Chance Outcome probability Very high Very high High Very high Discrepancy from expectation Open Dissonant Dissonant Open Conduciveness Conducive Obstructive Obstructive Obstructive

Urgency Low High Very high Low

Coping potential

Control Open High Open Very low

Power Open High Very low Very low Adjustment Medium High Low Medium Normative significance

Internal standards Open Open Open Open External standards Open Low Open Open

In summary, appraisal theories of emotion represent a family of highly detailed emotion theories in affective science. These theories posit that situations are automatically appraised according to criteria relating to personal concerns, and that these appraisals elicit and differentiate coherent responding in motivation, physiology, and expression. The conscious experience of all (or some of) these changes is integrated into a cohesive feeling, that can be labeled with an emotion word, such as

“anger”.

1.3 Empirical modeling of appraisal theory

The level of detail in appraisal theory has made this an appealing framework for empirical modeling, which has been a part of experimental, observational, and simulation-based research (Moors &

Scherer, 2013). At the same time, empirical modeling of appraisal theory has faced computational challenges that have deterred the testing of important hypotheses. In this thesis I identified three such

(23)

11

challenges, which are (C1) the high-dimensionality of the feature set, (C2) nonlinear associations, and (C3) time-dependence (Figure 1.1). In the following sections, I discuss these challenges and connect them to five important hypotheses of appraisal theories that have not been systematically tested. In addition, I review how existing literature has (or has not) dealt with the challenges.

Two general approaches to empirical modeling that have emerged in this literature are the classic computational approach and the classic statistical approach. The classic computational approach attempts to translate appraisal theories (e.g., prediction tables) directly into computer programs, which are often simply called computational models of appraisal (CMA; Marsella et al., 2010). This approach originated from the field of engineering, is largely theory driven, and has been used, for example, to create virtual agents endowed with “emotions”. The classic statistical approach collects empirical data (e.g., observational or experimental) and estimates relations among emotion variables with linear regression models. This approach has been the standard method for empirical research in psychological science, is largely data driven, and is used primarily for the direct testing of appraisal hypotheses (e.g., ANOVA). Each of the two classic approaches to empirical modeling possess relative advantages and disadvantages for dealing with the computational challenges that I identified. I discuss these advantages and disadvantages below.

1.3.1 Challenge 1: High-dimensional features

The first computational challenge (C1) concerns the high number of features that appraisal theories have implicated in emotional episodes. The componential definition of emotion requires five components to be accounted for (appraisal, motivation, physiology, expression, feeling) and each of these components, in turn, consists of numerous subcomponents or features. As an example of the potential size of features, the study by Fontaine and colleagues (2013) surveyed respondents on their knowledge of emotion and included 144 items in total, across the five components. An empirical model that includes all components of emotion is a necessary requirement for the testing of the synchronization hypothesis (H4), and the feeling integration hypothesis (H5; see previous sections;

Figure 1.1).

In the classic statistical approach, research that measures and models all components of emotion remains rare. The majority of psychological appraisal studies have instead focused on the estimation of bi-componential relations. Most of this research has studied the link between appraisal and feeling—the latter typically represented by category labels (e.g., “anger”, “fear”), but there is also also empirical work linking appraisal to (a) motivation changes, such as tendencies to attack or repair (e.g., Bossuyt, Moors, & De Houwer, 2014a,b; Fast & Chen, 2009; Frijda, Kuipers, & ter Schure, 1988;

Moors & De Houwer, 2001; Nelissen & Zeelenberg, 2009), (b) physiological changes, such as cardiac, respiratory, temperature, and electrodermal responses (e.g., Aue, Flykt, & Scherer, 2007; Delplanque et al., 2009; Gentsch, Grandjean, & Scherer, 2013; Gentsch, Grandjean, & Scherer, 2014; Grandjean

(24)

12

& Scherer, 2008; Kreibig, Gendolla, & Scherer, 2010; Kreibig, Gendolla, & Scherer, 2012; Lanctot &

Hess, 2007; van Reekum et al., 2004), and (c) expression changes, such as facial muscle activity, vocal activity, and body posture (e.g., Banse & Scherer, 1996; Dael, Mortillaro, & Scherer, 2012; Gentsch et al., 2014; Kaiser & Wehrle, 2001; Mortillaro, Mehu, & Scherer, 2011; Patel, Scherer, Björkner, &

Sundberg, 2011; Scherer, Mortillaro, Mehu, 2013; van Reekum et al., 2004). All these studies have generally supported componentiality in emotion but they do not allow testing multi-componential hypotheses.

A handful of studies have collected data on more than two components of emotion, and sometimes all five simultaneously (e.g., Bossuyt et al., 2014b; Fitness & Fletcher, 1993; Fontaine et al., 2013; Kuppens et al., 2003; Scherer & Wallbott, 1994) but none of these studies analysed the collected data in an integrated multivariate model. Moreover, emotions in these studies were treated as summarized states rather than as time-varying episodes. Although one can study cross-component coherence in time-independent data, a strict test of the synchronization hypothesis—according to Scherer’s definition—requires time-varying emotion data. Nevertheless, the GRID study by Fontaine et al. (2013) represents the most comprehensive attempt for multi-componential measurement in emotion to date. Discriminant analyses on these data have supported some degree of coherence in component patterning for major emotion categories.

In the classic computational approach it is standard practice to include multiple components of emotion simultaneously in CMAs (e.g., Becker-Asano & Wachsmuth, 2010; Bui et al., 2002; Elliot, 1992; El-Nasr et al., 2000; Gratch & Marsella, 2004; Marsella & Gratch, 2008; Velasquez, 1997;

Wehrle, 1995; Wehrle & Scherer, 2001). Typically, the components of appraisal, feeling, and expression are accounted for, sometimes motivation and—much less frequently—physiology (but see de Melo & Gratch, 2009, and Jung et al., 2009). This makes CMAs relatively more faithful to the assumption of componentiality than empirical modeling in the classic statistical approach.

Unfortunately, CMAs are not programmed to validate concrete scientific hypotheses but rather to achieve a plausible simulation or prediction of emotional behavior (e.g., decision making, naturalistic conversing). Emotional coherence is often hardwired by selecting prototypical response patterns to match an outputted label of an appraisal module (e.g., the OCC model). This type of architecture violates appraisal theories of the second flavor by assuming that relations between appraisal and other components of emotion are mediated by a “programmatic” emotion category (e.g., Bui et al., 2002;

Marsella & Gratch, 2008). In addition, the feeling component has no clear interpretation in a CMA, since there is no integration of other components and the model or avatar (usually) cannot report directly on its experience. The plausibility of the emotional output has to be judged largely on intuitive criteria (i.e., “does it look human?”).

1.3.2 Challenge 2: Nonlinear associations

(25)

13

The second computational challenge (C2) concerns nonlinearity. Many appraisal theorists have hypothesized that associations between appraisal and other emotion components are nonlinear. This hypothesis has been expressed in three prominent areas, which are (H1) interaction effects of appraisals on other emotion components (Ortony et al., 1988; Lazarus, 2001; Roseman, 2001; Scherer, 2001), (H2) curvilinear associations between appraisal criteria and other emotion components (Kappas, 2001; Tong et al., 2009a), and (H3) time-dependent feedback processes between emotion components (Lewis, 2005; Scherer, 2009a). Illustrative examples of these three hypotheses are depicted in Figure 1.3.

Figure 1.3. Illustrative examples of nonlinear hypotheses in appraisal theory. H1: Appraisals A and B interact to predict the response. H2: The effect of an appraisal on the response flattens for extreme values. H3: The effect of an appraisal on the response depends on past response states.

Two appraisal criteria interact when the effect of one appraisal criterion on emotional responding is modified by levels of another appraisal criterion. For example, in Figure 1.3 (left panel), the relation between an appraisal A and a component response differs for levels of a second appraisal B. That is, when B is “on” (i.e., B = 1), the effect of A is much stronger. An interaction is a type of nonlinear effect. All major appraisal theories predict interactions, including the theories of Ortony et al. (1988), Lazarus (1993; 2001), Roseman (2001; 2009), and Scherer (1984; 2001; 2009a). Interaction hypotheses in these theories usually take the form of configurational appraisal profiles associated with discrete emotion categories (e.g., Table 1.2), such as decision trees (Ortony et al., 1988, p. 19) or contingency tables (Roseman, 2001, pp. 70–71; Scherer, 2001, pp. 114–115). Such configurations imply that an emotion is not just differentiated by a single appraisal criterion but by a joint combination of multiple appraisal criteria.

Curvilinear associations between appraisal input and component responses have been proposed by Kappas (2001) and Tong and colleagues (Tong & Tay, 2011; Tong et al., 2009a; 2009b). Kappas

(26)

14

(2001) hypothesizes that the relation between appraisals and the intensity of emotional responding should be sigmoidal. A sigmoidal function is an s-shaped curve that transitions smoothly between a lower and an upper output limit (Figure 1.3, middle panel).² When applied to emotional responding, this corresponds to the assumption that the intensity of a response (in any component) should be bounded to a fixed minimum and maximum. More general shapes of nonlinear associations were proposed by Tong and colleagues (Tong & Tay, 2011; Tong et al., 2009a; 2009b), such as basic polynomials (e.g., quadratic or cubic curves). A nonlinear curve can be viewed as an interaction between an appraisal criterion and itself. That is, in Figure 1.3, the relation between the appraisal and the response is modified by levels of the appraisal itself.

Feedback between emotion components is generally expected due to the hardwired interconnectivity of different organismic networks, which includes feedforward and feedback connections (e.g., Damasio, 1998; Koelsch et al., 2015, Ledoux, 2012; Lewis, 2005; Sander, Grandjean, & Scherer, 2005). As a nonlinear process, feedback is a key aspect in the theories of Lewis (2005) and Scherer (2000, 2001, 2009a; Sander et al., 2005). These authors have applied principles of dynamic systems theory to characterize the time-dependent cross-influencing between emotion components. Dynamic systems theory is a branch of mathematics devoted to studying the behavior of interdependent systems over time. Feedback loops inside such systems can rapidly self-amplify ongoing activity, much faster than would be predicted in a linear system. Scherer (2000; 2009a) has invoked this principle to predict hysteresis effects in emotion processes, that is, feedback-dependent switches between emotion episodes as a function of ongoing appraisal (see also Sacharin, Sander &

Scherer, 2012).

The classic statistical approach to appraisal modeling has generally not applied nonlinear models to empirical data (Lewis, 2005). Observational studies on appraisal theory typically use linear methods such as linear regression³ or linear discriminant analysis (e.g., Brans & Verduyn, 2014;

Ellsworth & Smith, 1998; Frijda, Kuipers & Ter Schure, 1989; Hosany, 2011; Kuppens et al., 2003;

McGraw, 1987; Reisenzein & Spielhofer, 1994; Ruth, Brunel & Ötnes, 2002; Scherer, 1997; Scherer

& Ceschi, 1997; Siemer, Mauss & Gross, 2007; Smith & Ellsworth, 1985; Sonnemans & Frijda, 1995;

Tong et al., 2005). Experimental studies have tested interactions between appraisal criteria with factorial ANOVA (e.g., Aue et al., 2007; Gentsch et al., 2013; 2014; Kreibig et al., 2012; Nyer, 1997) but used limited two- or (at the most) three-way designs with categorical appraisal values. In general, such studies were not undertaken with the explicit purpose of testing nonlinear associations. An exception are the studies by Tong and colleagues (Tong et al., 2005; 2009a), who found evidence for a three-way interaction of appraisal criteria on joy intensity (Tong et al., 2005), and for curvilinear relationships (i.e., quadratic and cubic) between appraisal criteria and feeling intensities (Tong & Tay,

2 For example the logistic function or the hyperbolic tangent function.

3 Including multiple regression, t-tests, one-way ANOVA, and Pearson correlation analysis.

(27)

15

2011; Tong et al., 2009a). To date, no study has explicitly tested the hypothesis of time-dependent feedback in the context of a time series analysis of emotion measures.

In the classic computational approach, models have adopted nonlinearity primarily as a result of directly operationalizing configurational predictions of appraisal theories. The majority of CMAs have adopted the OCC model (see Bartneck, Lyons & Saerbeck, 2008 for a review), which contains interactions between appraisal criteria up to the fourth degree. In addition, emotional output—

especially categorical emotion states—is sometimes calculated in a probabilistic manner by logistic decision curves (e.g., Marsella & Gratch, 2008; Becker-Asano & Wachsmuth, 2010). Other models have adopted additional forms of nonlinearity, such as exponential decay (e.g., Bui et al., 2002; El- Nasr et al., 2000; Velasquez, 1997), fuzzy logic rules (El-Nasr et al., 2000), or threshold rules (e.g., Kim & Kwon, 2010). Some of these simulations have been instructive to expose limitations in appraisal models, such as the OCC model (Steunebrink, Dastani & Meyer, 2009), leading modelers to improve existing algorithms or integrate multiple appraisal theories simultaneously (e.g., Becker- Asano & Wachsmuth, 2010). However, manual adjustments to appraisal theories are subject to the same human limitations that created the theories in the first place. Although some models have implemented learning rules for data-driven adaptation (e.g., El-Nasr et al., 2000), by and large, CMAs remain strongly theory driven, and lack the flexibility of statistical models to detect complex nonlinear relations. Moreover, as with componentiality, CMAs are typically applied to evaluate the plausibility of the simulated emotion rather than validate the implemented algorithms.

1.3.3 Challenge 3: Time dependence

The third computational challenge (C3) concerns time dependence. Emotions are typically considered as time-varying and transient phenomena, hence the term emotion “episode”. Temporal parameters of emotion episodes include, for example, onset, dynamics, and offset, as well as the subjective perception of those parameters (e.g., felt duration; Kuppens & Verduyn, 2010). The study of these parameters has been referred to as “affective chronometry” (Davidson, 1998; 2015). Appraisal theories have been the first in addressing affective chronometry, especially the theory of Scherer (1984; 2001, 2009a), which applies dynamic systems theory to connect hypotheses of feedback (H3), synchronization (H4), and feeling integration (H5; see previous sections). Scherer (2000, 2009a) considers emotion episodes as attractors, that is, recurring sequences of component patterning that the five components of emotion will tend to evolve to, given a certain trigger (e.g., appraisal). Such attractors are a consequence of the mutual constraints that component subsystems impose on each other, including feedforward and feedback relations (Lewis, 2005; Sander et al., 2005). Scherer (2000, 2009a) assumes that emotional attractors are characterized by increased component synchronization and differentiated patterning, and that the degree of synchronization determines the emergence of

Computational modeling of appraisal theory of emotion

Thesis

Reference

Computational modeling of appraisal theory of emotion

COMPUTATIONAL MODELING OF APPRAISAL THEORY OF EMOTION

T

Table of Contents

1 General introduction