Introduction and summary of the thesis

1 General introduction

1.1 Introduction and summary of the thesis

Emotions are complex phenomena that involve potentially the entire organism, from subtle mental changes (e.g., a focussing of attention) to overt behavior (e.g., aggressive behavior). Such patterns of reactions stand out not only by their globality (Freeman, 2005) but by their subjective feeling that what goes on is cohesive and qualitatively distinct (Scherer, 2004). Hence, emotions are often perceived of as categorical phenomena and denoted by labels such as anger, fear, or joy. An important challenge in affective science is to account precisely for how distributed mental and bodily changes integrate into a unified emotional experience. A theoretical framework that has been foremost in addressing this challenge is appraisal theory. Appraisal theory represents a collective of individual appraisal theories (e.g., Arnold, 1960; Frijda, 1986; Lazarus, 1991; Ortony, Clore, & Collins, 1988;

Roseman, 2001; Scherer, 1984) that all share as their central tenet the idea that the link between events and emotional responding is causally mediated by a process called appraisal. The appraisal process is defined as a subjective evaluation of importance with respect to personal goals and desires. In other words, appraisal theory assumes that a situation provokes an emotion only when that situation is appraised as being personally relevant.

When compared to other theoretical frameworks of emotion, appraisal theories have been relatively detailed in describing emotion processes. Models of appraisal have attempted not only to explain emotion causation but have also attempted to specify the contents of emotional responding (e.g., which mental and bodily changes should be involved), the relation between appraisal and emotion categories (e.g., how anger and fear patterns can be differentiated), and various process aspects about emotion involving temporal dynamics and sequencing. This level of detail has made appraisal theory an appealing framework for empirical modeling, which has been a part of experimental, observational, and simulation-based research. At the same time, empirical modeling of appraisal theory has faced computational challenges that have deterred the testing of important hypotheses. In this thesis I identified three such challenges and five such hypotheses (Figure 1.1 and Table 1.1). The challenges are (C1) the high dimensionality of the set of mental and bodily changes to be modeled, (C2) the possibility of nonlinear associations, and (C3) the need to incorporate time.

With respect to challenge C1, many appraisal theories have subscribed to the view that emotions are componential. This term refers to the idea that the mental and bodily changes of an emotion episode can be grouped into components that correspond to major organismic subsystems. Frequently cited components of emotion are appraisal, motivation, physiology, expression, and feeling. Each of these components is assumed to encompass numerous subcomponents, for example, changes in heart rate, breathing rate, or skin conductance, for the physiology component. Collectively, the five

2

components of emotion cover most of the human mind and body and, as such, empirical modeling is tasked with the difficult problem of managing a large number of interdependent subcomponents simultaneously (Wehrle & Scherer, 2001). From a statistical point of view, the set of variables would be called high dimensional. This problem is relevant in particular for hypotheses that involve all components of emotion simultaneously, such as the hypothesis of emotional synchronization (H4) and the hypothesis of feeling integration (H5). The synchronization hypothesis claims—concisely stated—

that components of emotion should be more coherently organized during an emotion episode than during a non-emotion episode (e.g., Freeman, 2005; Gross, 2010; Scherer, 2001; 2009a). The feeling integration hypothesis states that the contents of the feeling component of emotion represent an integrated awareness of changes in the other components (Grandjean, Sander, & Scherer, 2008;

Moors, 2009; Scherer, 2004).

Figure 1.1. Schematic overview of computational challenges in modeling emotion and its related hypotheses. Absent arrows indicate that testing the hypothesis in question does not (necessarily) depend on the computational challenge to be solved.

With respect to challenge C2, many appraisal theorists have hypothesized that associations between appraisal and other emotion components are nonlinear. This hypothesis has been expressed in three prominent areas, which are (H1) interaction effects of appraisals on other emotion components (Ortony et al., 1988; Lazarus, 2001; Roseman, 2001; Scherer, 2001), (H2) curvilinear associations between appraisal criteria and other emotion components (Kappas, 2001; Tong, Ellsworth & Bishop, 2009), and (H3) time-dependent feedback processes between emotion components (Lewis, 2005;

Scherer, 2009a). Such hypotheses demand further complexity of empirical models of emotion and necessitate the use of nonlinear methods, which have not been widely used in affective science (Lewis,

3

2005). As a consequence, nonlinear hypotheses have not been researched systematically in empirical literature, despite being a core aspect of many appraisal theories.

Table 1.1. Overview of computational challenges and hypotheses.

Hypotheses of appraisal theories and related computational challenges

Hypothesis H1 Appraisal

interactions

Major theories of appraisal assume that appraisal criteria interact to elicit and differentiate responding in the other emotion components (Ortony et al., 1988; Lazarus, 2001; Roseman, 2001; Scherer, 2001).

H2 Curvilinear associations

Several appraisal theorists assume that there should be curvilinear associations between the appraisal component and other components of emotion, such as sigmoid or polynomial associations (Kappas, 2001; Tong & Tay, 2011; Tong et al., 2005; 2009).

H3 Component feedback

Temporal theories of emotion assume that components of emotion cross-influence each other over time, through feedforward and feedback relations (Lewis, 2005; Scherer, 2001, 2009a).

H4 Component synchronization

Multiple theories of emotion assume that components of emotion are organised during an emotion episode in a manner that is more coherent than during non-emotion episodes (e.g., Bulteel et al., 2013; Ekman, 1972, 1992; Freeman, 2005; Gross, 2010; Lazarus, 1991; Levenson, 1994, 2003; Matsumoto, Nezlek, & Koopmann, 2007; Scherer, 1984; 2001; 2009a; Tomkins, 1962)

H5 Feeling integration

The feeling component of emotion is often assumed to be an integrated awareness of some or all changes in the other emotion components (Moors 2009; 2013; Scherer, 2000; 2004; 2009a).

Challenge C1

High-dimensional features

Componential definitions of emotion claim that emotions consist of components (i.e., appraisal, motivation, physiology, expression, feeling; Frijda, 2008; Frijda & Scherer, 2009; Moors, 2009;

Scherer, 2005). These components consist of numerous subcomponents and, collectively, represent a substantial part of the mind and body. Computationally, this implies high-dimensional datasets of emotion features and requires multivariate models for analyzing emotion data.

C2 Nonlinear associations

Appraisal theories have posited that the relation between the appraisal component and the other emotion components are nonlinear (Kappas, 2001; Lazarus, 2001; Lewis, 2005; Ortony et al., 1988; Roseman, 2001; Scherer, 2000; 2001; 2009a). Computationally, this requires nonlinear models for analyzing emotion data.

C3 Time-dependence

Emotion theorists assume that emotions are time-varying phenomena, or episodes, with temporal properties such as onset, dynamics, and offset (Davidson, 1998; 2015; Kuppens &

Verduyn, 2010). Computationally, this requires emotion data to be modeled over time (e.g., in time-series models).

With respect to challenge C3, emotions are widely considered to be time-varying phenomena that are characterized by temporal properties such as onset, dynamics, offset, and duration (Davidson, 1998; 2015; Kuppens & Verduyn, 2010). Hypotheses have been formulated about all these properties, with some authors appealing to mathematical frameworks such as dynamic systems theory to explain

4

emotion unfolding (Lewis & Granic, 2000). In particular, the dynamic systems approach has been used to elaborate on the synchronization hypothesis (H4) and the feeling integration hypothesis (H5).

Scherer (2000; 2001; 2009a;b) has proposed that emotional synchronization is a product of the time-dynamic interaction between appraisal, motivation, physiology, and expression, and that the feeling component is emergent upon the degree of synchronization. However, time-dependent modeling of emotion processes has emerged only recently as a noteworthy branch of empirical research.

Traditional approaches to modeling appraisal theory have difficulty coping with the three challenges that I listed (Figure 1.1). These traditional approaches include classic computational models (Marsella, Gratch, & Petta, 2010), which directly operationalize theoretical appraisal models as computer programs, and linear statistical models, which estimate hypothesized relations from empirical data (e.g., linear regression, linear discriminant analysis). For this thesis, I adopted a hybrid computational approach that integrates the two traditional approaches by combining structural assumptions of appraisal theories with nonlinear statistical methods from the field of machine learning. Machine learning offers a wide range of models (e.g., artificial neural networks, support vector machines) that are capable of extracting complex patterns from empirical data, including data that are high dimensional or time varying (see Bishop, 2006, Hastie, Tibshirani, & Friedman, 2009, and Ripley, 1996, for overviews). These methods are well-suited to the computational demands of emotion theories—such as appraisal theories—and for testing complex hypotheses such as those given in Figure 1.1 and Table 1.1.

The primary goal of this thesis was to use machine learning to deal with the three computational challenges I identified and to test empirically the five hypotheses of appraisal theories that are affected by these challenges. To achieve this, I conducted four major studies with large datasets. In these studies, I modeled appraisal theory with an increasing degree of complexity, concluding with a full operationalization of an appraisal theory that incorporated all components of emotion, nonlinear processes, and time in a single statistical model.

This thesis is organized as follows. The remainder of this chapter is first dedicated to discussing the theoretical background of the studies in detail. In those sections, I elaborate further upon the issues that I raised in this introduction (e.g., definition of emotion, appraisal theory), I give an overview of computational challenges for modeling appraisal theory and I present the solution to these challenges for this thesis. Chapters 2 through 5 cover the empirical studies that I conducted for the thesis.

Chapters 2 and 3 focus on the modeling of interaction effects in appraisal theory for large databases of recalled emotion experiences (Chapter 2) and of cross-cultural emotion knowledge in all components of emotion (Chapter 3). Chapters 4 and 5 focus on the development and application of a practical computational model for emotion simulation, called Emergent Liquid State Affect (ELSA). This model operationalizes a concrete appraisal theory to allow the integration of time-varying data on all components of emotion and the analysis of emotional synchronization. Chapter 4 provides the theoretical background of the model and presents two data simulations, one using synthetically

5

generated data and one using real data from an appraisal experiment that manipulated appraisal criteria and measured electro-encephalogram (EEG) and electro-myogram (EMG) activity. Chapter 5 applies the ELSA model to a new study that was tailored to collect time-varying data on all emotion components simultaneously, in response to manipulated appraisal criteria. Chapter 6, finally, is devoted to a general discussion of the research findings presented in the four empirical studies.