patterns illustrations and comparison by clusters

In this section, we will explore to what extent the t-patterns detected by THEME are specifically related to the rating categories produced by the judges. Each t-pattern that is unique to one cluster may be considered either as an element that was used by the judges to score the videos or alternatively an element that co-occurs non-randomly with a latent variable that was not measured in this thesis and which does affect the categorization process (ex. Speech content).

We will also examine how additional nonverbal and paraverbal participate to the structure of t-patterns containing facial action units. When possible, we will discuss possible functional interpretations of the detected patterns. To do this, we will often refer to the predictions of Ekman as well as those of Scherer. Note, however that this is not done in an attempt to test the respective predictive values of these models. Rather In the discussion section of this thesis we will suggest possible methodologies that could be used to empirically test the specific meaning of the t-patterns found in this thesis in a more rigorous manner.

To illustrate t-patterns that are specific to each rating cluster, we generated FACSGen simulations that depict the unfolding of expressive actions in time. For readability reasons we will not discuss each instances of cluster specific t-patterns. However, the exhaustive list of all t-patterns detected in the clusters, as well as their frequency of occurrence expressed in percentages, is illustrated in the form of transition graphs available in appendix 3. Note, that by default, we added one “neutral baseline” slide, before the first event in the patterns. On this baseline slide, the subject is represented as if the head and gaze were oriented straight at the camera and no facial actions were produced. Remember also, that in order to look at the interviewer, situated at a 45° angle to the right of the subject; she needs to either move her eyes or head to the right. Unless an “offset” code appears in the pattern, the actions are maintained and codes accumulate on successive slides. This may, or not be an accurate description of what would visually occur depending on whether or not an “offset” score presents the required characteristics to be included as an event in a t-pattern.

Our discussion of the t-patterns will be structured by presenting the specificities of each cluster one by one.

108 Enjoyment cluster 

The enjoyment cluster includes 14 video samples rated high on the joy, amused and entertained adjectives. 24 distinct t-patterns were considered in the following exploration. Out of the 70 behavior units defined in the original annotation scheme, we only analyzed the 47 actions for which adequate inter-rater agreement was reach. Out of these 47 event types, 13 (28%) appear in the composition of t-patterns that have been rated as positive. They include three upper face action units: AU1+2 (combined), AU5, AU6 and five lower face action units:

AU10, AU12, AU14, AU17 and AU20. Additional non FACS codes include: participant looking at the interviewer (look at), participant looking away form the interviewer (look away), the head being turned to the side and away from the interviewer (head turned away), blinking (blink) and the participant starting to speak (speak). FACS action units that are never part of a t-pattern in sample files rated as positive include: AU1, AU4, AU7, AU9, AU10U, AU12U, AU14U, AU15, AU16, AU23, AU24, AU25, and AU26. Figure 27, shows the proportion of codes that initiate a t-pattern in decreasing order. The most pervasive kind of patterns (34%) in the “enjoyment” cluster is initiated by the participant starting to look at the interviewer.

Look At AU5 AU12 AU1+2 Look Away AU6 Speak AU14 AU20

Codes

Percentage

Figure 27 "Enjoyment" Cluster. First codes in T-patterns

This kind of opening seems most characteristic of this group. Neither the “Hostility”

nor the “Embarrassment” clusters contain patterns that start with a participant looking at the interviewer. We do find such patterns in the “Surprise” and “Sadness” clusters, but they are marginally represented in these groups, 3% and 2% respectively. In the positive emotion group, the majority (76%) of actions that follow the “look at” event include, either a social smile, (AU12 -54%) or a Duchenne-smile, (AU6+12 -22%). No AUs 6 or 12 are found in the other clusters where patterns start with a “look at” event.

109 Figure 28.

Neurtral

Baseline Look At AU12 AU5

Figure 29.

AU12 Neutral

Baseline        

Look At        

AU5        

AU1+2    

Blink    

110 Figure 30

Neutral

baseline AU6 AU1+2 AU12

AU6 stops

Look At

All these patterns present the characteristic of combining a social or a D-smile (a.k.a.

Duchenne smile) with a bilateral eyebrow raise, an upper lid raise or both. These two latter actions are part of the “surprise” prototype in Ekman’s terminology. Partial sequences of t-patterns including action unit 12 with either 1+2 or 5 are also found in the "Hostility" (5%) and "Embarrassment" (12%) clusters. However the combination of action unit 6+12 with 1+2 and/or 5 is unique to the positive emotions cluster. Moreover, in the other clusters the "look at" action is never part of these sequences. Ekman finds that happiness often blends with surprise (Ekman and Friesen, 2003 p.107). The typical eliciting scenario in his case would be one where something unexpected occurs and the evaluation of it is favorable. The appraisal theory interpretation of these sequenced actions would be quite comparable. AU1+2+5 would be interpreted as signaling an evaluation of suddenness on the novelty dimension. The AU6+12 actions could either signal an intrinsically pleasant or goal conducive situation. Note that in patterns 1 and 3; AU5 or AU1+2 but not both; occur in combination with 6+12 or 12 alone. In this case the appraisal model provides no interpretation. Also, in these two patterns the predicted, a) suddenness, b) intrinsic pleasantness or c) goal conduciveness sequence would not be respected as AU5 and 1+2 occur after and not before AU6 or AU12 have been activated. An additional sequence covering 6% of all the t-patterns in the PE group includes a

111

"look at" action combined with an AU12. In this case, the subject starts to speak, looks at the interviewer and finally produces a social smile (AU12). Not all the patterns including smiling actions start with the subject looking at the interviewer. In fact, 11% of all the patterns in the PE group are initiated with AU12. The most prevalent sequence in this case (51%) combines action unit 12 with AU17 as illustrated in figure 31. Other less complex sequences including AU12 the combination of 12 with 6, in a classic Duchenne smile (27%). We also find a simple two events sequence starting and ending with 12 (22%).

Figure 31

AU12 AU17 Stop AU6

AU6 (StopAU17)

Stop AU12 Neutral

Baseline

The enjoyment group is not devoid of action units that are traditionally associated with negatively valenced emotions. AU10, typically associated with "disgust" or "anger", is found in 9% of the patterns. In the appraisal framework, AU10 is thought to signal an unpleasant evaluation of olfactory or gustatory stimuli. Recent work in the field of embodied cognition has shown that “disgust” feelings induced by unpleasant odors do affect the severity of moral judgments that research participants’ are asked to produce (Schnall and al, 2008). This could be an illustration of how a facial action that was selected phylogenetically to reduce the inflow of odors potentially harmful to an organism, could through the development of symbolic thinking and language skills acquisition, become a nonverbal emblem conveying a symbolic repulsion that could be roughly interpreted as "This situation, action or person

112 stinks". Also, AU20 which is the main lower face element of the "fear" prototype is observed in 5% of the patterns. In opposition to most of the "smiling" patterns, these actions are systematically preceded by the subject either gazing or turning her head away from the interviewer. In no other cluster are AU10 or AU20 directly preceded by a “look away” action.

Note also, that AU12 is never aligned with AU10 or AU20 in a t-pattern. AU20 is believed to reflect a “low power” appraisal on the CPM “coping” dimension. It is as if the subject did not want the interviewer to see or think that these displays were addressed to him.

Hostility Cluster 

The hostility cluster includes 54 video samples in which 227 t-patterns corresponding tour inclusion criteria were detected. In this group 62% (N=29) of the annotation codes enter in the composition of at least one t-pattern. Because the raters did not make a differentiated use of the “angry”, "contempt” and “disgusted” adjective scales, we could expect to find t-patterns that exemplify any of these three facial prototypes in this cluster. Graphic 1 shows the event types initiating t-patterns in the “Hostility” group. Event types are ranked by order of importance.

Figure 32. Hostility Cluster. First codes in T-patterns (%) 25

AU4 AU7 AU9 AU17 AU1+2 AU5 AU15 AU10U AU14U AU20 AU10 AU12 AU14 AU12A AU23 AU24

Codes

Most of the EMFACS action units are found to initiate t-patterns in this cluster. One notable exception is the Duchenne marker of “enjoyment” smiles, action unit 6 (Cheek raiser).

The most prevalent event initiating a t-pattern in this group is AU4 (25%) which lowers and draws the brows together. After AU4, the next most frequent event types starting a t-pattern are AU7 (15%) and AU9 (10%). Those three actions together (4, 7 and 9) account for 50% of all the t-patterns in the videos rated as communicating some form of hostile demeanor.

113 According to the EMFACS taxonomy, AU4 and AU7 are constitutive elements of “anger”

prototypes, while AU9 signals “disgust”. The preeminence of these event types in the composition of “hostile” patterns is therefore no breaking news. If one adds AU10 as an alternative to the encoding of “disgust”; AU10U and 14U as signs of “contempt” and AU23 and AU24 as possible elements in “anger” displays, the proportion of t-patterns starting with an event type constitutive of facial prototypes conveying some form of hostility rises to 63%.

AU4 is the first action in 25% of the patterns present in the sample files rated as conveying a hostile attitude. According to Ekman, for an AU4 to be interpreted as a clear

“anger” signal, it must be associated with AU24 (lips pressing against each others) possibly also with AU23 (Lip Tightener), both in the lower face region (Ekman and Friesen, 2003, p83). If that is not the case, the expression is considered ambiguous and might take on other meanings: visual focusing efforts, concentration, determination or anger containment. In the cluster, we find no lower face action units aligned with AU4 in a-t-pattern. One possible explanation for this is that AU23 and AU24 are by definition only coded when subjects are not speaking. Because we find no “pause” codes in these patterns it is most likely that AU4 is produced when the subjects are speaking. This would preclude any association of AU4 with 23 or 24. Note also, that according to Ekman, the involvement of AU23 and 24 in anger expressions ought to be interpreted as attempts to control an impulse of saying something hostile or shouting. This implies that the decoding of “angry” faces would have to rely on converging signals from additional communicative channels to be disambiguated when subjects are speaking (ex: speech content, vocal acoustics modification). For Scherer also, action unit 4 should not be interpreted in isolation. He provides predictions for AU4+5 and AU4+7 when they overlap (both AUs together being an element in a sequence). The interpretation of the AU4+7 combinations varies depending on its position in the appraisal sequence. When it occurs at the beginning of a pattern it would signal a low familiarity or the occurrence of an event difficult to predict (novelty appraisal check). If these actions are found latter in the sequence they could signal that some situation is being evaluated as intrinsically unpleasant or goal obstructive. The association of AU4 with 5 is thought to convey an attitude of feeling powerful enough and willing, to cope with a challenging situation. In our dataset, the action units associated with AU4 in a t-pattern are: AU9 (63%), AU5 (50%), AU7 (41%) and AU1+2 (3%), (the proportions of independent t-patterns including these AUs are in brackets). For Ekman, AU9 is sufficient by itself to signal a “disgust” message. Actually, only the “disgust” and “contempt” prototypes in the EMFACS system can be defined by the

114 innervations of single action units. In this system, a simultaneous activation of AU4 and AU9 would be interpreted as blending some elements of anger with disgust. Our previous results have shown that no such blended expressions could be said to be characteristic of any clusters in particular. Nevertheless, we do find several instances of statistically valid sequences involving both AU9 and AU4 that are unique to the “hostility” cluster. The CPM model proposes two scenarios for such sequences. If AU4 and AU7 manifest before AU9, the sequence would be interpreted as communicating something like: “I see this event as a) new, unfamiliar and or unpredictable and b) it is intrinsically unpleasant.” If AU9 is followed by AU4+5, the message would then become: a) this is intrinsically unpleasant and b) I’m powerful enough to deal with it, and actually, I will do something about it. In the hostility cluster we do indeed find instances of both types of sequences. In fact 45% of all the t-patterns starting with AU4 are compatible with one CPM predicted sequence. Next follows some illustrations of t-patterns initiated by AU4, that are not found outside the group of videos rated as conveying hostility. As before, the complete set of transition diagrams illustrating all the sequences are in the appendix 3. Even though AU7 is most predominant in the constitution of patterns belonging to the hostility group (23%), it is also found in the t-patterns of videos rated as communicating embarrassment (19%) and to a much lesser extent in the t-patterns of the surprise and sadness groups; 4% and 2% respectively. Besides these quantitative differences, we also find structural variations in the composition of t-patterns with AU7 across these four clusters. In the "Hostility" group, AU7 appears in 84% of the t-patterns where AU4 is present. By contrast, in the "Embarrassment" and "Surprise" clusters no such association appears. We do find two short sequences in the "Sadness" cluster where AU4 is aligned in a t-pattern with AU4, with no other additional codes. By contrast, when AU4 and AU7 are associated in a t-pattern in the "Hostility" cluster they are always accompanied with some additional FACS codes. For example, in the illustration below (Hostility: T-pattern 1), AU9 is aligned in a t-pattern including both AU4 and AU7.

115 Figure 33.

Neutral

baseline AU7 AU9

Stop AU4 AU7

This specific t-pattern of length 4 repeats itself 19 times in the hostile group. By itself, it represents 5% of all the sequences starting with AU4. Another sequence covering 6% of the AU4 patterns, presents itself as follows:

Figure 34.

Baseline

(AU1+2) AU4 AU9

Stop AU1+2 Stop

AU9

This t-pattern represents 6% of the sequences that start with an action 4 in the hostility cluster. Before the pattern starts, two action units are already activated: AU1+2. According to Ekman, the combination of the frontalis and corrugator actions produce a typical fear brow.

AU1+2+4, is one of the few EMFACS combination that is statistically more represented in some clusters than others. Event though this facial configuration is present in the hostility group, it was found to be more characteristic of the sadness and embarrassment clusters.

Interestingly, the association of AU1+2 with AU4 is never aligned in a T-pattern in either the sadness or the embarrassment cluster. This is a good illustration of how quantitative differences detected across the clusters are not automatically reflected in the structural composition of the t-patterns detected by THEME. The results of the structural analysis

116 suggests that 1+2+4 is part of a repetitive sequence found only in video samples rated as

“hostile”, which additionally involves AU9 and where the activation of AU4 is maintained after AU1+2 has receded. In this case, this suggests that AU1+2+4 could be seen not as a discrete facial signal but as a momentary configuration in a sequence of intertwining events where AU1+2 and AU4 converge at times, but nevertheless follow distinct non random dynamic trajectories. We do find some t-patterns starting with AU4 in the “sadness” (2%) and

“Embarrassment” (1%) cluster that are also found in the “Hostility” group. In the

“Embarrassment” group AU4 is aligned with AU5 in one t-pattern. In the sadness group AU4 is associated with AU7 in two short patterns. But, in both those cases no other EMFACS codes are associated to these sequences. Additionally, neither in the “Positive emotions” nor the “Surprise” cluster, does the program detect any t-patterns involving AU4. The inner and outer brow raise action (AU1+2) in the hostility cluster initiate 7% of all the t-patterns. All of them are directly followed by an action unit 5. According to Ekman, the association of AU1+2 with AU5 is a combination constitutive of both the “fear” and “surprise” prototypical expressions. Scherer associates AU1+2+5 with possibly two appraisal dimensions: “novelty”

and “goal significance”. When these action units occur together, they are interpreted either as a reaction to a sudden change in the person’s external or internal environment (novelty); or alternatively as the expression of a perceived discrepancy between what is happening and what was expected (goal significance). In 51% of the t-patterns starting with the sequence AU1+2 +5, the next event in the sequence is an AU12. T-pattern 3 is a good illustration of such a case. By itself it represents 16% of the patterns starting with the following sequence a) AU1+2, b) AU5, and c) AU12.

117 Figure 35.

Stop

AU20 Stop

AU5 Neutral

baseline AU1+2 AU5 AU12 AU20

Interestingly, none of the facial actions that are most typically associated with expressions of disgust, contempt or anger are included in the composition of this pattern identified in sample files rated as conveying some form of hostile attitude. AU5 could be an element of an anger prototype, if it were associated with AU4, AU7 or both, which is not the case here. Ekman’s interpretative framework would lean towards an explanation involving some element of surprise. To this initial sequence a smiling action (AU12) is added, directly followed by a lip stretch (AU20). The AU20 is seen as an element of a “fear” expression in Ekman’s dictionary and Scherer interprets its message value as conveying something like: “I have little or no power”. Ekman would probably argue that the smile should not be seen as a true signal of enjoyment, for lack of AU6 involvement, but rather as a deceiving attempt to mask a “fear” expression. In this case both frameworks would seemingly fail to explain why the sequence is perceived as communicating hostility. What strikes us in this sequence is the fact that the frontalis action is maintained throughout the sequence. According to Ekman, surprise is the briefest emotion, and longer displays of surprise are seen by him as reliable

118 cues to an “unfelt emotion” (Ekman, 2003, p.165). In this case the sequence would be interpreted as a voluntary emblem communicating attitudes related to surprise like disbelief and amazement. The CPM model proposes an alternative explanation to the longer than would be expected duration for the AU1+2+5 activation. Once the initial processing of a novel stimulus ends, the goal relevance appraisal starts. Interestingly, the facial signs predicted for an event evaluated as “relevant” is precisely the maintained innervations’ of

118 cues to an “unfelt emotion” (Ekman, 2003, p.165). In this case the sequence would be interpreted as a voluntary emblem communicating attitudes related to surprise like disbelief and amazement. The CPM model proposes an alternative explanation to the longer than would be expected duration for the AU1+2+5 activation. Once the initial processing of a novel stimulus ends, the goal relevance appraisal starts. Interestingly, the facial signs predicted for an event evaluated as “relevant” is precisely the maintained innervations’ of