The perception of emotional facial expressions in normals and in schizophrenic patients

Thesis

Reference

The perception of emotional facial expressions in normals and in schizophrenic patients

CASATI, Paola

Abstract

The research compares the perception of facial expressions (FE) of emotions in controls and schizophrenic patients. Three studies are presented. The first and the second study use a new morphing technique to test whether Categorical Perception (CP) exists for static displays of FE. The joint analysis of an identification (first study) and a discrimination task using morphing transitions between Neuter-Happiness and Neuter-Anger expressions demonstrate the presence of CP only for Anger in controls. CP does not occur in first-episode schizophrenia patients. The third study tests the identification of the six basic emotions in dynamic conditions. The results of a recognition task of FE from video clips showing the transformation between a neutral face and an emotional face demonstrates that schizophrenia patients need more time and a more intense facial activity than controls to detect basic emotions. It also shows that patients show selective impairments in the identification of Fear and Disgust.

CASATI, Paola. The perception of emotional facial expressions in normals and in schizophrenic patients. Thèse de doctorat : Univ. Genève, 2010, no. FPSE 453

URN : urn:nbn:ch:unige-96202

DOI : 10.13097/archive-ouverte/unige:9620

Available at:

http://archive-ouverte.unige.ch/unige:9620

Disclaimer: layout of this document may differ from the published version.

1 / 1

1

FACULTE DE PSYCHOLOGIE ET DES SCIENCES DE L’EDUCATION

SECTION DE PSYCHOLOGIE 40, Boulevard du Pont d’Arve CH-1205 Genève

Sous la direction du professeur Paolo Viviani

et la co-direction du professeur Susanne Kaiser

THE PERCEPTION OF EMOTIONAL FA- CIAL EXPRESSIONS IN NORMALS AND IN

SCHIZOPHRENIC PATIENTS

(LA PERCEPTION DES EXPRESSIONS FACIALES DES EMOTIONS CHEZ DES SUJETS NORMAUX ET DES PATIENTS SCHIZOPHRENES)

T H E S E

453

PRESENTEE A LA FACULTE DE PSYCHOLOGIE ET DES SCIENCE DE L’EDUCATION DE L’UNIVERSITE DE GENEVE POUR L’OBTENTION DU GRADE

DE DOCTEUR EN PSYCHOLOGIE PAR

Paola CASATI

2

REMERCIEMENTS

Je remercie en premier lieu le professeur Paolo Viviani, mon directeur de thèse, qui a di- rigé mon apprentissage dans les domaines de la Psychologie Expérimentale et de la Statisti- que; ses enseignements, sa patiente et son professionnalisme ont permis la consolidation et la croissance des mes connaissances et capacités.

Je remercie les professeurs Susanne Kaiser et Marco Battaglia, et le docteur Maryse Ba- dan, de l’honneur qu’ils me font en acceptant de faire partie du Jury de thèse.

Je voudrais exprimer toute ma reconnaissance au docteur Maryse Badan pour sa dispo- nibilité et pour ses précieuses suggestions dans le domaine clinique. Merci aux Hôpitaux Uni- versitaires de Genève (HUG, Programme Jade) et à la “Comunità Riabilitativa Alta Assisten- za” (CRA) de Romano in Lombardia (docteurs Maria Rosa Castelli, Giuseppe Primerano et Alessandra Mombrini), sans les quels ce travail n’aurait pas été possible. Merci aux patients psychotiques et schizophrènes du HUG et CRA qui ont passé les tests.

Je remercie le docteur Susanne Schmidt de l’Université de Turin qui a permis, avec son importante contribution, l’analyse FACS de la troisième expérience.

Toute ma gratitude va à mes collègues, les docteurs Christelle Aymoz et Chiara Fioren- tini, pour leurs suggestions et pour leur exemple d’assiduité et d’excellence.

Enfin, je remercie ma mère, mon mari et ma soeur pour leur affection et soutien.

3

A mes parents, Antonietta et Walter

4

TABLE OF CONTENT

INTRODUCTION 9

CHAPTER I : THEORETICAL BACKGROUND 10

1.

1.1 THE EMOTIONS: HISTORICAL BACKGROUND 10

1.2 THE THEORIES ABOUT FACIAL EXPRESSIONS OF EMOTIONS 12

1.2.1

1.2.2 Tomkins’ ‘Innate Affect Program’ 13

1.2.3 The Facial Affect Program 14

1.2.4 The Facial Action Coding System 15

1.2.5 The Basic Emotion Theory (BET) 16

1.2.6 The Dimensional-Contextual Theory 17

1.2.7 The Componential Appraisal Theory 18

2. PERCEPTION OF EMOTIONS 20

2.1 THE PERCEPTION OF FACIAL EXRESSIONS OF EMOTIONS 20

2.1.1 The Direct Perception Theory 20

2.1.2 The Dimensional-Contextual Theory 21

2.1.3 The Motor Theory 23

2.2 THE CATEGORICAL PERCEPTION PHENOMENON 26

2.2.2 CP Evidences for Faces 29

3. SCHIZOPHRENIA AND EMOTIONS 34

3.1 SCHIZOPHRENIA: THE HISTORICAL BACKGROUND 34

3.2 DIAGNOSTIC CRITERIA FOR SCHIZOPHRENIA AND FOR THE

SCHIZOAFFECTIVE DISORDER 36

3.2.1 Diagnostic Criteria for Schizophrenia 36

5

3.2.2 Diagnostic Criteria for the Schizoaffective Disorder 36

3.3 THE SYMPTOMS OF SCHIZOPHRENIA 37

3.4 EMOTIONAL DEFICIENCIES IN SCHIZOPHRENIA 38

3.4.1 The Flattening Effect 38

3.4.2 Deficits in the Perception of Facial Expressions of Emotions 39 3.4.3 Emotional Perceptive Impairments and Subtypes of Schizophrenia 41

3.5 SCHIZOPHRENIA AND NEUROANATOMY 41

3.6 SCHIZOPHRENIC DEFICIT IN EMOTION PERCEPTION FROM FACES: IS IT

AN EMOTIONAL OR A COGNITIVE DISORDER? 42

3.6.1 Delimited Emotional Deficit: Theories and Neuronal-Biological Evidences 43 3.6.2 The Cognitive Deficit: Theories and Neuronal-Biological Evidences 46

CHAPTER II: THE PRESENT STUDY: MATTERS AND AIMS 51

CHAPTER III: EXPERIMENTS 1 AND 2 53

1. INTRODUCTION 53

2. EXPERIMENT 1 : THE STATIC IDENTIFICATION TASK 54

2.1 METHODS 54

2.1.1 Participants 54

2.1.2 Stimuli 55

2.1.3 Morphing Algorithm 56

2.1.4 Procedures 58

2.1.5 Data analysis and Modelling 59

2.2 RESULTS 61

2.2.1 The Identification Task 62

2.2.2 The Questionnaire 73

2.2.3 Emotion Identification, Psychotic Symptoms and Cognition 74

2.3 CONCLUSION 77

6

3.

3.1 METHODS 79

3.1.1 Participants 79

3.1.2 Stimuli 79

3.1.3 Procedures 79

3.1.4 Data Analysis and Modelling 80

3.2 RESULTS 81

3.2.1 The Discrimination Task 81

3.2.2. Emotion Discrimination, Psychotic Symptoms and Cognition 86

3.3 CONCLUSION 88

CHAPTER IV: EXPERIMENT 3: THE DYNAMIC IDENTIFICATION TASK 89

1. INTRODUCTION 89

2. METHODS

2.1 PARTICIPANTS 90

2.2 STIMULI 91

2.2.1 Generation of the Stimuli 91

2.2.2 Facial Coding 93

2.2.3 Facial Expression Analysis 94

2.3 PROCEDURES 100

3. RESULTS

3.1 COMPARING STIMULI WITH FAPS AND CPM PREDICTIONS 102

3.1.1 Comparing stimuli with FAPS predictions 102

3.1.2 Comparing stimuli with CPM predictions 104

3.2 THE EMOTION IDENTIFICATION TASK 108

3.2.1 Response Probabilities 108

3.2.2 Response Times 113

3.2.2 The Identification Performance 117

7

4. CONCLUSION

CHAPTER V: GENERAL DISCUSSION

1. THE STATIC TASK

1.1 PRINCIPAL RESULTS 138

1.2 POSITIVE vs NEGATIVE EMOTIONS: EVIDENCES CONCERNING HAPPINESS

AND ANGER PERCEPTION 140

1.3 CATEGORICAL PERCEPTION IN CONTROLS AND EARLY EPISODE

SCHIZOPHRENIA PATIENTS: AN INTERPRETATION 142

1.4 COGNTIVE COMPONENTS OF THE STATIC PERCEPTION TASK 144

2. THE DYNAMIC TASK

2.1 PRINCIPAL RESULTS 147

2.2 THE MOTOR THEORY 148

2.2.1 The Resonance Theory 149

2.2.2 The Cognitive Mediation of Emotional Expressions Perception 150 2.3 NEUROPSYCHOLOGICAL SYSTEMS FOR EMOTION PERCEPTION IN

HEALTHY AND SCHIZIOHRENIA SUBJECTS 151

2.4 EVIDENCES ABOUT FEAR AND DISGUST PERCEPTION 154

3. CONCLUSION AND PERSPECTIVES

REFERENCES

8

ANNEXES

STATIC TASKS: Diagnosis of the Clinic group 182

STATIC IDENTIFICATION TASK: The Questionnaire 192

DYNAMIC TASK: Diagnosis and Medical Treatment of the Clinic group 193

DYNAMIC TASK: Analysis Plots 194

RESUME EN FRANCAIS 303

9

INTRODUCTION

The subject of this study lies on the borderline between two specific fields of research;

the attempt to link them constitutes the innovative element of this work. The first area relates to the study of emotion perception of facial expressions and - in particular - to the hypothesis of categorical perception (CP) of universal expressions (happiness, sadness, anger, fear, sur- prise and disgust). The second field of study considers the emotional sphere of psychotic- schizophrenic patients: several experimental studies indicate a massive presence of emotional dysfunctions in schizophrenics, especially in the perception of emotional expressions of faces.

However, perceptive deficits do not affect the recognition of all emotional expressions in the same fashion: negative emotions such as anger, fear and sadness are perceived as being worse than positive ones by clinical subjects. Moreover, it is not known whether the impairment in facial emotion recognition in patients occurs at the outset of mental disorders and whether perception deficits vary with schizophrenic symptoms. Most importantly, we questioned the presence of categorical perception in schizophrenia. To resolve this issue, we applied the psy- chophysical method used to test categorical perception in the clinical field of schizophrenia.

Three experiments are presented in this study. The first trial is an emotion identifica- tion task (neutral-happiness and neutral-anger) which estimates the indifference point (the median) and the JND (just noticeable difference) of the psychometric function representing perceptual choice between emotions. If CP occurs, the median will be the boundary point be- tween two emotional categories while JND will inform about the difficulty of stimuli identifi- cation. CP hypothesis is directly tested by the second experiment which implies a discrimina- tion task between the same emotions of the previous experiment. Our aim is to confirm the presence of CP in controls and, at the same time, to test how emotions are perceived by schizophrenic patients. Finally, the third experiment studies facial expression perception in dynamic conditions. In the first and second experiments the stimuli are static photos repre- senting emotions, whereas in the third experiment the stimuli are video clips showing the transformation between a neutral face and a specific emotional face. The purpose lies in char- acterising - from a psychophysical point of view - the performances of the two groups in a task of emotion identification which occurs in real-life conditions.

10

I. THEORETICAL BACKGROUND

The following theme is composed of two main axes: the first concerns the emotion theories and the second the study of emotions in the clinical context of psychotic disorders and schizophrenia. The main subjects of the first two chapters are emotion theories, facial expressions of emotions and their visual perception as well as psychophysical models of fa- cial emotion perception. In the third chapter, concerning clinical theory, we will explain schizophrenia and schizoaffective disorder. We will outline the most significant symptoms that such mental diseases cause in patients and, in particular, we will examine emotional dys- functions of patients and the manner in which they perceive facial expressions. Our general aim is to compare controls and schizophrenics’ perception of emotions and to highlight the different perceptive mechanisms which exist between the two groups of subjects.

1. EMOTIONS

1.1: EMOTIONS: HISTORICAL BACKGROUND

The term emotion derives from latin, “e + movere”. It means to transfer from one place to another. It also refers to states of agitation or perturbation, in both a physical and psychological sense. However the term emotion has only recently been applied to states of affection. For two thousand years, from the Ancient Greece up to the eighteenth century, peo- ple spoke of emotions as “passions” (from Greek, “pathos”; from Latin, “pati” – to suffer).

The concept being that an individual is undergoing, or suffering, some form of change. Thus, the passivity of emotions connotes a sense of being ‘gripped’ or ‘torn’ by emotions as one were possessed by them. This widespread idea led to a negative consideration of emotions;

they were viewed - until the middle of 19^th century - as being irrational, involuntary and ani- mal-like; beyond self-control. However, emotional reactions are not completely involuntary:

for example, a fearful person seeks to escape and an angry person intends to damage the sub- ject of their anger. Viewed from this perspective the individual who experiencing an emotion is ‘coerced’ by their own desires and aims, being strongly committed thereto, but having little freedom of choice.

11

Since Darwin’s theory of emotions (1873) it has been common to consider emotional reactions as products of man’s evolutionary past and to assign biological roots to emotions. It is known that emotional predispositions are not biologically more primitive than intellectual capacities. In fact, both emotions and intelligence are biological adaptations which achieve the highest levels of complexity in the primates, especially humans (Hebb and Thompson, 1954; Jolly, 1966). A great number of modern theories show that cognitive appraisal is an integral part of emotions for the reason that emotional concepts imply objects: for example, we must be angry about something, afraid of something, proud of something (Kenny, 1963).

Moreover, the relationship between an emotion and its object occurs in delimiting and defin- ing the type of response.

The complexity of the nature of emotions is found in the heterogeneity of modern theories in this field of research. James (1884) and Cannon (1927) concurred in viewing emo- tions as autonomic responses and emotional experiences as the consciousness of this corporal upset, even though they differed in the attribution of the cause-effect relationship between the emotional experience and the behavioural reaction. Later, Schacher (1964) postulated that the type of emotion depends on the cognitive attribution of the cause of corporal upset, in the sense that the way in which events are interpreted causes specific emotional reactions. In a similar way, cognitive emotion theories (Arnold, 1960; Solomon, 1976; Roseman, 1984;

Scherer, 1984; Smith & Ellsworth, 1985; Weiner, 1985; Frijda, 1986) affirm that different cognitive evaluations generate different emotions. Consequently, the evaluation of an event’s components such as pleasantness, incertitude, duration, sense of control and so forth give rise to a specific emotional reaction. Moreover, cognitive evaluation theories (Arnold, 1960; Laza- rus, 1966, 1974, 1978) divide emotions into two main classes, the positive and the negative.

The nature of emotions stems from a cognitive appraisal of the usefulness or the harmfulness for an organism of the interaction between the individual and the environment and the result, which gives rise to a specific emotional state, derives out from the fusion between cognitive, motivational, vegetative and motor components.

Dimensional theory offered a different approach (Wundt, 1913) whereby the emotions are generated by the elaboration of three main dimensions that are central to the organism:

envy/aversion (positive vs negative), excitement/appeasement and tension/relief (active vs passive).

This complex theoretical background form the basis from which the three principal emotion theories that we will analyse thoroughly in the following paragraphs derive. The fol-

12

lowing section is dedicated to the facial expressive behaviour of emotions. Out of all expres- sive behaviours of emotions, facial expressions are considered as being central and have been chosen as an object of study in the majority of research into emotions. Indeed, the face pro- vides fundamental elements for the differentiation and the recognition of specific emotions.

In synthesis, the first of these theories is the Basic Emotion Theory (BET) (Ekman, Izard, Friesen, 1971) which hypothesizes about the universality of some basic emotions such as anger, fear, happiness, surprise, disgust and sadness; each of these emotions is distin- guished by a specific neurophysiologic pattern. The second theory is the Dimensional- Contextual one (Russell, 1997) which explains the emotion recognition process as being the result of two types of information: the physical configuration of stimulus and the level of pleasure and arousal of the expresser. Finally, we will consider the cognitive theory postu- lated by Scherer (1984) for which the emotion perception stems from the evaluation of several sequential psychological elements with which the observer generates an idea about the ex- presser’s mental states.

1.2: THE THEORIES ABOUT FACIAL EXPRESSIONS OF EMOTIONS

1.2.1: Darwin’s theory

Darwin’s theory holds that individuals having advantageous hereditary variations in a specific environment and ethological situation are more likely to survive and to reproduce.

They tend to manifest some, or all, of the following; an augmentation of dimensions, new behaviours, new body shapes, new biological functions etc; all factors which facilitate adap- tion. It therefore follows that organisms which casually vary in these directions tend to prolif- erate more than other organisms of the population. Furthermore the species to which they be- long will contain a high frequency of these genes which determine favourable characteristics.

This process gives rise to the evolution of the species.

Darwin (1872) postulated an evolutionary theory of emotions based on the phyloge- netic origin of expressive behaviours and also on the functional meaning thereof. He offered three general issues for the expression of emotions: 1) the principle of associated useful be- haviours; 2) the antithesis principle and 3) the principle of behaviours determined by the con- stitution of a nervous system. According to his stance, the evolutional approach enables emo- tions to be described as being functional and adaptive elements that are useful for the survival

13

of both the individual and the species. For the individual to survive, its functional systems must be able to produce actions, sometimes rapidly, which facilitate adaptation to interactions with other biological organisms or with the external environment. Thus, for Darwin, emotions would have appeared in the evolutional process as an instrument favouring rapid fulfilment of the needs of organisms when encountering situations of emergency, or changes in internal and/or external parameters.

The author concluded that expressions are universal, innate and genetically transmit- ted. He considered facial expressions as adaptive behaviours and signals to communicate emotional states to members of the same species. Accordingly, the same state of mind is ex- pressed uniformly by humans throughout the world.

The evolutionary theory of emotions postulated by Darwin forms the basis of the modern theories of emotions, which agree with the author about the universality and the ge- netic basis and transmissibility of emotions.

1.2.2 : Tomkins’ ‘Innate Affect Program’

Starting from Darwin’s point of view, Tomkins’ (1962) theory of emotions considered the face as a fundamental source of information; he affirmed the existence of an innate sub- cortical program (called the innate affect program) which created correspondences between the stimuli which generated emotions and the specific patterns of facial, vocal and physiologi- cal responses. The activation of these responses provides feedback, which upon achieving consciousness lead to the feeling affects thereof. Thus, facial expressions were held as being the driving factor which differentiate and cause the subjective experience of emotions, as op- posed to being a mere reflection of an internal state. Tomkins postulated nine fundamental affects and affect auxiliaries, each having a characteristic and a distinct affect program. The affect programs were regarded as being innate, genetically transmitted, and localized in spe- cific sub-cortical structures. When an affect program is activated, it may simultaneously stimulate the facial, body, and heart muscles, as well as autonomic activity and generate a pattern of related and well defined responses; characteristic of specific emotions. Although the number of affects is limited, a great variety of feelings may be experienced because af- fects could blend and other inputs to consciousness could influence the emotional experience.

According to Tomkins, the activation of facial muscles was, on the one hand, the primary

14

cause for emotion differentiation through feedback processes, whereas, on the other, the nec- essary means to communicate these emotional states.

1.2.3: The Facial Affect Program

The Facial Affect Program (FAP; Ekman, 1972) clarified the relationship between the activity of different facial muscles, caused by distinctive patterned sets of neural impulses, and specific emotions.

Considering the previous theories, Ekman and Izard (1971, 1972) demonstrated the existence, previously postulated by Darwin, of a small number of basic emotions common to all humans. The authors emphasized that the primary function of facial expressions is univer- sal. Ekman & Friesen (1971) showed that observers of different cultures agreed on how to label facial expressions in terms of emotional categories. They studied 21 countries including Argentina, Brazil, Chile, China, England, Estonia, Ethiopia, France, Germany, Greece, Italy, Japan, Kirghizistan, Malaysia, Scotland, Sweden, Indonesia, Switzerland, Turkey, the USA and some African countries. In all cases the observers inspected each picture and then selected the most appropriate emotional term from a short list of six to ten items. All the experiments included photographs showing happiness, anger, fear, sadness, disgust, and surprise. They concluded that all the expressions were judged as showing the same emotion by the majority in each country.

Later, several Ekman’s studies examined the link between facial expressions of emo- tion and patterns of emotion-specific physiological correlates. Indeed, “if the association be- tween facial expressions and emotions is in some part given, then it is logical to expect that facial expressions should be related to changes in the physiology of emotion” (Ekman, 1999).

Ekman and Davidson analysed EEG activity while subjects watched emotionally provocative films and found that different patterns of brain activity occurred when disgust or a smile was spontaneously elicited (Davidson, Ekman, Saron, Senulis & Friesen, 1990; Ekman, Davidson

& Friesen, 1990). Ekman (1999) stressed that these differences were consistent with previous findings according to which there were asymmetries in the cerebral activity for negative and positive emotions. In addition to these findings, another set of studies by Ekman and Leven- son showed that different patterns of autonomic nervous system (ANS) activity occurred in conjunction with different facial expressions (Ekman, Levenson & Friesen, 1983; Levenson, Ekman & Friesen, 1990).

15

Ekman and Friesen built a specific model (Facial Action Coding System, FACS; Ek- man & Friesen, 1978) for describing all the possible configurations of the activity of facial muscles that could occur simultaneously in the production of facial expression of emotions.

1.2.4: The Facial Action Coding System

The Facial Action Coding System is an anatomically-based model created for measur- ing the facial movements occurring in expression of emotions. This system categorises all facial activity in terms of 44 distinct Action Units (AUs) and of several head and eyes posi- tion and movements. The numerical codes 1-44 are arbitrary and do not have any particular significance except that 1-7 referred to brows, forehead or eyelids (Ekman, 1976). In order to read facial expressions as events having a muscle activity evolution, FACS encodes the inten- sity of each facial action unit in a scale composed of 5 intensity levels whereby A corresponds to minimal muscle activity and E to a maximal level.

Each AU is labelled with a numeric code and the entire list is laid out in the Table I.1.

To summarize, Ekman affirmed that “it is reasonable to propose that the universals in facial expressions of emotion are the connection between particular facial configurations and specific emotions”. In order to answer the question of the biological basis of the universality of expressions, Ekman postulated the emotion theory that we will analyse in the following paragraph.

16

AU number Description Muscular basis

1 Inner brow raiser Frontalis, Pars medialis

2 Outer brow raiser Frontalis. Pars lateralis

4 Brow lowerer Depressor gabellae, Depressor supercilli, Corrugator

5 Upper lid raiser Levator palpebrae Superioris

6 Cheek raiser Orbicularis oculi, Pars orbitalis

7 Lid tightener Orbicularis oculi, Pars palebralis

9 Nose wrinkler Levator labii superioris, alaeque nasi

10 Upper lip raiser Levator labii superioris, caput infraorbitalis

11 Nasolabial fold deepener Zygomatic minor

12 Lip corner puller Zygomatic major

13 Cheek puffer Caninus

14 Dimpler Buccinator

15 Lip corner depressor Triangularis

16 Lower lip depressor Depressor labii

17 Chin raiser Mentalis

18 Lip puckerer Incisivii labii superioris; Incisivii labii inferioris

19 Tongue out

20 Lip stretcher Risorius

21 Neck tightener

22 Lip funneler Orbicularis oris

23 Lip tightener Orbicularis oris

24 Lip pressor Orbicularis oris

25 Lips part Depressor labii, or Relaxation of mentalis or orbicularis

oris

26 Jaw drop Masetter, Temporal and Internal pterygoid relaxed

27 Mouth stretch Pterygoids, Digastric

28 Lip suck Orbicularis oris

29 Jaw thrust

30 Jaw sideways

31 Jaw clencher

32 Lip bite

33 Cheek blow

34 Cheek puff

35 Cheek suck

36 Tongue bulge

37 Lip wipe

38 Nostril dilator

39 Nostril compressor

41 Lid droop

42 Slit

43 Eyes closed

44 Squint

45 Blink

46 Wink

Table I.1: Single Action Units (AU) in the Facial Action Code

1.2.5: The Basic Emotion Theory (BET)

The term basic means that there are a number of separate emotions which significantly differ from one another. According to Darwin’s theory, these emotions evolved due to their adaptability in dealing with fundamental life tasks. The BET is considered as being a Neuro- Cultural theory: the term ‘Neuro’ directly refers to the Facial Affect Programs, which are in- nate and genetically transmitted while the term “Cultural” refers to expressive variability, which is due to several cultural factors. According to this theory, basic emotions are distin- guished by four main characteristics: i) distinctive universal signals (emotions provide specif- ics about what is required to generate a given expression and about what will occur after the

17

expression); ii) distinctive physiology (autonomic nervous system and central nervous system activities are unique for each emotion and are not found in other mental activity); iii) auto- matic appraisal (a high speed mechanism of evaluation which occurs in the interval between the stimuli that elicits the emotion and the actual emotion response); iv) distinctive universals in antecedent events (some common elements in the contexts in which emotions are found to occur) (Ekman, 1999). Moreover, Ekman & Friesen (1975) delineated the prototypic interper- sonal events which would universally call forth each set of the basic emotions: “An actual or threat of harm for fear; the loss of an object to which one was attached for sadness; an event that is either unexpected or contrary to expectation for surprise; something that is repulsive, to the senses or one’s beliefs, for disgust; disapproving and feeling morally superior to someone for contempt; five antecedents for anger: frustration resulting from interference with one’s activity, a physical threat, seeing someone do something that violates one’s values, another person’s anger directed at oneself; four antecedents for happiness: sensory pleasure, excite- ment, praise, relief when something unpleasant has ceased.” (Ekman, 1994)

1.2.6: The Dimensional-Contextual theory

In the Dimensional-Contextual (DC) theory Russell (1997) sought to combine theories supporting the primary role of context in eliciting facial behaviour in emotion detection (Sherman, 1927; Pudovkin, 1970) with theories assigning a central function to actual facial expressions in emotion recognition. According to Russell (1997), “the face does not signal specific emotions, but the observer does infer much about the expresser from the face that is relevant to emotion”. More specifically, when seeing a face, the observer automatically ob- tains two kinds of information: first, he obtains a non-emotional, quasi-physical information such as eyes and mouth which open and close, or a change in the direction of the gaze (infor- mation about an expresser’s actions and attention); second, the observer judges the ex- presser’s overall level of pleasure (pleased vs. displeased) and arousal (agitated vs. sleepy). It is precisely this automatic recoding of the facial expression in terms of pleasure and arousal, which actually provides the primary source of information about the expresser’s emotional state. Pleasure and arousal were considered by Russell as being general features common to many different emotions and these two dimensions may be described a ‘psychological judge- ment space’ within which any perceived stimulus may be located. The two dimensions of pleasure and arousal are represented as orthogonal axes of a Cartesian (two-dimensional)

18

space so that the output becomes a ‘graduation’ of emotional states along dimensions: the horizontal axis ranges from displeasure (e.g. “misery”, “unhappiness”) at one end, to pleasure (e.g. “happiness”, “contentment” ) at the other, passing through a neutral point; the vertical axis ranges from sleep (e.g. “drowsiness”, “lethargy”) through the same neutral point to ex- treme arousal (e.g. “agitation”, “frenzy”). The neutral point forms a point of central tendency, corresponding to the adaptation level.

Russell affirmed that “Judgments of quasi-physical information, pleasure and arousal are automatic, effortless inferences that our perceptual system imposes on a facial pattern”.

However, the cognitive processing of a face generates further inferences helping the detection of the emotion. Thus, Russell did not deny that categorization of emotions occurred at all, but rather suggested that the level of processing at which such an operation would take place was not the perceptual one.

1.2.7: The Componential Appraisal Theory

Componential Appraisal Theories (CAT; Scherer, 1984a,b; Frijda & Tcherkassof, 1997; Smith, 1989; Scherer, 1992; Smith & Scott, 1997; Ellsworth & Scherer, 2003) adopt a markedly cognitive approach to the interpretation of emotions, casting doubt upon the notion that emotions are holistic and discrete entities mapped to stereotyped expressions by an innate neuro-motor program.

The principal assumption is that emotion is a dynamic process encompassing several components such as cognitive activity, motor expression, physiological arousal, action ten- dencies, and subjective states of feeling; each of these elements correspond to a different function, and are developed through a distinct system. The term ‘emotion’ is reserved to those periods of time during which most of these systems are coupled or synchronized to produce an adaptive reaction to an event that considered as being central to the individual’s well-being (Sander, Grandjean & Scherer, 2005).

A second assumption of this theory is that emotional processes are elicited and dy- namically patterned as the individual continuously appraises objects, behaviours, events, and situations in relationship to their effect on his values, needs and aims. Scherer formulated a detailed Componential Patterning Model (CPM, see Scherer, 1984a, 1984b, 1987, 1993, 2001) by postulating that appraisals concern five major dimensions of the emotion-inducing event: novelty, intrinsic pleasantness, goal significance, coping potential, and compatibility.

19

Crucially, the appraisals of these emotion-antecedent dimensions, termed Stimulus Evaluation Checks (SECs), are supposed to occur in a specific temporal sequence.

Finally, the CAT assumes that the outcome of the SECs directly define the emotional reaction at a detailed level since the result of each SEC provokes a selective effect on each emotion component. In particular, Scherer (1984, 1987, 1992, and 2007) makes specific pre- dictions concerning the sets of facial actions associated with emotion-antecedent appraisal and, in relation to action tendencies; he suggests that expressions reflect the integration of individual actions taking place in a sequential-cumulative fashion as the appraisal process unfolds. The following table (Table I.2) summarises the predictions of CPM which associate the cognitive appraisals of SECs with specific facial Action Units (AUs).

The combination of these three assumptions implies that a large number of highly- differentiated emotional states will emerge from varying links within the underlying appraisal processes. Furthermore, they also imply that the great variability of real-life expressions does not result from a blend of a small number of discrete configurations, but originates directly from the link between SECs and specific facial actions.

1) NOVELTY CHECK Novel: AUs 1+2, 5; or 4, 7, 26, 38; gaze directed

Not novel: no change

2) INTRINSIC PLEASANTNESS CHECK Pleasant: AUs 5, 26, 38; or 12, 25; gaze directed

Unpleasant: AUs 4, 7, 9, 10, 15, 17, 24, 39; or 16, 19, 25, 26; gaze aversion

3) GOAL/NEED SIGNIFICANCE CHECK Not relevant: no change

Relevant and consistent: relaxation of facial muscle tone

(if conductive to goal, trophotropic shift plus elements from pleasantness response; if obstructive to goal, trophotropic shift plus elements from unpleasantness response)

Relevant and discrepant: AUs 4, 7, 23, 17; gaze directed

(if conductive to goal, ergotropic shift plus elements from pleasantness response; if obstructive to goal, ergotropic shift plus elements from unpleasantness response)

4) COPING POTENTIAL CHECK

No control: hypotonus of facial musculature, AUs 15, 25, 26, 41, 43; (if tears: 1 and 4); gaze aversion Control and high power: AUs 4, 5; or 7, 23, 25; or 23, 24, 38; stare

Control and low power: AUs 1, 2, 5, 26, 20, 38, switching between gaze direction and aversion 5) NORM/SELF COMPATIBILITY CHECK

External/internal standards surpassed: ergotropic shift plus elements of pleasantness and high-power response External/internal standards violated: ergotropic shift plus elements of unpleasantness and low-power response

Table I.2: Component Patterning Theory Predictions for Facial Action Units Following Outcomes of Major Stimulus Evaluation Checks

20

2. PERCEPTION OF EMOTIONS

2.1: THE PERCEPTION OF FACIAL EXPRESSIONS OF EMOTIONS

How are emotions recognized from facial expressions? In order to answer this ques- tion in the next paragraphs we will set out the two principal hypotheses which explain the same perceptive mechanism in two different modalities. The first hypothesis, the Direct Per- ception Theory (DP) aligns itself with the BET (see above) while the second one, the Dimen- sional Contextual Theory (DC), is formulated within the context of Russell’s Conception of Emotions. Moreover, we will expose the Motor Theory which associates the perception to the motor production of facial expression of emotions.

2.1.1: The Direct Perception Theory

The DP affirms that the variety of human facial expressivity may be organized around certain prototypic expressions which correspond to a small number of discrete basic emo- tional categories. Expressions may then be perceived as belonging to qualitatively distinct categories, with sharp divisional boundaries.

Several studies have developed specific codification systems which describe objective muscle movements implied in the facial expression of specific emotions (FACs, Ekman &

Friesen, 1978; Maximally Discriminative Facial Movement Coding System, Izard, 1979).

The main experimental studies supporting this theoretical perspective are summarized hereunder.

1) Babies respond selectively to different facial expressions (Field, Woodson, Greenberg,

& Cohen, 1982)

2) Several neuro-physiological studies have demonstrated that facial expressions are elabo- rated by independent neuronal systems. Neurons specifically responsive to facial ex- pressions have been found primarily in the cortex of the superior temporal sulcus (STS) of the macaque monkey (Hasselmo, Rolls, & Baylis, 1989), while neurons responsive to identity have been found in the inferior temporal gyrus (Young & Yamane, 1992). Since the output from the temporal cortical visual areas reaches the amygdale and the orbito- frontal cortex, which are particularly involved in social and emotional responses to faces, it then follows that these temporal networks would be involved in social interac-

21

tions (Rolls, 1992). This was confirmed by the observation that lesions of the temporal lobe in the monkey leaded to deficits in social behaviour, that were similar to those as- sociated with the Kluver-Bucy syndrome (Leonard, Rolls, Wilson, & Baylis, 1985).

3) Neuro-image studies (PET, fMRI) have demonstrated that perceiving emotional facial expressions involves the activation of different areas of the brain to those employed in perceiving facial identities (Phillips, 1998)

4) Clinical studies regarding Huntington’s disease (Sprengelmeyer, 1997) and neuropsy- chological studies (Adolph, Tranel, Damasio & Damasio, 1994) support the thesis that distinctive neuronal systems are employed in the manifestation of different emotions.

2.1.2: The Dimensional-Contextual Theory

This alternative perspective is based on the assumption that the relationship between emotions and facial expressions is graded and continuous, rather than categorical, discrete, and “deterministic” as basic emotions theories suggest. From a more ‘dimensional’ perspec- tive, the affective content of facial expressions is characterised by attributes varying continu- ously along the dimensions of a hypothetical spatial model (e.g. Nummenmaa, 1992; Russell

& Bullock, 1986).

Study results confirm the existence of a ‘face space’, where standard emotion catego- ries are represented by points in a space whose major axes refer to some relevant dimensions that differ from model to model. As suggested by Russell and Fehr (1987), the emotional space is a geometric metaphor for the internal scale within which facial expressions are per- ceptually judged. In this framework emotional categories still exist to some extent, but they are fuzzy, and establishing whether an emotion belongs to one category or another depends on a set of continuous parameters. For example, Woodworth (1938) and Schlosberg (1954) iden- tified happiness (in which they included love and mirth), surprise, fear (including suffering), anger (with determination), disgust, and contempt as distinct and recognisable categories of emotion. However they found that emotion category ratings and subjects’ ‘errors’ (e.g. the likelihood of their labelling a putatively disgusted expression as contempt), could be accu- rately predicted by arranging emotion categories around an ellipse with two orthogonal di- agonals corresponding to the dimensions pleasantness versus unpleasantness (running from happiness region to the region of anger) and attention versus rejection (running from the sur- prise-fear boundary to the disgust-contempt boundary). Woodworth and Schlosberg (1954)

22

explicitly noted the analogy between this two-dimensional solution to the perceptual classifi- cation of facial expressions and the colour circle, in which hue was arranged around the cir- cumference and saturation along red-green and blue-yellow opponent axes (Young et al, 1997).

Russell’s (1980) circumplex model provides a more recent variant of the same idea.

This represents the structure of emotion concepts in a two-dimensional emotion space defined by orthogonal bipolar dimensions, pleasure and arousal. While more extreme degrees of an emotion fall around the edge of such space, milder emotion is located more towards the cen- tre. Russell and Bullock (1986) extended the model by proposing that emotion categories were best thought of as ‘fuzzy’ sets at the level of recognition. They maintained that, although we categorize facial expressions of emotions into discrete categories, only a few facial ex- pressions belong fully and exclusively to one particular category (100% membership). Ac- cordingly, expressions will exhibit intermediate degrees of membership of more than one category, and the categorization of an emotional expression will therefore depend on the con- textual relation to other expressions with which it may be compared. Taken from this perspec- tive, the facial expression confusion data supporting structural models such as Schlosberg’s (1952) simply reflect the overlap of these fuzzy categories.

To test the circumplex model, Russell and Bullock asked subjects to rate a variety of facial expressions by indicating how well each of them exemplifies categories such as ‘ex- cited’, ‘happy’, ‘sleepy’, ‘mad’, and so forth. The results indicate that the categories do in- deed overlap, and that in the case of Ekman’s basic emotion categories the maximum esti- mated degree of membership actually exists for Ekman’s prototypical expressions. A subse- quent similarity structure analysis (multidimensional scaling [MDS]) performed on the sub- jects’ ratings yielded two dimensions which were highly correlated with pleasure and arousal ratings. Two further tests confirmed this pattern: when asked to indicate whether a facial ex- pression corresponded to a given emotion, a higher level of consensus was reached for proto- typical expressions than for borderline expressions. Finally, asking subjects to choose exem- plars for each emotion revealed a graded membership function for emotions. On the basis of these results, Russell and Bullock hold that decoding a facial expression involves first an ap- praisal in terms of pleasure and arousal, and that choosing a label for the expression merely represents a subsequent optional step. They, like Schlosberg, maintain that contextual infor- mation is required to elicit finer, more reliable judgements. The basic premise that facial ex- pressions are coded in a multidimensional, perceptual space, and that similarity judgments

23

reflect the proximity structure within this space has also been supported by several other stud- ies (Schiano, Ehrlich, Sheridan, & Beck, 2000; Katsikitis, 1997; Carroll & Russell, 1996;

Russell, 1980; Russell, Lewicka, & Niit, 1989)

2.1.3: The Motor Theory

Understanding actions of others is essential pre-requisite for coexistence between sub- jects belonging to the same species. Consequently, movements communicate specific mean- ings about states of mind and intentions of others and may thus be considered as social factors facilitating interaction and comprehension between group members. Such ability in predicting and understanding the intentions of others, and moreover the capacity of imitating others, may imply the existence of a system associating the representations of one’s own actions with oth- ers’. Under this hypothesis perception is constrained by action and, more precisely, perception is the internal simulation of action.

The core premise of the core of the Motor Theory of perception is the fact that the human brain contains structures which become activated during both the first-person experi- ence and the third-person observation of actions and emotions; such structures create a bridge between ourselves and others and are embodied by the system of mirror neurons. Studies have shown that when an individual observes someone performing an action, this prompts the acti- vation of several cerebral visual areas as well as the motor neuronal system responsible for the production of the observed action (Gallese, Keysers and Rizzolatti; 2004). Thus mirror neurons are a particular class of visual-motor neurons which relate visual and motor proprie- ties transforming visual information (action observation) into knowledge (action understand- ing) and that are activated both while doing an action and while observing the same action in others. As far as communication is concerned, the system of mirror neurons is the anatomical basis linking the two players in the context of communication, the sender and the recipient;

such system enables the observer to immediately comprehend another’s action, without re- sorting to a sophisticated cognitive elaboration. This mechanism is well detailed by the ‘motor resonance’ of the mirror neurons system (Gresez & de Gelder, 2005) whereby the perception of a movement of the mouth activates the brain area representing the movements of the mouth (likewise the perception of a movement of the hand activates the brain region representing the hand). Motor resonance underpins all biological movements managed by the observer. The

24

activation of the corresponding system of mirror neurons is more intense when the observer is able to perform the observed movement than when he not.

The existence of a specific anatomical location of mirror neurons system has been demonstrated both in monkeys and humans. When monkeys perform an action, and merely observe the same action performed by another, they activate, in the pre-motor cortex, the same area F5 of superior temporal sulcus, STS (Rizzolatti & Craighero, 2004). Two types of mirror neurons exist in monkeys in area F5: the strictly and the broadly congruent, depending on the degree of correspondence between the action they see and the motor response they code. However, visual observation of an action is not required to activate such neurons; if monkeys listened to actions (acoustic perception) or imagined them (mental representation) then their mirror neurons are activated to the same degree as when observing such actions.

Several neurophysiologic and brain imaging studies have found evidence regarding the exis- tence of mirror neuron system in humans. EEG recording with Magneto Encephalographic technique and the Transcranial Magnetic Stimulation method has demonstrated that the corti- cal excitability of the motor cortex varies according to the observed movements and that spi- nal cord excitability inversely varies to the observed action due to the effect of an inhibitory mechanism (Cochin, Bartelemy, Lejeune, Roux and Martineau, 1998; Baldissera, Cavallari, Craighero and Fadiga, 2001). Moreover, fMRI and PET investigations during the observation of an action, have localised the human mirror neurons system; they have found simultaneous activity of occipital, temporal and parietal visual areas and of the parietal and premotor re- gions of the inferior parietal lobule (rostral part), the inferior frontal gyrus and the precentral gyrus (Grezès et al; 1998, 2003).

Williams et al (2001) investigated the impairments of autistic subjects in imitating the actions of others and associated such deficits with several dysfunctions in the parietal cortex of patients.

The question which arises is whether the simulation system of motor resonance forms the basis of the understanding of emotions between individuals.

Several studies suggest that the perception of emotional behaviours, such as anger and expressions of fear, activate in the observer the mechanism responsible for the production of the very same emotion. Dimberg (1982) measured the activity of facial muscles of subjects exposed to facial expressions of anger and happiness and found spontaneous facial reactions selective for each emotion while Wallbot (1991) found that the perception of an emotional expression triggered in the observer the automatic imitation of such expression. In general,

25

the perception of an emotional facial expression activates, in the observer, the motor represen- tation corresponding to the generation of the same emotion (Gresèz, de Gelder, 2005). More- over, the motor resonance concerning emotional expressions is associated with alterations of the internal emotional states, for example with the increase of blood flow in specific brain regions. In the emotional contagion, emotional states are associated with specific facial ex- pressions so that when observing a facial expression the result is a mirrored premotor activa- tion of the observer and a corresponding feedback of the emotional state. The neural system responsible for the emotional contagion is the connection between superior temporal sulcus, mirror neurons and amygdale (Williams et al, 2001). However, the motor resonance effect is not uniform for all emotions: according to de Gelder et al (2004) motor system activations are more relevant for perception of fear than for perception of happiness. This may be due to the intrinsic differences between the basic emotions postulated by Gresèz et al (2005) that read basic emotions considering two distinctive dimensions: their link with the expected ac- tion/reaction and their degree of socialization with others. Accordingly, emotions of fear and disgust are similar in nature in that they are connected to the following reactions which bring about, for both emotions, quasi-automatic reflexes; escaping from danger out of fear and avoiding something undesired out of disgust. From this perspective, understanding fear and disgust appears to require less cognitive mediation than other basic emotions. It has been demonstrated that disgusting stimuli activate left anterior insula in the brain (Royet et al, 1999; Zald et al, 2000; Small et al, 2003); Phillips et al (1997) and Wicker et al (2003) con- firmed that anterior insula is also activated upon sight of a disgusted facial expression. Con- cerning fear, several studies support the central role played by the amygdale in both produc- tion and perception of fearful facial expressions (Morris, Frith, Perret, Rowland, Young, Cal- der & Dolan, 1996; Calder et al, 1996). However, the simulation mechanism is not the only system for understanding the emotions of others because such process is also mediated by the cognitive interpretation of emotions, based on the elaboration of the visual aspects of their expressions (Goldman & Spirada, 2005). Finally, all the studies concerning the perception of facial expressions suggest that the coexistence of two methods of understanding emotions: the cognitive elaboration of the stimuli and motor resonance.

26

2.2: THE CATEGORICAL PERCEPTION (CP) PHENOMENON

Informally, the phenomenon known as categorical perception refers to any situation where sensory systems appear to superimpose discontinuous category boundaries upon a con- tinuous stimulus space, so that the category membership of a stimulus qualitatively affects the perception thereof. Before providing a more formal definition let us illustrate this phenome- non with a simple example involving a natural sensory continuum, namely the perception of colours. The colour spectrum corresponds to a continuous range of light wavelength. Never- theless, our perception of this range of wavelength is not smooth: stimuli varying continu- ously along the wavelength dimension are perceived as being ‘carved up’ into discrete

‘chunks’ of colours. The effect is more reliably demonstrated by the fact that two colours straddling a category boundary (green-yellow) are easier to discriminate than two colours falling within the same category (green-green) even though the physical difference in wave- length is the same for both pairs (Bornstein & Korda, 1984).

The enhancement of ‘between-category’ differences, and the associated reduction of

‘within-category’ differences is known as the “categorical perception effect” (Harnad, 1987).

It was initially demonstrated in the case of one-dimensional stimuli with an experimental paradigm coupling two psychophysical tasks, namely an identification and a sequential (or simultaneous) discrimination task. Although it is generally assumed that categorical percep- tion (CP) effects arise as a result of low-level perceptual processes of simple stimuli varying along one dimension, some authors (see Harnad, 1987 for an exhaustive review) have sug- gested that CP may provide a representative model for the categorization process in general.

To support this ‘unifying hypothesis’, recent research has searched for evidence of categorical perception at higher (cognitive) levels, including the perception of faces. It should be stressed, however, that CP represents a highly specific psychophysical phenomenon, and - as we will see in the following paragraphs - a number of conditions must be satisfied in order to validate the existence thereof.

2.2.1: The Psychophysics of Categorical Perception

CP effects emerge as qualitative differences in how similar stimuli look or sound de- pending on whether or not they fall into the same category. More precisely, equal-sized physical differences between stimuli are perceived as being larger or smaller depending on

27

whether or not the stimuli are fall into the same category. Clearly, qualitative differences in perception cannot be demonstrated objectively. The only direct evidence comes from verbal reports and our own introspective experiences. However, quantitative differences can be tested and evaluated experimentally, with methods and assumptions typical of the psycho- physical approach. Psychophysics is basically concerned with the relationship between a range of physical stimuli of well-controlled intensity (the physical continuum) and the per- ceived intensity of the corresponding sensation (the psychological continuum). Since the first experiments with synthesized acoustic continua of speech sounds, psychophysical research on CP has focused on developing a quite defined set of criteria as well as a basic standard meth- odology through which the existence of CP may be assessed operationally. More specifically, categorization is studied by examining the limits of discrimination (how small a physical difference we can distinguish) and of identification (what classes of stimuli we reliable corre- late). The reasoning behind this approach will emerge more clearly following a formal defini- tion of the necessary and sufficient conditions for CP to occur.

The term Categorical Perception applies to all cases where the following conditions are met:

1) We consider two physical stimuli A and B and their connecting path. Stimulus C along the path also belongs to such a class of physical stimuli and for each intermediate stimulus C we can measure its objective distance d(C,A) and d(C,B) from the path endpoints A and B.

2) The objective distances d(C,A) and d(C,B) have correspondent perceived distances δ(C,A) and δ(C,B) which can be experimentally estimated. C is a stimulus moving along the con- necting path, thus the perceived distance is a monotonously increasing function of the ob- jective distance, such as δ(C,A) = F(d(C,A)).

3) We can always find two stimuli A and B, whereby the derivative of F with respect to d(C,A) has a unique maximum lying somewhere along the path from A to B.

Where CP occurs, the transformation mapping the stimuli into the perceptual space preserves the topological, but not the metrical, properties of the objective space. Specifically, the transformation is such that, up to a point along the path from A to B, C is perceived as being closer to A than it really is. After such point, C is perceived as being closer to B than it really is.

28

Likewise, the phenomenon can be characterized as the result of competing ‘attractor fields’ centered on A and B (Tanaka, Giles, Kremen, & Simon, 1998). CP is a graded as op- posed to an all-or-nothing phenomenon, with the strength of the effect depending on the rela- tive value of the maximum of the derivative of F. In the ideal case of ‘pure’ CP, F is a step function, and its derivative is zero everywhere but at a single point, where it is infinite.

In essence, the function F is a classical psychophysical function relating the perceived to the objective value of the distance from a given endpoint. Thus, it would be conceivable to experimentally address the study of CP with the standard techniques for estimating psycho- physical functions (Guilford, 1954). In almost all cases, however, taking an indirect route, by measuring the effects of the warping of the representational space on identification and dis- crimination performances (cf Harnad, 1987) has been found to be more profitable. The most utilized experimental strategy for detecting CP consists in coupling two experiments: the first is an identification task and the second a discrimination task. First, the observers familiarized themselves with the endpoint stimuli A and B. Then, upon having been shown an intermediate stimulus C, they had to decide whether C was more similar to A or B. The results were col- lected as psychometric functions relating the probability p(B) of answering B to the objective distance d(C,A). The experimental estimation of the intermediate stimulus along the path from A to B where p(B) = 0.5 enables the researcher to affirm that, at that point, C is per- ceived as being equally distant from A and B (point of indifference). However nothing can be inferred about CP directly from the psychometric function (for an exhaustive discussion of this point, see Viviani, Binda, & Borsato, [2006]) but, having demonstrated that CP occurs, the point of indifference can be considered as being the boundary between two space regions A and B. If the warping of the space is quite strong, any stimulus within either region will be difficult to discriminate from the respective centers, and the regions can be interpreted as fuzzy categories. The direct evidence of CP is obtained though the second experiment, involv- ing a discrimination task, either in the ABX or in the ‘Same-Different’ version. Where CP occurs, pairs of stimuli that lie on opposites sides along the path with respect to the point of indifference should be easier to discriminate than pairs that lay on the same side, even though the distance between the stimuli is the same in both cases.

29 2.2.2: CP Evidences for Faces

Categorical Perception (CP) occurs whenever we impose boundaries within an other- wise continuous stimulus space. An example of CP is provided by the way we perceive cer- tain speech sounds. For instance, Liberman, Harris, Hoffman, & Griffith, (1957) demon- strated that when a sequence of synthesized sound stimuli vary continuously from /ba/ to /pa/, at some intermediate point along the sequence the perceived sound shifts abruptly from the first to the second syllable. Moreover, pairs of stimuli equidistant along a continuous physical dimension (the voice-onset-time in this case) are perceived differently, depending on their position along the continuum. When the pair straddles a categorical boundary, it gives rise to clearly distinct syllables. When it does not straddle the boundary, the sounds are indistin- guishable: both are perceived as either /ba/ or /pa/, not as a mixture of the two syllables. If indeed facial expressions constituted perceptual categories, one might expect similar phenom- ena to occur when viewing a sequence of images ranging continuously from one well-defined expression to another. However, such an expectation must be qualified for both theoretical and methodological reasons. In all classical examples of CP (phonemes, hues, musical inter- vals, etc.) the stimuli varies along a single physical parameter (voice onset time, wavelength, and time, respectively). Therefore, both the objective distance of an intermediate stimulus from the end-point templates, and the objective distance between two stimuli are univocally defined. The case of facial expressions is less straightforward. Firstly, they are very complex stimuli varying across many relevant dimensions. More importantly, variations along one di- mension are probably correlated to variations along others. Indeed, although not much is known about how facial information is conveyed, it is likely that the face, as a holistic con- figuration, plays an important role in semantic communication.

It has been demonstrated that recognition of face identity depends critically on con- figurative information about the spatial relations among features (Moscovitch, Winocur &

Behrmann, 1997; Sergent, 1984), especially when viewing the face in the normal upright ori- entation (Searcy & Bartlett, 1996; Yin, 1969; A.W. Young, Hellawell, & Hay, 1987). The issue of feature-based versus configuration-based processing is far less well understood in the case of facial expressions. Although some evidence suggests that feature-based processing might be sufficient to map facial expressions to emotion categories (Cottrell, Dailey, Padgett,

& Adolphs, 2001), other studies have indicated that perception of facial emotion in humans require at least some configurative processing of the relation between a number of facial fea-

30

tures (Calder, Young, Keane, & Dean, 2000). Therefore, to generate an artificial continuum of faces all the relevant facial features must simultaneously and linearly covariate, while preserv- ing their mutual relations within the entire configuration (note: morphing techniques).

Several studies have demonstrated the occurrence of categorical perception for faces:

firstly CP was discovered in emotion recognition from faces (Etcoff & Magee, 1992; Calder, Young, Perret, Etcoff, & Rowland, 1996; de Gelder, Teunisse, & Benson, 1997; Young, Row- land, Calder, Etcoff, Seth, & Perret, 1997; Bimler & Kirkland, 2001; Campanella, Quinte, Bruyer, Crommelinck, & Guerit, 2002; Pollak & Kistler, 2002; Suzuki, Shibui, & Shigemasu, 2004), and it was later demonstrated in race perception from faces (Levin, 1996; Levin and Beale, 2000; Levin and Angelone, 2002) and gender discrimination from faces (Campanella, Chrysochoos, & Bruyer, 2001; see, however, Bülthoff and Davidoff, & Valentine, 2001;

Campanella et al, 2003; Levin and Angelone, 2002; McKone, Martini and Nakayama, 2001;

Stevenage, 1998; Rossion, Schiltz, Robaye, Pirenne, & Crommelinck, 2001; Rotshtein et al, 2005; and also Viviani, Binda, & Borsato, 2006).

Etcoff and Magee (1992) carried out the first study into categorical perception for fa- cial expressions, in order to decide whether the perceptual mechanisms responsible for facial expression recognition are actually attuned to emotion categories, with category membership

‘assigned by higher conceptual and linguistic systems’ (Etcoff and Magee, 1992). They con- verted photographs from the Ekman and Friesen (1976) series of pictures of facial affect into line drawings and used a computer program to generate 10 morphs. The pairs of templates used in the experiment included happy-sad, angry-sad, and angry-afraid, which were sup- posed to be easy to discriminate, and surprised-afraid and angry-disgusted, which were sup- posed to harder to discriminate. The stimuli also included the pair happy-neutral and sad- neutral to test for category boundaries along the dimension of presence versus absence of an emotion and happy-surprised as an example of transition between positive emotions. The ex- periment adopted the standard two-step procedure to assess CP (identification followed by an ABX discrimination task).

Etcoff and Magee concluded that all expressions except Surprise are perceived cate- gorically. They highlighted two aspects of their results as providing evidence of CP effects: 1) in the ABX task (see chapter III.3), morph pairs straddling the 50% category boundary were significantly better discriminated than those closer to the prototypes, and, 2) in the identifica- tion task, subjects placed ‘sharp boundaries’ between the two regions in each continuum where morphs were consistently perceived as corresponding to either one or the other expres-