Mood and Emotional Classification
5.2 Emotion and Mood: Taxonomies, Communication, and Inductionand Induction
5.2.1 What Is Emotion, What Is Mood?
What lies beneath the mood and emotional classification of music is the desire to
(i) understand music in terms of what listeners, beyond cognition of musical structures, identify in music or feel about it, and then
(ii) use that understanding for improving retrieval of music.
An important initial step toward achieving these goals is to clarify what emotions and moods are. The New Oxford American Dictionary [3] defines emotion as “a natural instinctive state of mind deriving from one’s circum-stances, mood, or relationships with others,” “any of the particular feelings that characterize such a state of mind,” and “instinctive or intuitive feeling as distinguished from reasoning or knowledge.”
While an emotion is a state of mind that is instinctive and peculiar, mood is something more persistent and obscure. Again, the New Oxford American Dictionary defines mood as “a temporary state of mind or feeling” and, in particular, “the atmosphere or pervading tone of something.” Thayer [75]
defines moods to be “background feelings that last for a time and that often have no particular cause” and often the moods are just “good or bad.”
Type Duration Awareness Target
Emotion Instinctive Full Often present
Mood Persistent Partial, or even absent Absent
Table 5.1
Comparison between Emotion and Mood
These definitions enable us to draw some line between mood and emotion.
While emotions are instinctive and sometimes attached to particular objects, mood may drift from one to another, people may not recognize their mood un-til they pay attention to them, people may not be able to figure out why they are in a specific mood, and while being in a certain mood people may experi-ence various emotions. Table 5.1 summarizes the differexperi-ence between emotion and mood.
5.2.2 A Hierarchical Model of Emotions
The literature in developmental psychology states that humans learn basic emotions in their early developmental stage, which is through simple stimuli, and more complex emotions in later stages (see, for example, Bower [7]). This developmental distinction between basic ones and more specific ones suggests a two-level hierarchical organization of emotion. By adding to it the division between positive and negative emotions, one obtains the hierarchical diagram of Schaver et al. [64]. In this hierarchy, at the top level are super-ordinate categories (positive and negative), at the middle level are the basic emotions, which Schaver et al. think of as universally understood emotions, and at the bottom level are the most specific 135 emotions, which may exhibit individual variability in understanding. Figure 5.1 shows the hierarchy in a simplified form with just a few bottom-level emotions present.
5.2.3 Labeling Emotion and Mood with Words and Its Issues For description of emotion and mood we usually use words that represent emo-tion (or affect), for example, those appearing in the emoemo-tion hierarchy. The use of such words is very convenient and natural, but has peculiar characteristics that we must keep in mind.
The first is their multiplexity; i.e., all at the same time people can be in more than one mood and experience multiple emotions. As Michael Franks sang, people may feel like: “I don’t know why I’m so happy and sad.”1
The second is individual variability in emotion assessment. This phe-nomenon is more prominent when emotions are assessed at the subordinate
1Michael Franks, “I Don’t Know Why I’m So Happy and Sad,” from the albumThe Art of Tea, Reprise Records, 1976.
EMOTIONS
SUBORDINATE CATEGORIES SUPERORDINATE
CATEGORIES
BASIC CATEGORIES
Positive Negative
Love Joy Anger Sadness Fear
Fondness Infatuation
Bliss
Contentment Pride
Annoyance Hostility
Contempt Jealousy
Horror Worry Agony
Guilt Grief
Loneliness
Figure 5.1
The hierarchy of emotions according to Schaver et al. [64]. At the subordinate level (the bottom level) only a small number of selected emotion categories are shown.
level than at the basic level or at the top level. On hearing a man telling the story about a very unreasonable customer representative, a friend may think that the man is angry but the man himself thinks that he is just frustrated.
This issue is prominent when humans provide emotion and mood labels to music, in particular, instrumental music.
The last one, which has much to do with the second, is overlap among emotions and among moods. Emotion and mood labels words, which are usu-ally adjectives or nouns, have synonyms—“words having exactly or nearly the same meaning as another word” according to the New Oxford English Dic-tionary. If the size of overlap between an emotion word,A, and another,B, is significant but not 100%, treating the two as different makes assessment of emotion, both by human and by machine, very difficult with respect to these two, since the distinction among “bothAandB,” “Abut notB,” and “Bbut notA” is obscure. The Schaver hierarchy contains as many as 135 emotions, and large numbers of emotion words have been identified, for example, in ANEW (Affective Norms for English Words) [8]. Thus, an attempt to classify music into presence/nonpresence with such a large number of labels appears to be extremely difficult, if not utterly impossible.
5.2.4 Adjective Grouping and the Hevner Diagram
The aforementioned characteristics suggest that to study emotions and moods in music we will perhaps have to use groups that combine various subordinate-level emotions and will perhaps have to treat the classification problem as a multilabel problem.
Hevner [25] is the first to experimentally found adjective groups through subject experiments. In this study, 450 subjects were asked to listen to 26 pieces of classical music and for each piece select from a list of 66 adjectives any number of words that seemed “appropriate to the music.” Through co-occurrence analysis of the response Hevner divided the 66 adjectives into eight groups and laid out the groups into a circle in such a way that the collection of emotions represented by group members gradually changes as we go around the circle (seeFigure 5.2).
Hevner studied the scores of the pieces and attempted to correlate their characteristics with the adjective groups chosen by the subjects. Correlations were found in melody contour (ascending versus descending), harmony (simple versus complex), mode (major versus minor), and rhythm (firm versus flow-ing), with by far the clearest correlation between mode and “Happy” versus
“Graceful” as well as between mode and “Sad” versus “Dreamy.”
5.2.5 Multidimensional Organizations of Emotion
Psychologists have been using two-dimensional plots for organizing emotions.
In such a representation each axis corresponds to a pair of adjectives having opposite meanings (for example, happy and sad). A point in the space thus
1
The eight adjective groups discovered by Hevner.
Fear &
The facial expression categories by Schlosberg.
specifies in each axis, which of the two opposite feelings is more strongly felt.
The use of two independent factors in representing emotion was first suggested by Wundt [89]. Psychological research in the late 20th century expanded the idea of two-dimensional emotion plots and produced a number of plausible representations, perhaps beginning in the seminal work of Schlosberg [67] to classify facial expressions. These ideas were further extended in the work of Russell [63], Thayer [75], and Watson and Tellegen [85].
The diagram of Schlosberg [67] is, like the one by Hevner, circular. In it the whole area of facial expressions is a circle and the area is divided into six sectors: “Fear and Suffering,” “Anger and Determination,” “Disgust,” “Con-tempt,” “Love, Mirth, and Happy,” and “Surprise” (seeFigure 5.3). This idea of circular emotional changes is further investigated by Russell [63], in which 28 emotion words are organized in a circumplex. The two components that span emotion are the Positiveness and the Arousal (i.e., excitement). The Thayer model [75] is inspired by the Russell model. Like the Russell model, the vertical axis represents the arousal level, but the horizontal axis represents the level of tension felt. The two diagrams are shown inFigure 5.4. The emo-tion model of Russell has been simplified in the Barrett-Russell model, which is shown inFigure 5.5. Two-dimensional emotion models assume that a human emotion can be identified with a location on the two-dimensional diagram. In all the three models, Russell, Thayer, and Barrett-Russell, there is an axis that represents positive and negative affects. Watson and Tellegen propose a model using an axis representing Positiveness and another representing Neg-ativeness (seeFigure 5.6) and view the two diagonal axes as Engagement and Pleasantness. For example, the combination of high positive and low negative represents Arousal. From this model Watson, Clark, and Tellegen developed a method called PANAS (Positive Affect–Negative Affect Schedule) for self-appraisal of affect using 20 emotional terms (provide a five-point scale answer with respect to each term) [84]. This has been further extended to PANAS-X, a schedule involving 60 labels [83].
5.2.5.1 Three and Higher Dimensional Diagrams
Some research suggests that the space of perceived emotion has at least three dimensions. Wedin [86] examined rank correlations of words in an adjective checklist on 50+ pieces of music and identifies three pairs of adjective sets.
Wedin suggests that these three pairs correspond to:
• Intensity versus Softness
• Pleasantness versus Unpleasantness or Gaiety versus Gloom,
• Solemnity versus Triviality
Also, Leman et al. [44] study correlations among 15 adjective pairs by having 100 students label 60 pieces of music that cover many genres using the pairs.
Through principal component analysis of the covariance matrix of the pairs,
Astonished Excited
Delighted
Glad Happy
Calm Pleased
Content Satisfied
Serene At Ease Relaxed
Sleepy Miserable
Alarmed Afraid
Tense Angry Frustrated
Distressed Annoyed
Depressed Sad
Tired Bored Gloomy
Droopy Aroused
AROUSAL
POSITIVE AFFECT
Energy
Calm-Tiredness Tension-Tiredness
Tension-Energy Calm-Energy
Calm Tension
Tiredness
Figure 5.4
Top: The two-dimensional circumplex emotion word mapping by Russell.
Bottom: The two-dimensional emotion map suggested by Thayer.
ACTIVATION
LOW NEGATIVE AFFECT HIGH NEGATIVE AFFECT
STRONG ENGAGEMENT
DISENGAGEMENT PLEASANTNESS
UNPLEASANTNESS
Figure 5.6
The emotion diagram by Watson and Tellegen, rotated by 45 degrees clockwise from its often used diagram so as clarify the two axes.
they find that the first three components cover 56.5% of covariance and suggest that those three together capture most of the perceived emotion in music. Each of the three components is a linear combination of six fundamental dimensions (that is, six pairs of adjectives with opposite meanings). Those dimensions were identified as Valence, Activity, and Interest.
5.2.6 Communication and Induction of Emotion and Mood Whether music can communicate any emotion and any mood is perhaps an unsolved question. For any emotion, one can write a lyric that expresses it and attach a melody to the lyric thereby producing a song that communi-cates the emotion. This rather deceptive argument clearly does not apply to instrumental music.
We can tell, however, from the aforementioned work of Hevner [25], that a piece of instrumental music can communicate a certain set of emotions and moods and that such emotions and moods may have to do with the musical structures of the pieces listened to. Hampton [21] shows that there is substantial consistency in emotions recognized by listeners. The emotional expressiveness of music structures, as suggested by Hevner, has been further investigated. Gabrielsson and Lindstr¨om [20] offer a summary of the state of knowledge in this area.
In addition to the emotions and moods represented by structures of music, researchers have been studying how the music is performed has a role in emotion and mood communication. Researchers have found that performers can communicate specific emotions and to do so they use a set of musical cues [12, 19, 31, 56, 76]. Juslin [32] and Gabrielsson [18] are excellent surveys of this area. It is known at least that the basic emotions of Thayer (seeFigure 5.4) can be well communicated using instrumental performances using cue. It is unclear, however, whether subordinate emotions can be well communicated.
Another important topic related to emotion and mood in music is the induction of emotion by music listening. This topic is first extensively inves-tigated by Meyer [53]. As mentioned early the main questions are the types of emotions induced by music and their relations with musical structures and personality. Such studies have possible applications in music therapy and work performance improvement [35, 45, 58]. The extensive literature in this area [14, 23, 33, 38, 57, 65, 66, 71, 72] informs three important points we need to be careful with when studying emotion and mood classification in mu-sic. First and foremost, we should not get confused between perceived emo-tions and induced emoemo-tions. Second, emoemo-tions perceived in music are not necessarily induced in the listeners. Third, whether music listening success-fully induces intended emotions depends on the personality and disposition of the listener.