HAL Id: hal-03094445
https://hal.archives-ouvertes.fr/hal-03094445
Submitted on 4 Jan 2021
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Speaking while moving: Does the head compensate for the hands not being able to move?
Hélène Serré, Susanne Fuchs, Marion Dohen, Amélie Rochet-Capellan
To cite this version:
Hélène Serré, Susanne Fuchs, Marion Dohen, Amélie Rochet-Capellan. Speaking while moving: Does the head compensate for the hands not being able to move?. ISSP 2020 - 12th International Seminar on Speech Production, Dec 2020, New Haven, United States. �hal-03094445�
Speaking while moving: Does the head compensate for the
hands not being able to move?
Hélène Serré, GIPSA-lab, helene.serre@gipsa-lab.fr Susanne Fuchs, fuchs@leibniz-zas.de Marion Dohen, GIPSA-lab, marion.dohen@gipsa-lab.fr Amélie Rochet-Capellan, GIPSA-lab, amelie.rochet-capellan@gipsa-lab.fr
Background
Question
Procedure
Analysis
Re su lt s
25 German speakers (19 women and 6 men) between 20 and 29 years old (mean=23.3 years, standard deviation=2.5 years)
Main task : recall 4 stories freely, in one of the following conditions: hands free (HF), hands blocked (HB), biking with the legs (LBi), biking with the
arms (ABi)
HF HB LBi ABi
• Not being able to move the hands can reduce vocabulary diversity and precision (Rimé, 1984; Hostetter, Kita & Alibali, 2007) and increase
disfluency (Finlayson et al., 2003; Hostetter, Kita & Alibali, 2007). However, for contradictory results see Hoetjes et al., 2014.
• While speaking, people also move their head and eyebrows (e.g. Kendon, 2004) potentially in synchrony with prosodic parameters (Graf et al., 2002;
Hadar, 1983).
• Eyebrow, eye and finger movements would increase when speakers cannot gesture with their hands (Rimé et al., 1984).
• Participants could compensate for not being able to move their hands by
moving other parts of their body (Finlayson et al.,2003). This compensation might remove the effects of the constraint.
• Do head movements compensate for the hands not being able to move?
• In this preliminary study, we analyzed the kinematics of head movements using data from a motion capture experiment investigating speech-limb-
breathing interaction. People were asked to recall stories in different limb movement conditions.
• Automatic analysis of motion using motion capture could be helpful for analysing larger data sets and provide complementary evidences to
qualitative approaches (Zhao et al, 2012).
Conclusions
❖ There is no difference in average head displacement whether the participants have the hands blocked or not when they speak.
❖ Maybe because they do no gesture a lot. The relationship between hands and head movement quantity suggests that when the speakers are free to move, the more they move their hands, the more they also move their head.
❖ When limbs and hands are blocked in a 50-minute conversation, the activity of eyebrows, eyes, mouth, and fingers increases (Rimé et al., 1984) vs a 2-minute story retelling with a much higher cognitive load ➔ less communicative gestures.
❖ MoCap equipment may also affect speakers’ motions. Eyebrows movements may be a more reliable indicator.
❖ Entrainment between body parts, rather than compensation, may support speakers’
behavior.
❖ Motion capture will enable the exploration of different aspects of kinematics such as limb synchronization or rotation motion.
Wagner et al, 2014
Switch from fixed to moving coordinate referential
measure of synchronisation between back and head movements on each axis: coherence
3D displacement and rotation movements
But if participants don’t hand gesture when they are free to do so, they do not need to compensate with the head when they are constrained. What about the relationship between hands movement quantity and difference in head movement quantity from free to blocked condition ?
• No difference
between HF and HB
• As predicted by the coherence, More
head movement for ABi
In the HF condition, the more the participants gesture with the hands, the more they move the head, but it is not correlated with head movement
quantity in the HB condition.
Violin plots of average displacement of the head (y-axis) as a function of movement conditions (x-axis).
Scatterplot with difference of displacement of the head between HB and HF (y-axis) as a function of the averaged hand displacement in HF (Pearson correlation is -0.8, t = -6.21, df = 22, p-value = 2.8e-06).
Averaged displacement and rotation of the head on each trial
1
2
4 3
Antero-posterior
axis mediolateral
axis vertical
axis
Acknowledgments This research is jointly supported by the French National Research Agency (ANR-17-FRAL- 0005-01) and the German Research Foundation (DFG) as part of the SALAMMBO project (Spoken Language in Motions: Learning and Adaptation of Speech Communication in the context of Body Motions) ; thank you to François Bérard from the LIG, Grenoble, and Silvain Gerber from Gipsa-Lab, for their help.
References [5] Rimé, B., Schiaratura, L., Hupet, M., & Ghysselinckx, A. (1984). Effects of relative immobilization on the speaker's nonverbal behavior and on the dialogue imagery level. Motivation and Emotion, 8(4), 311-325. [6] Hostetter, A. B., Alibali, M. W., &
Kita, S. (2007). Does sitting on your hands make you bite your tongue? The effects of gesture prohibition on speech during motor descriptions. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 29), 1097-1102. [7] Finlayson, S., Forrest, V., Lickley, R., & Beck, J. M. (2003). Effects of the restriction of hand gestures on disfluency. Proceedings of DiSS’03, Disfluency in Spontaneous Speech Workshop, 5–8 September 2003, Göteborg University, Sweden. Robert Eklund (ed.), Gothenburg Papers in Theoretical Linguistics 90, ISSN 0349–1021, p 21-24. [8] Hoetjes, M., Krahmer, E., & Swerts, M. (2014). Does our speech change when we cannot gesture? Speech Communication, 57, 257-267. [9] Graf, H. P., Cosatto, E., Strom, V., & Huang, F. J. (2002).
Visual prosody: Facial movements accompanying speech. In Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition (pp. 396-401). [10] Hadar, U., Steiner, T. J., Grant, E. C., & Rose, F. C. (1983). Kinematics of head movements accompanying speech during conversation. Human Movement Science, 2(1-2), 35-46. [12] Zhao, W., Chai, J., & Xu, Y. Q. (2012).
Combining marker-based mocap and RGB-D camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation (pp. 33-42). [2] Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press.
