VR: a naturalistic environment in which to measure language processing 16

One of the major challenges faced by embodied semantics studies is providing eco-logical validity while maintaining experimental control. Embodied theories place great im-portance on environmental and physical contexts, thus calling for multimodal experimental protocols that are closer to real life than those generally used in psycholinguistic studies (Tromp, Peeters, Meyer & Hagoort, 2018; Peeters, 2019). Importantly, embodied semantics is strongly linked to theories of grounded or situated cognition, which claim that the en-vironment plays a crucial role in cognition. Accordingly, language processing and language learning environments, as well as interactions with such environments, are an integral part of cognition (Atkinson, 2010; Black, Segal, Vitale & Fadjo, 2012). On the other hand, given the need for control, experimental protocols that examine “embodied” L2 language processing very seldom provide ecologically valid environments and fail to allow for naturalistic physical

siders that real-world language processing generally occurs in visually rich environments and that physical cues have a strong influence on how language is understood, it becomes clear that the decontextualized 2-D conditions generally used in language processing studies limit their ecological validity (Knoeferle, 2015; Tromp et al., 2018). Studies that analyze the neural correlates of language processing and learning using techniques such as fMRI and EEG are especially limited when it comes to physical interaction and naturalistic movements, given their need to control for artifacts related to muscular activity (Luck, 2014).

Recently, language research has become increasingly invested in providing participants with more ecologically valid multimodal environments where they can engage in semi-natural movement, making for results that can generalize to everyday situations (Peeters, 2019). A multimodal approach to examining the neural underpinnings of language processing should allow for situated cognition in realistic environments, while giving experimentators the same amount of control as they would have in traditional experiments (Peeters, 2019; Tromp et al., 2018). In fact, the more real-world and situated a language processing environment is, the more physically implicated, and natural, participants will feel, and the more applicable the ensuing results will be to real-life processing (Peeters, 2019). Fortunately, modern tech-nology makes it possible to study cognitive processes in richer contexts (Ladouce, Donaldson, Dudchenko, & Ietswaart, 2016). Virtual reality (VR), in particular, offers an environment in which to observe language processing and learning that is both controlled and ecologically valid (Peeters, 2019; Repetto, Cipresso & Riva, 2015; Tromp et al., 2018).

Peeters (2019) argues that VR eliminates the spatial divide between stimulus and participant and that it is an especially interesting approach for topics of a multimodal na-ture, and hence for investigating embodied semantics. There exist three basic types of VR environments: the flat-screen computer monitor, the head-mounted display (HMD) and the Cave automatic virtual environment (CAVE) system. The least immersive of these is the flat-screen computer monitor as it only occupies a small percentage of the participant’s vi-sual field (Repetto, 2014). HMDs provide an immersive experience as they vivi-sually isolate the participant from the real world, creating a sense of full immersion. A step further, the CAVE system allows participants to see their own bodies in motion, adding to their sense of presence (Moore, Wiederhold, Wiederhold & Riva, 2002), while surrounding them with 3 or 4 screens that provide a sensory illusion of a real environment (for a review see Bohil, Alicea

& Biocca, 2011). Both the HMD and the CAVE technologies provide a sense of agency due to the participants’ ability to move their arms and hands freely, as well as manipulate objects

which the participant, immersed within a virtual environment, behaves in accordance with the affordances provided by this environment […]” (Mestre, 2015, p.1). In ecologically valid VR environments, participants are free to interact with objects, while receiving real-time feedback via a graphic rendering system. Input tools (finger trackers, gloves, a mouse or joystick) are often used to record participants’ movements (Burdea & Coiffet, 2003). Given the interactive and immersive nature of VR environments, it has been suggested that the sensory-motor system becomes more fully engaged than in traditional experiments, and that elicited responses are closer to what probably occurs in real life (Bohil et al., 2011).

1.5.2 Combining VR and EEG in language processing and learning studies Within the framework of embodied semantics, the level of immersion, presence and ecological validity provided by VR makes it a very attractive methodology to pair with EEG for studying the interaction of motor and linguistic processing. In VR, multimodal sensory stimulation is fully controlled, rendering possible the direct observation of brain activity as a correlate of specific sensory input, whether visual or auditory, in an environment that allows for naturalistic actions. According to Peeters (2019, p.6), what is most promising about virtual reality as an experimental tool is that it will “shift theoretical focus towards the interplay between different modalities in dynamic and communicative real-world environments, moving beyond and complementing studies that focus on one modality in isolation”.

Virtual reality has been used to replicate results from previous traditional language processing studies. For instance, predictive language processing was shown through language-mediated anticipatory eye movements in a VR environment (Eichert, Peeters & Hagoort, 2018). Another study compared human-human to human-avatar communication using a syn-tactic task and found similar priming effects (Heyselaar, Hagoort & Segaert, 2017). Relevant to embodied semantics, disrupting the visual feedback of participants’ pointing trajectory in VR led to a delay in speech production, showing an interplay between hand movement and speech production mechanisms (Chu & Hagoort, 2014).

For now, however, very few studies have combined cortical measures and virtual reality to investigate language processing. A recent study used time-frequency analysis to investigate competition between gesture representations as participants perceived 3D objects in a VR en-vironment. Results evidenced suppressed mu desynchronization when participants processed conflictual manipulatable objects in peripersonal space, possibly reflecting action selection processes (Wamain et al., 2018). Another study had unbalanced bilinguals name pictures

these results via a greater late central-parietal positivity for switch costs across languages and a late positive effect for the L1 and not the L2 when comparing non-switch to blocked naming trials. In an exploratory study, participants wore a head-mounted VR display as they listened to a sentence (“I just ordered this salmon”) and saw a virtual object that either matched (salmon) or mismatched (pasta) the object in the sentence (Tromp et al., 2018).

A match-mismatch N400 effect was found for incorrectly orally labeled items compared to correctly labeled items. In sum, the combination of cortical measures using EEG and VR has clear potential for addressing questions related to language processing within an embodied framework.

In the work presented in this dissertation, we were especially interested in combin-ing EEG with CAVE and HMD virtual environments to measure the effect of stimulatcombin-ing movement during linguistic processing, in order to provide novel and compelling insight into embodied semantics. This dissertation includes one published study and one registered re-port that make use of this seldom-used combination to investigate language processing and encoding. The study presented in chapter 4 combined a CAVE and EEG to measure motor resonance during single verb processing. In the registered report in chapter 6, participants learn a vocabulary in an L2 using an HMD and a handset to perform real actions on a virtual object in an immersive 3D environment. Motor activation and learning were measured using EEG, pre and post-training.

1.6 Completed Studies

1.6.1 Cross-linguistic gender congruency effects during lexical access in novice L2 learners: evidence from EEG

In chapter 2, we present a currently submitted study that explores the mechanisms un-derlying adult L2 word learning. We created digital games in collaboration with the Mediter-ranean Virtual Reality Center (CRVM) to teach a new lexicon to completely novice learners over four consecutive days. We observed behavioral and cortical changes during the very first phases of L2 learning. Thus, we were able to follow progression from zero knowledge to the understanding of simple declarative sentences involving an action verb and two nouns. Very few studies have examined the cortical response to the integration of novel words during the very early stages of learning (O’Neil, Lagarrigue, Newman & Frenck-Mestre, unpublished).

(L1) affects the integration an L2 lexicon (Grosjean & Byers-Heinlein, 2018). According to the dominant models of bilingual lexical representation, such as the Bilingual Interactive Activation + model (Dijkstra & Van Heuven, 2002), lexical access is non-selective across a bilingual’s L1 and L2. Grammatical gender, an intrinsic syntactic feature of lexical items in a number of languages, provides an interesting tool for measuring the influence of L1/L2 feature congruity on L2 learning. The first aim of our study was to examine how L1/L2 gender congruency might influence L2 lexical integration from the very initial stages of learning.

Based on previous research showing cross-language gender congruency effects (Sá-Leite, Fraga

& Comesaña, 2019) we hypothesized that if gender is shared across lexical entries, convergence of grammatical gender across the L1 and L2 would have immediate effects on learning. No specific training of gender was provided, hence any gender information acquired was exposure-based and implicit. All language exposure was oral and no L1 support was provided. An effect of gender congruency would suggest that L1/L2 differences interfere in L2 lexical integration during the early stages of L2 learning. Alternatively, if no effects of grammatical gender congruency are found, we could conclude that learners do not access L1 grammatical features such as gender online while treating L2 words, at least during the early stages of learning.

1.6.2 What role does motor activation play in action language processing?

An EEG study

In our subsequent experiments, we took the interactive aspect of language processing a step further by exploring the role of the body in first language processing, and later in-vestigated its influence in L2 learning. In these experiments, we continued to use behavioral measures and ERPs and added time-frequency analysis to measure motor activation during language processing.

In chapter 3, we report our second experiment, which examined the implication of physical movement for language processing. Using an Action-sentence compatibility effect (ACE) paradigm, we manipulated language and action congruency to measure the effect of such on language processing. Given that the direction of the ACE effect – whether facilitatory or inhibitory – is highly dependent on when motor preparation takes place in relation to action language processing (Aravena et al., 2010), we were interested in whether simultaneously planning a movement that was congruent with the action language being understood would interfere with semantic processing. Only one study has looked at the neural response to action language and movement congruency during overlapping motor and semantic processes

and RAP). In our study we likewise observed changes in the N400 time window to examine semantic processing. However, ERPs only show phase-locked activity and are hence not the ideal means of measuring activation in the motor cortex. Therefore, in order to obtain a more comprehensive view of possible interactions between motor and linguistic processing, we also used time-frequency analysis, which takes into account both phase-locked and non-phase-locked activity (Pfurtscheller & Lopes da Silva, 1999).

1.6.3 Motor Resonance during linguistic processing as shown by EEG in a naturalistic VR environment

To further explore embodied effects in language processing, we developed a novel paradigm combining EEG and a CAVE. In the published study presented in chapter 4, we took into consideration theories of situated and grounded cognition, which claim that cognition is strongly constrained by one’s surrounding environment and physical state (Atkinson, 2010).

Participants performed a Go-Nogo task in which they heard action verbs before manipulating virtual objects using real and varied actions, in an ecologically valid VR environment. We used time-frequency to measure motor activation, as evidenced by mu and beta ERD (Niccolai et al., 2014; Pfurtscheller & Lopes da Silva, 1999), during verb processing and prior to movement proper. We also examined ERP language-related components. We hypothesized that if motor activation proved to be contingent on language processing, this would go in the direction of embodied studies demonstrating that action language directly produces motor activation (Klepp, van Dijk, Niccolai, Schnitzler & Biermann-Ruben, 2019). Recording EEG in a VR environment is particularly challenging from a technical viewpoint, given the possibility of crosstalk between systems and physical movement (Török et al., 2014) as well as the need for precise synchronization between stimuli presentation and electrophysiological measures (Chapter 5, Tromp et al., 2018). Besides testing the hypothesis that motor activation is involved in linguistic retrieval, we aimed to provide proof of concept of the combination of EEG and VR.

1.6.4 The neural correlates of embodied L2 learning. Does embodied L2