Article
Reference
Differential parietal activations for spatial remapping and saccadic control in a visual memory task
PIERCE, Jordan Elisabeth, SAJ, Arnaud, VUILLEUMIER, Patrik
Abstract
Remapping is a process that updates visual information in internal spatial representations across eye movements, allowing for stable perception of the environment. Previous work has demonstrated visual remapping activity in parietal cortex during saccades, but it remains unclear whether remapping is triggered by overt saccades only (or by attentional shifts also), and whether it engages parietal areas only (or other cortical areas). Here, we used fMRI to investigate spatial remapping during two visuospatial memory tasks requiring either overt (accompanied by a saccade) or covert (with central fixation) attention shifts to peripheral distracters. Participants had to remember the position and color of a lateralized dot during a saccade or attention shift, requiring them to update the dot position in memory, and then indicate if a second dot matched the first. Differential activation patterns were observed within parietal cortex as a function of the different visual, motor, and interhemispheric remapping demands in the saccade task, presumably mediating the maintenance of spatial position in perceptual and motor maps. [...]
PIERCE, Jordan Elisabeth, SAJ, Arnaud, VUILLEUMIER, Patrik. Differential parietal activations for spatial remapping and saccadic control in a visual memory task. Neuropsychologia, 2019, vol. 131, p. 129-138
PMID : 31102598
DOI : 10.1016/j.neuropsychologia.2019.05.010
Available at:
http://archive-ouverte.unige.ch/unige:121582
Disclaimer: layout of this document may differ from the published version.
1 / 1
1 2 3
Differential parietal activations for spatial remapping and saccadic control 4
in a visual memory task 5
6 7 8 9 10
Jordan E. Pierce1,3, Arnaud Saj2, and Patrik Vuilleumier1,2,3 11
12 13 14
1Department of Neurosciences, University Medical Center, University of Geneva 15
2Department of Neurology, University Hospital of Geneva 16
3Campus Biotech, University of Geneva 17
18 19 20 21 22 23 24
Keywords: remapping, parietal cortex, spatial attention, saccade, fMRI 25
Abstract 26
Remapping is a process that updates visual information in internal spatial representations 27
across eye movements, allowing for stable perception of the environment. Previous work has 28
demonstrated visual remapping activity in parietal cortex during saccades, but it remains 29
unclear whether remapping is triggered by overt saccades only (or by attentional shifts also), 30
and whether it engages parietal areas only (or other cortical areas). Here, we used fMRI to 31
investigate spatial remapping during two visuospatial memory tasks requiring either overt 32
(accompanied by a saccade) or covert (with central fixation) attention shifts to peripheral 33
distracters. Participants had to remember the position and color of a lateralized dot during a 34
saccade or attention shift, requiring them to update the dot position in memory, and then 35
indicate if a second dot matched the first. Differential activation patterns were observed 36
within parietal cortex as a function of the different visual, motor, and interhemispheric 37
remapping demands in the saccade task, presumably mediating the maintenance of spatial 38
position in perceptual and motor maps. Remapping engaged parietal areas adjacent to, but not 39
overlapping with, those activated by saccade execution, while it did not engage the frontal 40
eye fields, pointing to distinct neural substrates for ocular motor and spatial updating 41
processes. No differential activation related to remapping was found during the covert 42
attention shift task, suggesting that this condition did not necessitate the same remapping as 43
the saccade condition. Overall these results further elucidate the mechanisms of spatial 44
remapping in human parietal cortex and their relationship with attention processing and 45
ocular motor behavior, with implications for understanding visuospatial attention deficits in 46
hemispatial neglect.
47
1. Introduction 48
Visual remapping is a process by which the brain maintains spatial information about 49
a stimulus across eye movements. This remapping or updating is necessary to create the 50
perception of a stable visual environment and to plan spatially oriented actions to target 51
objects despite the changing retinal input at successive fixations. Neurons that respond to a 52
stimulus at a given location must therefore transmit information to the neurons that 53
correspond to the post-saccadic location of the stimulus. In nonhuman primates, the parietal 54
cortex (Duhamel, Colby, & Goldberg, 1992), frontal eye fields (Umeno & Goldberg, 1997, 55
2001), and superior colliculus (Walker, Fitzgibbon, & Goldberg, 1995) have been shown to 56
be affected by remapping around the time a saccade is generated, shifting receptive fields 57
based on the internal motor plan of the impending eye movement. All these areas are also 58
directly linked to saccadic eye movement control (Leigh & Zee, 2015; McDowell, Dyckman, 59
Austin, & Clementz, 2008).
60
In parietal cortex, remapping may take place within multiple retinotopic 61
representations of the visual field that are generated based on sensory inputs, motor plans, 62
spatial memory, and top-down goals (Colby, Duhamel, & Goldberg, 1996; Jerde, Merriam, 63
Riggall, Hedges, & Curtis, 2012; Sereno, Pitzalis, & Martinez, 2001; Silver & Kastner, 64
2009). These maps can direct attention to salient stimuli and guide behavior in a changing 65
environment (Connolly, Kentridge, & Cavina-Pratesi, 2016; Lauritzen, D'Esposito, Heeger, 66
& Silver, 2009; Serences & Yantis, 2006; Szczepanski, Konen, & Kastner, 2010). A previous 67
study specifically sought to characterize remapping in the human parietal cortex by 68
investigating the memory trace of a peripheral stimulus following a saccade (Merriam, 69
Genovese, & Colby, 2003). In their task, a saccade from one side of the stimulus location to 70
the other would have caused it to appear in the opposite visual hemifield, but the stimulus 71
was extinguished during the eye movement. Nonetheless, by using functional magnetic 72
resonance imaging (fMRI), activation was demonstrated in parietal cortex in both the 73
hemisphere contralateral to the original stimulus (visual response) and the ipsilateral 74
hemisphere, suggesting that the spatial information was remapped in anticipation of the 75
visual scene after the saccade [see also (Medendorp, Goltz, Vilis, & Crawford, 2003)].
76
In addition to its role in healthy visual functioning, remapping also has been studied 77
in relation to symptoms of spatial neglect in patients with lesions that encompass temporal- 78
parietal cortex or the underlying white matter fibers (Pisella et al., 2011; Pisella &
79
Mattingley, 2004; Vuilleumier et al., 2007). Most commonly, right lateralized lesions result 80
in a left spatial neglect that manifests in numerous ways, including inattention to 81
contralesional space, despite intact early visual cortex [e.g., (Danckert & Ferber, 2006;
82
Mesulam, 1981; Vuilleumier & Saj, 2013)]. The inability to establish continuity of visual 83
objects across saccades via remapping could contribute to patients’ difficulty with spatial 84
awareness including poor visual search and visual spatial memory (Pierce & Saj, 2018;
85
Pisella, Berberovic, & Mattingley, 2004; Verdon, Schwartz, Lovblad, Hauert, & Vuilleumier, 86
2010).
87
In a study by Vuilleumier and colleagues (2007), the role of remapping was tested 88
empirically with a visual short-term memory task in neglect patients with right hemisphere 89
lesions. Patients had to remember the spatial position of a lateralized dot while making a 90
saccade to the far left or right. The results demonstrated a large drop in position memory 91
when patients made a rightward saccade, regardless of the initial position of the dot.
92
Presumably this effect was caused by a deficit in remapping the dot position leftward in 93
internal maps, an ability relying on damaged right parietal cortex (Vuilleumier et al., 2007).
94
However, brain lesions associated with neglect are usually large, often extending beyond 95
parietal cortex to other areas and connections with distant regions in frontal and occipital 96
cortex, as well as the thalamus (Bartolomeo, Thiebaut de Schotten, & Chica, 2012; Vaessen, 97
Saj, Lovblad, Gschwind, & Vuilleumier, 2016; Verdon et al., 2010), precluding definitive 98
conclusions about the exact neuroanatomical substrates of spatial remapping.
99
In the current study, healthy participants were tested with fMRI on a similar 100
visuospatial memory task requiring an intervening saccade. They had to remember the 101
position and color of a dot (i.e., spatial and perceptual features, respectively) appearing either 102
left or right of central fixation, then make a saccade to identify a peripheral letter, and finally 103
indicate whether a second dot matched the first dot’s position and color. The saccade 104
putatively initiated remapping processes in order to transfer and maintain the first dot’s 105
location accurately in internal spatial representations, either within or across visual 106
hemifields. Thus, comparing behavioral performance for location and color features, we 107
could probe for any differential impact of interhemispheric remapping on spatial as opposed 108
to non-spatial features.
109
More critically, at the neural level, activations within parietal cortex corresponding to 110
interhemispheric remapping processes were of particular interest, both in support of previous 111
neuroimaging work in healthy humans and in relation to lesion localization in neglect 112
patients. Interhemispheric remapping requires transfer of position information from the 113
cortical spatial representation of one visual hemifield into the opposite visual hemifield, 114
constituting a greater reorganization than intrahemispheric remapping that requires only a 115
shift in position within the same visual hemifield. Clarifying the role of right parietal cortex 116
in remapping following rightward or leftward saccades should provide novel insight into why 117
neglect patients might show deficits in certain tasks but not others, and open new 118
perspectives for rehabilitation procedures. We hypothesized that our healthy participants 119
would be able to perform the task with relatively few errors, and that fMRI activation during 120
the task would reveal brain regions involved with visuospatial memory and interhemispheric 121
remapping of a target location, including posterior parietal cortex and frontal eye fields 122
(FEF).
123
In addition to the saccade remapping task, our participants performed a second task 124
where they had to maintain central fixation while a distracter stimulus was flashed in the 125
periphery, putatively causing an exogenously-driven covert attention shift. This manipulation 126
was intended to investigate whether such an attention capture would be enough to initiate 127
remapping in the absence of a saccade. It was expected that behavioral performance would be 128
better and remapping activation would be weaker or absent compared to the saccade task, as 129
remapping may not be needed when the dot’s position is not altered by intervening eye 130
movements. On the other hand, since covert spatial attention shifts and overt eye movements 131
recruit partly overlapping areas in parietal and frontal cortex (Corbetta, 1998; Rizzolatti, 132
Riggio, Dascola, & Umilta, 1987; Steinmetz & Moore, 2014), it is possible that some 133
remapping also takes place during shifts of attentional focus without overt eye movements. In 134
particular, the FEF are critically implicated in both voluntary saccadic control and 135
endogenous attention orienting (Corbetta & Shulman, 2002; Grosbras, Laird, & Paus, 2005;
136
McDowell et al., 2008), and were reported to be modulated by remapping during saccadic 137
tasks in the monkey (Umeno & Goldberg, 1997, 2001).
138
Therefore, as a final comparison, participants also completed a saccade functional 139
localizer task to determine how remapping activation in the visuospatial memory task related 140
to that from a simple ocular motor task. Indeed, saccadic control also relies on parietal and 141
frontal regions implicated in spatial attention and the neglect disorder (McDowell et al., 142
2008; Mesulam, 1999; Pierce & McDowell, 2017). Taken together, these tasks should yield 143
novel insights into the role of parietal cortex and other areas (e.g., FEF) in perisaccadic 144
remapping processes in humans and aid in understanding how brain lesions can lead to 145
symptoms of visuospatial neglect.
146
147
2. Methods 148
2.1 Participants and procedure 149
Thirty individuals were recruited from the University of Geneva and paid 20 CHF for 150
their participation (11 males, mean age = 25.7 years, SD = 4.1). After providing written 151
informed consent, participants completed a brief practice version of each task before entering 152
the MR environment. In the scanner, participants viewed the stimuli on an LCD screen 153
located at the back of the scanner via a mirror attached to the head coil, and made responses 154
using an MR-compatible button device (HH-1 × 4-CR, Current Designs Inc., USA). All 155
experimental procedures were approved by the local ethics committee.
156
2.2 Task design 157
Participants completed two versions of the task in separate scans: the first required an 158
overt saccade to a peripheral letter, while the second required maintenance of central fixation 159
despite a peripheral checkerboard distracter. Both tasks began with a black central fixation 160
cross on a medium gray background (500 ms), followed by the addition of a colored dot 161
(1000 ms) either left or right of fixation. The dot could be colored red, green, or blue and 162
appeared pseudorandomly within an area 3° high extending +/-5° to 15° from center (to 163
minimize verbal encoding of position as e.g., “far”, “near”, “middle” for discrete locations).
164
After 1750 ms (for the peripheral distracter stimuli described below), another dot appeared, at 165
either the same or a different position (always the same direction from center) in the same or 166
a different color. Participants then had to respond with a button press whether the position 167
and color of the two dots matched using successive question screens (i.e., Position: same or 168
different? Color: same or different?) in order to probe for any differential behavioral recall 169
performance for spatial and perceptual features, respectively.
170
The peripheral stimulus for the first task consisted of a letter (D, C, Q, or O) 171
appearing at the far left or right of the screen (20°), or in the center. Participants were 172
required to make a saccade to the letter to ascertain and remember its identity (which they 173
reported at the end of each trial), and then return gaze to the center of the screen. The 174
stimulus for the second task consisted of a black and white checkerboard (left, right, or center 175
of the screen) that flashed with inverting contrast for 250 ms, which the participant was 176
instructed to ignore while maintaining central fixation. While the visual content of peripheral 177
stimuli differed between the two tasks, these were chosen to maximize the engagement of 178
precise foveation and exogenous orienting, respectively, and should cancel out in the critical 179
contrasts of interest for remapping that involve comparisons within (not between) tasks (see 180
below). The trials for both tasks were blocked according to the location of the peripheral 181
stimulus with 48 trials in each run. The first half of participants performed each task once and 182
the second half of participants performed each task twice (in counterbalanced order in all 183
cases).
184
Finally, after the remapping tasks, participants also performed a separate saccade 185
localizer task in which colored dots appeared around a central fixation cross. The task 186
alternated between fixation blocks (12 sec), horizontal (5, 10, or 15° left/right of center) dot 187
blocks (18 trials x 3 blocks), and vertical (4, 8, 12° above/below center) dot blocks (18 trials 188
x 3 blocks). For each saccade trial the colored dot appeared for 1000 ms followed by 500 ms 189
of fixation.
190
During all tasks, eye movements were tracked by an infrared eye tracking system 191
(ASL 450, 60 Hz sampling rate). Eye position was monitored continuously in all participants 192
during fMRI acquisition and data were recorded for offline analysis using custom Matlab 193
scripts to ensure general adherence to the given instructions. As fixation was well maintained 194
overall, trials were not excluded from fMRI analyses based on eye tracking data. However, 195
trials were individually scored for number of saccades (when the change in eye position 196
exceeded 3°), and we excluded from the eye tracking analysis those with loss of signal due to 197
artefacts/technical recording issues or frequent blinks. Based on these criteria, eye tracking 198
data from 18 participants were included in the analysis of task compliance.
199
2.3 Behavioral analysis 200
The button press responses for the position, color, and letter questions were measured 201
for percent of correct responses. The position question was the main variable of interest to 202
assess spatial memory and remapping, while the color and letter questions served as controls 203
to check that participants were maintaining attention to the task and making saccades to 204
identify the letter. Responses for the position and color questions were entered into 3x2x2 205
repeated measures ANOVAs [distracter location (left/right/center x dot location (left/right) x 206
task (letter/checkerboard)], and responses for the letter question were entered into a 3x2 207
ANOVA (distracter x dot location). Analyses were conducted in SPSS version 23 (IBM 208
Corp., Armonk, NY), with significant differences identified at p<.05.
209
2.4 MRI parameters 210
All participants were scanned at the Brain and Behaviour Laboratory at the University 211
of Geneva on a Siemens 3T MR scanner (Trio Tim, Siemens Medical Solutions, Erlangen, 212
Germany) using a 12-channel head coil. Scans included a T1-weighted structural image (3D 213
MPRAGE, matrix size = 256 × 246, 192 slices, voxel size = 0.9 mm3, flip angle = 9°, 214
repetition time (TR) = 1900 ms, inversion time = 900 ms, echo time (TE) = 2.32 ms) and 215
functional T2*-weighted echo planar imaging scans to assess blood oxygenation level 216
dependent (BOLD) activation (matrix size = 64 x 64, 36 oblique slices, descending 217
acquisition, voxel size = 3.2 mm3, slice gap = 0.6 mm, flip angle = 81°, TR = 2200 ms, TE = 218
30 ms). 209 volumes were collected for the saccade letter task, 164 volumes for the 219
checkerboard task, and 120 volumes for the saccade localizer.
220
2.5 MRI analysis 221
Functional images were processed using Statistical Parametric Mapping 12 (SPM12, 222
Wellcome Trust Centre for Neuroimaging, University College London, UK) running in 223
MATLAB R2015b (Mathworks, Natick, MA, USA). Preprocessing steps included slice 224
timing correction, realignment to account for subject motion, coregistration with the subject’s 225
anatomical image, normalization to a standard MNI template, and spatial smoothing with an 226
8 mm FWHM kernel. Functional images were resliced to a 4 mm3 voxel grid.
227
Individual data from the remapping tasks were entered into a first-level general linear 228
model with six regressors of interest for each of the two tasks: left/right/center position of the 229
letter/checkerboard by left/right position of the colored dot. Only trials on which participants 230
responded correctly for the position question were modelled to ensure that the dot’s spatial 231
information was maintained through the trial. Error trials and the six motion regressors for x, 232
y, and z shift and rotation were entered as additional nuisance variables and the data were 233
high pass filtered with a 128 second cut-off. (The first 15 participants performed each task 234
only once, while the last 15 completed each task twice. This resulted in different first level 235
models, but each subject contributed the same number of images to the group analyses.) All 236
participants’ contrast images for each task condition versus baseline were then entered into a 237
group ANOVA. A flexible factorial model was used to perform group analysis with random 238
effects statistics and probe for specific effects of interest. Significance levels for main effects 239
were set to p<.05 (family-wise error corrected), with a minimum cluster size of 5 voxels.
240
Given our hypotheses, significance levels for the remapping interaction were set to a more 241
liberal p<.001 (uncorrected) at the peak level, with a minimum cluster size of 10 voxels, to 242
ensure that differences in activation locations across conditions were detectable and not 243
driven by trivial variability in peak response location. Brain maps were produced using the 244
xjView toolbox (http://www.alivelearn.net/xjview).
245
The saccade localizer task data was entered into a first-level model for each 246
participant with regressors for fixation, horizontal saccade blocks, and vertical saccade 247
blocks, as well as the six motion parameters. A group level t-test was performed for the 248
comparison of horizontal saccades versus fixation using a random-effects flexible factorial 249
design. To further investigate the role of remapping in canonical saccade regions, contrast 250
values for the letter task were extracted from four regions of interest (ROIs) defined by 251
significant clusters in the saccade localizer task: the left and right FEF and superior parietal 252
lobule (SPL). A repeated-measures ANOVA was conducted in SPSS for each ROI comparing 253
the activation during the four conditions based on left/right letter and dot location.
254 255
3. Results 256
3.1 Behavior 257
On each trial, participants had to report whether the location and the color of the 258
second dot appearing after the peripheral distracter matched the first dot presented at the 259
beginning of the trial. Accuracy was analyzed for these two questions separately. The 3x2x2 260
(distracter location x dot location x task) ANOVA for responses on whether the position of 261
the two dots was the same or different, in both the saccade and checkerboard tasks, revealed a 262
main effect of task (F(1,29)=9.4, p=.005). Participants had overall higher percent accuracy 263
scores in the checkerboard task (Table 1), indicating that, as expected, spatial location 264
memory was harder to maintain after making a saccade than a covert shift of attention. In 265
contrast, the ANOVA for responses on whether the color of the two dots was the same or 266
different showed no such main effect of task, with performance near ceiling in both 267
conditions (all >96% correct, see Table 1), suggesting that color memory unlike location 268
memory was not differentially affected by an intervening saccade. However, there was an 269
interaction between dot location and task (F(1,29)=4.48, p=.043), where the letter task had 270
higher percent correct for right dot location trials, while the checkerboard task had higher 271
percent correct for left dot location trials.
272
In addition, in the saccade task, participants had to report the identity of the letter (D, 273
C, Q, or O) presented in the right or left periphery or at central fixation. The ANOVA for 274
responses on this question revealed a main effect of distracter location (F(2,58)=9.43, 275
p<.001), with higher percent correct for center distracter trials (when the letter was presented 276
at fixation) than for left or right distracter trials (when the letter discrimination required a 277
saccade).
278
Eye tracking data were recorded and compared across the different task conditions to 279
ensure that subjects were not making additional inappropriate saccades that could influence 280
remapping processes. An analysis (N=18) of eye position and number of saccades within 281
individual trials of each condition indicated that gaze was maintained at central fixation 282
equally well across all conditions in the checkerboard task and equally well during all trials in 283
the checkerboard task versus center trials in the letter task (all t<2, p=n.s.), when no saccades 284
should have been generated (overall mean = 0.61 saccades (SD=.68)). Eye tracking data also 285
confirmed that participants correctly executed saccades (overall mean = 1.61 (SD=.58)) to the 286
peripheral distracter location in the letter task (right/left location versus center location trials, 287
t(17)=9.5, p<.001). The average number of trials scored for the first half of participants was 288
10.5 trials (SD=5.0) out of 16 per condition, and for the second half of participants 26.4 trials 289
(SD=7.6) out of 32 per condition.
290 291
3.2 Functional MRI data – main effects 292
In the fMRI data, we first examined stimulus and task related responses regardless of 293
remapping. The main effect of dot location (right or left visual field) on functional activation 294
in the letter and checkerboard tasks revealed strong and symmetric activations in occipital 295
cortex, contralateral to the dot location (Figure 2A/Table 2), consistent with sensory-driven 296
responses. The main effect of task (letter versus checkerboard) revealed clusters with greater 297
activation in the right precuneus, left middle frontal gyrus, and left cerebellum for the 298
checkerboard task, and in right medial occipital lobe for the letter task (Table 2), which may 299
be related to differences in visual input between the two tasks.
300
The main effect of distracter location was evaluated separately for the two tasks, since 301
visual differences in peripheral stimuli did not allow for direct comparison between tasks. For 302
the checkerboard task, where fixation had to be maintained centrally, differential activation 303
for left- and right-side trials was observed in symmetrical clusters in the lingual gyrus and 304
cuneus of the contralateral hemisphere (Table 2), consistent with retinotopic visual responses.
305
For the letter task, which required overt saccades, greater activation was observed for left- 306
side trials in the right posterior parietal and right inferior frontal cortex, and for right-side 307
trials in the left occipital cortex only (Figure 2B/Table 2).
308 309
3.3 Functional MRI data – remapping effects 310
Crucially, we next examined remapping-related effects by computing the interaction 311
between dot location and peripheral distracter location (R dot/R distracter + L dot/L distracter 312
> R dot/L distracter + L dot/R distracter), for each task separately, given the differences in 313
visual input for each task. Such an interaction should reveal regions that were activated 314
during interhemispheric remapping of the dot position across the two hemifields and 315
presumably into a new cortical representation (e.g., dot in left visual field moving to right 316
visual field after leftward saccade), more than during remapping within one visual hemifield 317
(e.g., dot in left visual field remaining in left visual field after rightward saccade). For the 318
letter task, we found significant clusters in bilateral posterior parietal cortex, corresponding to 319
brain areas commonly implicated in visuospatial processing (Corbetta & Shulman, 2002;
320
Duhamel et al., 1992; Hopfinger, Woldorff, Fletcher, & Mangun, 2001; Jerde et al., 2012;
321
Szczepanski et al., 2010), but also additional clusters in bilateral precentral and right 322
inferior/middle prefrontal cortex (Figure 3/Table 3). Neural responses in the parietal cortex 323
showed greater activation in both the left and right hemisphere for R dot/R letter and L dot/L 324
letter trials (requiring interhemispheric remapping) than for the L dot/R letter and R dot/L 325
letter trials.
326
For the checkerboard task, however, there were no such effects in parietal or frontal 327
areas, even at the lower uncorrected thresholds. Significant clusters were found only in 328
bilateral extrastriate visual areas including fusiform and lingual gyri (Table 3). These 329
occipital areas did not show an interaction pattern reflecting interhemispheric remapping, but 330
instead a strong visual response for contralateral visual inputs (R dot/R checker trials, see 331
Figure 3B).
332 333
3.4 Functional MRI data – remapping in saccadic control areas 334
Finally, we directly tested whether remapping effects during the letter task involved 335
similar brain areas as those implicated in saccadic control, by comparing activations evoked 336
during remapping with those engaged during a saccade localizer task (Figure 4). The saccade 337
localizer task identified the expected canonical ocular motor regions (Leigh & Zee, 2015;
338
McDowell et al., 2008; Pierce & McDowell, 2016) including bilateral posterior parietal 339
cortex (SPL) and middle frontal gyrus (putative FEF), in addition to lingual gyrus and middle 340
occipital gyrus (Figure 4/Table 3). Within parietal cortex, the peak saccade localizer and 341
remapping interaction regions recruited nearby regions in both hemispheres, but these did not 342
overlap (no voxels were common to all three statistical maps at p<.001). Remapping effects 343
were located slightly anterior to saccadic effects on both sides (Figure 4), with only 3 344
overlapping voxels in right parietal cortex (out of 70 in the saccade localizer cluster, see 345
Table 3) and 5 overlapping voxels (out of 59) in left parietal cortex. Remarkably, prefrontal 346
areas engaged by saccades showed very weak and limited modulation by remapping 347
conditions, even when tested at a liberal / low threshold. Only 10 voxels (out of 131) in the 348
left lateral FEF were activated during remapping and shared with the saccade localizer task, 349
but no remapping voxels overlapped with saccade activation in right frontal cortex.
350
Activation in these saccade localizer regions during remapping in the letter task was 351
explored further by extracting contrast values from four ROIs in the left FEF, right FEF, left 352
SPL, and right SPL (Figure 5). In three of these four ROIs, the ANOVAs indicated no 353
significant effects of dot or letter location, nor an interaction effect (all F<2.5, p=n.s.). In the 354
left FEF, there was a significant effect of dot location (F(1,29)=3.7, p<.01), with greater 355
activation for contralateral right dot trials, but no remapping interaction.
356
Additionally, remapping activations were compared to the distracter location effect 357
activations during the letter task (which required saccades to the left vs. right). In parietal 358
cortex, the remapping interaction clusters were more anterior and medial than the letter 359
location clusters, with only 8 (out of 125) overlapping voxels in right parietal cortex and 11 360
(out of 96) overlapping voxels in left parietal cortex. These separate, but adjacent, parietal 361
clusters for each task suggest that distinct representations or functional modules likely were 362
being utilized in each condition.
363 364
4. Discussion 365
In this study, healthy young adults completed a visuospatial memory task with an 366
intervening saccade or flashed peripheral checkerboard while functional MR images were 367
collected. The saccade and overt attention shift to the periphery during the letter task 368
necessitated visual interhemispheric remapping processes to update the spatial location of the 369
to-be-remembered dot in short-term memory representations. However, such remapping 370
seemingly did not occur after covert attention shifts during the checkerboard task, where 371
fixation had to be maintained and no saccade was performed. Our behavioral results indicated 372
that participants could answer correctly whether a second dot was in the same position as the 373
first, regardless of where the distracting letter or checkerboard appeared, but showed a cost in 374
memory for location (but not for color) during the saccade task only. Critically, fMRI data 375
revealed differential activation patterns within parietal cortex corresponding to the different 376
visual, motor, and remapping demands in the letter task, while no differential activation 377
related to spatial remapping was found during the checkerboard task in either parietal or 378
frontal cortices.
379 380
4.1 Behavioral costs of an intervening saccade 381
Behavioral performance on memory for the dot position showed no significant 382
difference in percent correct based on distracter or dot location, indicating that the 383
participants were able to maintain the dot position in spatial memory accurately throughout 384
the task regardless of its original or remapped location. The percent of correct responses was, 385
however, lower overall in the letter than checkerboard task, indicating extra processing 386
demands when position memory had to be maintained during an intervening saccade as 387
compared with a condition where only an exogenous shift in attention was required. These 388
additional demands presumably involved remapping and working memory processes 389
following the saccade during the retention interval in the letter task that were not necessary in 390
the checkerboard task. Importantly, this remapping cost was selective to location memory, 391
whereas recalling the dot color was not affected by the intervening saccade, suggesting that 392
these costs were not due to non-specific interference or task load but restricted to spatial 393
representations.
394 395
4.2 Functional asymmetry for left versus right letter location 396
In the fMRI data, an expected activation of contralateral occipital areas was observed 397
in the comparison between left and right dot locations across both tasks, and for the 398
comparison of left and right distracter locations in the checkerboard task. The distracter effect 399
in the letter task, however, showed an asymmetry between left and right locations with 400
stronger right parietal activation for left letter trials (that became more bilateral at the lower 401
threshold used in the overlap analysis, Fig. 4). This asymmetry may be consistent with a right 402
hemisphere advantage in spatial processing where right parietal areas control the distribution 403
of attention and eye movements towards both visual hemifields, while left parietal areas 404
control those towards the contralateral right hemifield only (Heilman & Valenstein, 1979;
405
Mesulam, 1981; Schwartz et al., 2005), such that left-sided trials led to higher engagement of 406
the right parietal cortex in our task. These findings also converge with other results indicating 407
a bihemispheric left-hemifield superiority in the activation of parietal attention networks, 408
attributed to stronger initial recruitment when a left visual field stimulus is sent from right to 409
left parietal cortex, but a weaker response to right visual field stimuli in both hemispheres 410
(Siman-Tov et al., 2007). However, it is also possible that this higher parietal activation 411
reflects general attention or working memory demands that were greater for left than right 412
letter trials, perhaps related to other hemispheric differences in linguistic processing of the 413
letter itself.
414 415
4.3 Parietal activations for spatial attention, visual remapping, and motor generation 416
The most critical comparison in our study concerned remapping effects on trials when 417
the dot and peripheral distracter were on the same side of the screen during the letter task 418
(requiring a saccade towards the distracter and hence the transfer of the to-be-remembered 419
dot position from one visual hemifield into the other) versus trials when the dot and letter 420
were on opposite sides of the screen (requiring remapping only within the same visual 421
hemifield). For example, a dot on the right side initially would activate right visual field 422
representations (presumably held in the left hemisphere), but when followed by a saccade to 423
the far right side the remembered dot location would then correspond to a point now located 424
to the left of current gaze position (at the letter), which would require the dot memory to be 425
transferred temporarily to left visual field representations (presumably held in the right 426
hemisphere). Such transfer therefore would necessitate remapping across the two 427
hemispheres. In line with our predictions, the fMRI results showed a significant remapping 428
effect in bilateral parietal cortex with stronger activation for these across-visual-field transfer 429
trials relative to the within-visual-field trials, specifically in the letter task. This effect does 430
not correspond to a simple visual or motor response, where stronger activation would be 431
expected for trials with contralateral stimuli. Furthermore, this remapping activity was 432
spatially separate (i.e., located in slightly more anterior parietal cortex, see below) from both 433
the letter location effects (visual) and the saccade localizer (motor) effects. In addition, 434
remapping activations were similar in both the left and right parietal clusters, perhaps 435
reflecting a combination of visual (contralateral) and remapping (ipsilateral) responses 436
recruiting both hemispheres regardless of the direction of remapping.
437
In contrast to the interhemispheric remapping pattern observed here, we found no 438
significant BOLD increase for the within-hemifield remapping conditions (right dot/left letter 439
and left dot/right letter). This suggests that remapping across neurons within the same cortical 440
area/same hemisphere did not produce differential activity, and converges with the notion 441
that interhemispheric transfer might be a critical condition that is disrupted after parietal 442
damage in neglect patients (Pierce & Saj, 2018; Pisella & Mattingley, 2004; Vuilleumier et 443
al., 2007).
444
In an earlier study of visual remapping in human parietal cortex, Merriam and 445
colleagues (2003) also reported interhemispheric remapping activation of a single stimulus 446
memory trace. The current study involved a more complex paradigm with different visual, 447
motor, and memory components than in this previous work, making it difficult to compare 448
the two studies. Indeed, the saccade by visual field interaction effect we observed may reflect 449
more than remapping processes, as these effects are potentially weaker and more transient 450
than visual or motor responses. The additional components of our task, however, allowed us 451
to examine how remapping mechanisms integrate with other cognitive functions and extend 452
the findings of Merriam et al. (2003) by directly comparing remapping across versus within 453
the visual hemifields. Interestingly, the current pattern of responses suggests that 454
interhemispheric transfer of the dot position occurred selectively in posterior parietal regions, 455
adding support to the results of Merriam et al. (2003).
456
Another important and novel finding in our study was that, even though remapping 457
activity was closely neighboring saccade-related activity in parietal cortex, these processes 458
appeared to be mediated, at least partly, by anatomically segregated areas. Indeed, cortical 459
regions modulated by across-hemifield remapping showed no (or very little) overlap with 460
cortical regions that were activated during saccadic execution, but consistently were located 461
in slightly more anterior, abutting portions of parietal cortex. It is unlikely that these 462
differences reflect different eccentricities of the to-be-remembered dots and saccade targets, 463
because it has been shown that even much larger eccentricity differences (4° vs 30°) do not 464
result in any univariate BOLD differences in parietal cortex (Grosbras, 2016). Furthermore, 465
although we were unable to separate saccade localizer trials based on saccade direction, the 466
bilateral remapping activation observed was distinct from the combined ocular motor 467
activation from right and left saccades, which would not be expected to differ spatially if 468
right and left saccades were analyzed separately.
469
These distinct parietal remapping activation patterns could represent visual salience 470
and top-down priority maps that were influenced to different degrees by sensory, motor, and 471
attentional processes (Bogler, Bode, & Haynes, 2011; Gottlieb, 2002; Silver & Kastner, 472
2009). For example, in a previous study, the intraparietal sulcus was associated with a 473
contralateral visual response, spatial memory, and saccade planning within a single task 474
(Medendorp, Goltz, & Vilis, 2006). In addition, it has been suggested that posterior parietal 475
areas might hold a sensory representation of visual space primarily based on bottom-up 476
inputs, while more anterior parietal areas might integrate these representations with top-down 477
factors related to attention and motor demands of the task (Bogler et al., 2011; Vuilleumier, 478
2013). The relatively anterior location within parietal cortex for the current remapping effects 479
(compared to the saccadic and letter location effects; red vs. green and blue in Fig. 4) could 480
therefore correspond to a less sensory and more top-down representation of visual space, with 481
the first dot location being reactivated by working memory rehearsal or modulated by ocular 482
motor corollary signals during the interval until the second dot appeared. Future research 483
using high-field MRI or intracranial cortical recordings will be useful to better delineate 484
functional subregions within parietal cortex and clarify their role in different spatial, 485
attentional, and/or ocular motor processes.
486 487
4.4 The role of frontal cortex 488
Remarkably, most of the FEF voxels identified by the saccade localizer task did not 489
reach significance in the remapping conditions, suggesting this region may not be as central 490
to visuospatial remapping as more posterior cortex. This result was confirmed with our ROI 491
analysis of activation during the letter task in four saccade-related regions, showing no 492
significant remapping interaction in FEF or SPL. A lack of FEF activity for remapping is 493
consistent with a study using a delayed saccade task (Curtis, Rao, & D'Esposito, 2004) which 494
found that a specific motor code was maintained in frontal ocular motor regions, whereas 495
stimulus location without a specific saccade target was maintained in parietal cortex.
496
Although participants performed a saccade to the letter in our study, the salient information 497
being held in spatial memory was the dot location, which should not correspond directly to a 498
motor plan in FEF. We did observe activation of another frontal cluster in the right 499
middle/inferior frontal gyrus, however, which showed a selective difference in activation for 500
the letter remapping effect. This region may be related to supervisory control of the spatial 501
transformation in parietal cortex, perhaps specifically during the more demanding condition 502
with interhemispheric remapping.
503 504
4.5 Remapping during an exogenous attention shift 505
In contrast to the effects observed in the letter task, the checkerboard task did not 506
show any significant remapping activation differences in parietal cortex when testing for the 507
critical interaction of dot location by distracter side. This suggests that the presumed 508
exogenous attentional capture by a peripheral distracter and the resulting covert attention shift 509
in this task did not initiate remapping of the dot location within cortical maps of space, or that 510
this process generated much weaker neuronal activity as compared to the saccade task (that 511
we were underpowered to detect). In terms of behavioral responses, the checkerboard task 512
also showed no evidence for spatial memory costs in remapping conditions, with higher 513
percent correct overall relative to the saccade task. The comparison between the two tasks, 514
however, showed a stronger fMRI response for the checkerboard task in a number of brain 515
areas including right precuneus, left cerebellum, and left middle frontal gyrus, whereas 516
stronger activation for the letter task was found in right cuneus. Increased saccadic inhibition 517
and inhibition of a visual response to the checkerboard may account for these differences, 518
which also imply that the remapping differences seen in the letter task are not simply a 519
function of greater overall activation for a more demanding task.
520
The performance differences between the two tasks contrast with a previous 521
behavioral study of spatial working memory utilizing a saccade and covert attention shift 522
paradigm (Vasquez & Danckert, 2008). Their study assessed remapping costs to spatial 523
memory when matching a probe to a previous target location, and they reported impaired 524
performance following a saccade, particularly for trials requiring remapping into right space 525
(leftward saccade). Interestingly, these costs were even greater when the task required a 526
covert attention shift, unlike in the current study. In their paradigm, however, the spatial 527
memory assessment for the covert attention task was made at the peripheral location to which 528
attention was shifted, with the probe presented relative to the new center of attention and not 529
at the original physical location of the target on the screen like in their saccade task and the 530
current study. This sustained shift of spatial attention would have required complete 531
remapping of the remembered visual stimulus to the new location and, therefore, presumably 532
necessitated greater effort and produced larger behavioral costs as compared with their 533
saccade task. In contrast, the transient checkerboard distracter in the current study may have 534
been too brief to elicit this same full remapping, particularly because participants knew that 535
the dot location would need to be recalled relative to central fixation.
536
Another means of examining covert attention shifts and their effects on remapping 537
processes is to study patients with spatial neglect. A recent study by Saj and colleagues (Saj 538
et al., under review) used a checkerboard task similar to the current design with stroke 539
patients who had left visual neglect, and reported poorer performance when covert attention 540
shifts occurred compared to no attention shift. Notably, it was the right-sided checkerboard 541
trials (remapping into left, neglected space) that resulted in the greatest behavioral costs in 542
these patients, comparable to the deficits seen for overt saccades in Vuilleumier et al. (2007).
543
Their right hemisphere lesions evidently disrupted the remapping of the dot location in 544
memory and made the additional demand of attending to (but not looking at) the peripheral 545
checkerboard particularly difficult for these patients, even though the young, healthy 546
participants in our current study could perform this task easily. Further research is needed to 547
determine whether any remapping effect associated with attentional shifts might be subserved 548
by regions outside posterior parietal cortex, which could not be identified with the current 549
task design but may be damaged by stroke lesions in neglect patients. It would also be of 550
interest to investigate whether patients with left hemisphere lesions exhibit similar remapping 551
deficits in this task, possibly in the opposite direction to patients with right hemisphere 552
lesions, even in the absence of consistent neglect symptoms.
553 554
4.6 Conclusions 555
In the current study, healthy participants completed a visuospatial memory task with 556
an intervening saccade that necessitated remapping of salient spatial information. The fMRI 557
results indicated differential parietal cortex activation for trials requiring remapping of a 558
visual dot location across versus within visual hemifields (i.e., interhemispheric transfer), but 559
also greater parietal recruitment for trials with a left versus right letter location (i.e., saccade 560
direction). Notably, these clusters were distinct from ocular motor parietal activation evoked 561
in a separate saccade localizer task. These different parietal responses may form distinct 562
internal representations of space that correspond to successive stages of visuospatial 563
processing from perception to selective attention to maintenance to motor output.
564
Furthermore, no remapping activity was found in frontal areas in or around FEF.
565
The location of the current remapping results primarily in parietal cortex, rather than 566
frontal cortex, has relevance to studies of spatial neglect syndrome. Neglect patients with 567
lesions affecting parietal cortex (and the underlying white matter) could be affected more 568
strongly by remapping deficits that disrupt their ability to maintain spatial information. This 569
would be consistent with neuropsychological findings that neglect in line bisection and text 570
reading is more severe in patients with parietal versus frontal lesions (Binder, Marshall, 571
Lazar, Benjamin, & Mohr, 1992; Verdon et al., 2010), and with the notion that these tests 572
make greater demands on stable spatial codes and remapping as compared to other neglect 573
tests (Saj, Verdon, Vocat, & Vuilleumier, 2012). Future work on visuospatial working 574
memory in neglect may further highlight specific remapping deficits that correlate with 575
particular lesion locations and lead to more targeted rehabilitation approaches. Remapping 576
processes help establish visual stability across saccades and the current results demonstrate 577
that parietal cortex critically contributes to updating and maintaining spatial representations 578
for perception and action.
579 580 581 582
5. Acknowledgements 583
This work was supported by grants from the Swiss National Science Foundation 584
(SNSF, grant no. 166704 to PV and no. 162744 to AS).
585
6. References 586
Bartolomeo, P., Thiebaut de Schotten, M., & Chica, A. B. (2012). Brain networks of 587
visuospatial attention and their disruption in visual neglect. Front Hum Neurosci, 6, 588
110. doi:10.3389/fnhum.2012.00110 589
Binder, J., Marshall, R., Lazar, R., Benjamin, J., & Mohr, J. P. (1992). Distinct syndromes of 590
hemineglect. Arch Neurol, 49(11), 1187-1194.
591
Bogler, C., Bode, S., & Haynes, J.-D. (2011). Decoding Successive Computational Stages of 592
Saliency Processing. Current Biology, 21(19), 1667-1671.
593
doi:doi.org/10.1016/j.cub.2011.08.039 594
Colby, C. L., Duhamel, J. R., & Goldberg, M. E. (1996). Visual, presaccadic, and cognitive 595
activation of single neurons in monkey lateral intraparietal area. J Neurophysiol, 596
76(5), 2841-2852.
597
Connolly, J. D., Kentridge, R. W., & Cavina-Pratesi, C. (2016). Coding of attention across 598
the human intraparietal sulcus. Exp Brain Res, 234(3), 917-930.
599
Corbetta, M. (1998). Frontoparietal cortical networks for directing attention and the eye to 600
visual locations: Identical, independent, or overlapping neural systems? Proc Natl 601
Acad Sci U S A, 95(3), 831-838.
602
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven 603
attention in the brain. Nat Rev Neurosci, 3(3), 201-215. doi:10.1038/nrn755 604
Curtis, C. E., Rao, V. Y., & D'Esposito, M. (2004). Maintenance of Spatial and Motor Codes 605
during Oculomotor Delayed Response Tasks. The Journal of Neuroscience, 24(16), 606
3944-3952. doi:10.1523/jneurosci.5640-03.2004 607
Danckert, J., & Ferber, S. (2006). Revisiting unilateral neglect. Neuropsychologia, 44(6), 608
987-1006. doi:10.1016/j.neuropsychologia.2005.09.004 609
Duhamel, J. R., Colby, C. L., & Goldberg, M. E. (1992). The updating of the representation 610
of visual space in parietal cortex by intended eye movements. Science, 255(5040), 90- 611
92. doi:DOI: 10.1126/science.1553535 612
Gottlieb, J. (2002). Parietal mechanisms of target representation. Current opinion in 613
neurobiology, 12(2), 134-140.
614
Grosbras, M. H. (2016). Patterns of Activity in the Human Frontal and Parietal Cortex 615
Differentiate Large and Small Saccades. Front Integr Neurosci, 10, 34.
616
doi:10.3389/fnint.2016.00034 617
Grosbras, M. H., Laird, A. R., & Paus, T. (2005). Cortical regions involved in eye 618
movements, shifts of attention, and gaze perception. Hum Brain Mapp, 25(1), 140- 619
154. doi:10.1002/hbm.20145 620
Heilman, K. M., & Valenstein, E. (1979). Mechanisms underlying hemispatial neglect.
621
Annals of Neurology: Official Journal of the American Neurological Association and 622
the Child Neurology Society, 5(2), 166-170.
623
Hopfinger, J. B., Woldorff, M. G., Fletcher, E. M., & Mangun, G. R. (2001). Dissociating 624
top-down attentional control from selective perception and action. Neuropsychologia, 625
39(12), 1277-1291.
626
Jerde, T. A., Merriam, E. P., Riggall, A. C., Hedges, J. H., & Curtis, C. E. (2012). Prioritized 627
maps of space in human frontoparietal cortex. J Neurosci, 32(48), 17382-17390.
628
doi:10.1523/jneurosci.3810-12.2012 629
Lauritzen, T. Z., D'Esposito, M., Heeger, D. J., & Silver, M. A. (2009). Top–down flow of 630
visual spatial attention signals from parietal to occipital cortex. Journal of vision, 631
9(13), 18-18.
632
Leigh, R. J., & Zee, D. S. (2015). The neurology of eye movements: Oxford University Press, 633
USA.
634
McDowell, J. E., Dyckman, K. A., Austin, B., & Clementz, B. A. (2008). Neurophysiology 635
and neuroanatomy of reflexive and volitional saccades: evidence from studies of 636
humans. Brain and Cognition, 68(3), 255-270. doi:10.1016/j.bandc.2008.08.016 637
Medendorp, W. P., Goltz, H. C., & Vilis, T. (2006). Directional selectivity of BOLD activity 638
in human posterior parietal cortex for memory-guided double-step saccades. J 639
Neurophysiol, 95(3), 1645-1655. doi:10.1152/jn.00905.2005 640
Medendorp, W. P., Goltz, H. C., Vilis, T., & Crawford, J. D. (2003). Gaze-centered updating 641
of visual space in human parietal cortex. J Neurosci, 23(15), 6209-6214.
642
Merriam, E. P., Genovese, C. R., & Colby, C. L. (2003). Spatial updating in human parietal 643
cortex. Neuron, 39(2), 361-373. doi:https://doi.org/10.1016/S0896-6273(03)00393-3 644
Mesulam, M. M. (1981). A cortical network for directed attention and unilateral neglect. Ann 645
Neurol, 10(4), 309-325. doi:10.1002/ana.410100402 646
Mesulam, M. M. (1999). Spatial attention and neglect: parietal, frontal and cingulate 647
contributions to the mental representation and attentional targeting of salient 648
extrapersonal events. Philos Trans R Soc Lond B Biol Sci, 354(1387), 1325-1346.
649
doi:10.1098/rstb.1999.0482 650
