Article
Reference
Neurocomputational correlates of learned irrelevance in humans
ABERG, Carl Kristoffer, KRAMER, Emily, SCHWARTZ, Sophie
ABERG, Carl Kristoffer, KRAMER, Emily, SCHWARTZ, Sophie. Neurocomputational correlates of learned irrelevance in humans. NeuroImage , 2020, vol. 213, p. 116719
DOI : 10.1016/j.neuroimage.2020.116719 PMID : 32156624
Available at:
http://archive-ouverte.unige.ch/unige:136013
Disclaimer: layout of this document may differ from the published version.
Neurocomputational correlates of learned irrelevance in humans
Kristoffer Carl Aberg
a,*, Emily Elizabeth Kramer
b,c, Sophie Schwartz
d,e,faDepartment of Neurobiology, Weizmann Institute of Science, Rehovot, Israel
bProgram in Neurosciences and Mental Health, Hospital for Sick Children, Toronto, ON, Canada
cInstitute of Medical Sciences, University of Toronto, Toronto, ON, Canada
dDepartment of Neuroscience, Faculty of Medicine, University of Geneva, Geneva, Switzerland
eSwiss Center for Affective Sciences, University of Geneva, Geneva, Switzerland
fGeneva Neuroscience Center, University of Geneva, Geneva, Switzerland
A R T I C L E I N F O Keywords:
Associative learning Decision making Entorhinal cortex Maladaptive behavior Nucleus accumbens
A B S T R A C T
Inappropriate behaviors may result from acquiring maladaptive associations between irrelevant information in the environment and important events, such as reward or punishment. Pre-exposure effects are believed to prevent the expression of irrelevant associations. For example, learned irrelevance delays the expression of as- sociations between conditioned (CS) and unconditioned (US) stimuli following their uncorrelated presentation.
The neuronal substrates of pre-exposure effects in humans are largely unknown because these effects rapidly attenuate when using traditional pre-exposure paradigms. The latter are therefore incompatible with neuro- imaging approaches that require many trial repetitions. Moreover, large methodological differences between animal and human research on pre-exposure effects challenge the presumption of shared neurocognitive sub- strates, and question the prevalent use of pre-exposure effects in animals to model symptoms of human mental disorders. To overcome these limitations, we combined a novel learned irrelevance task with model-based fMRI.
We report the results of a model that describes learned irrelevance as a dynamic process, which evolves across trials and integrates the weighting between two state-action values pertaining to‘CS-no US’associations (acquired during pre-exposure) and‘CS-US’associations (acquired during subsequent conditioning). This relative weighting correlated i) positively with the learned irrelevance effect observed in the behavioral task, ii) positively with activity in the entorhinal cortex, and iii) negatively with activity in the nucleus accumbens (NAcc). Furthermore, the model updates the relative weighting of the two state-action values via two separate prediction error (PE) signals that allow the dynamic accumulation of evidence for the CS to predict the‘US’or a‘no US’outcome. One PE signal, designed to increase the relative weight of‘CS-US’associations following‘US’outcomes, correlated with activity in the NAcc, while another PE signal, designed to increase the relative weight of‘CS-no US’asso- ciations following‘no US’outcomes, correlated with activity in the basolateral amygdala. By extending previous animal observations to humans, the present study provides a novel approach to foster translational research on pre-exposure effects.
1. Introduction
The ability to learn associations between stimuli and important events, such as rewards or punishments, is integral for behavioral adaptation and its neurocognitive correlates are well understood (Maren, 2001;Martin-Soelch et al., 2007;Schultz and Dickinson, 2000). How- ever, in everyday life, associations are formed in the presence of numerous stimuli. Therefore,filtering out irrelevant information in the environment, i.e. information that should not be retained in priority, is equally important for optimal behavioral adaptation.
The identification and filtering out of irrelevant information has mainly been studied via pre-exposure effects, such as latent inhibition (Lubow and Moore, 1959) and learned irrelevance (Mackintosh, 1973).
Latent inhibition occurs when a to-be-conditioned stimulus (CS) is initially presented (i.e. pre-exposed) in the absence of an unconditioned stimulus (US), while learned irrelevance results from a prior, uncorre- lated presentation of the CS and the US. Such pre-exposure procedures cause delays in conditioned responding (CR), as commonly expressed by slower and more inaccurate responding and/or slower learning of the CS-US association in a subsequent conditioning phase (as compared to
* Corresponding author. Department of Neurobiology, Weizmann Institute of Science, Rehovot, 76100, Israel.
E-mail address:kc.aberg@gmail.com(K.C. Aberg).
Contents lists available atScienceDirect
NeuroImage
journal homepage:www.elsevier.com/locate/neuroimage
https://doi.org/10.1016/j.neuroimage.2020.116719
Received 29 May 2019; Received in revised form 24 February 2020; Accepted 6 March 2020 Available online 7 March 2020
1053-8119/©2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
NeuroImage 213 (2020) 116719
novel, not pre-exposed CS). Pre-exposure effects thus reduce inappro- priate behavioral responding, which the CR represents in this case, by making irrelevant associations (i.e. between a pre-exposed CS and the US) less likely to be expressed (Lubow, 1973).
Methodological limitations that prevent the combination of modern neuroimaging methods with traditional paradigms of latent inhibition and learned irrelevance have severely hindered the investigation of the neuronal correlates of pre-exposure effects in humans (Young et al., 2005a). More specifically, in these paradigms human participants show fast learning of CS-US associations following pre-exposure, thus severely attenuating the impact of the pre-exposure phase and reducing the number of experimental trials showing pre-exposure effects. This is incompatible with modern neuroimaging methods that require many trial repetitions, such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG). By contrast, extensive animal research has identified the brain regions critical for expressing pre-exposure effects (for a review, see Lubow and Weiner, 2010). In addition, while thefindings from animal studies have been frequently used to model symptoms of human mental disorders (Lubow, 2005;
Weiner, 2003), the presumption of neurocognitive similarities between pre-exposure effects in humans and animals are heavily debated (for elaborate discussions, seeByrom et al., 2018;Schmidt-Hansen and Le Pelley, 2012).
In an attempt to bridge the gap between animal and human research on pre-exposure effects, here we used a recently developed task that al- lows many trial repetitions without attenuating the effects of learned irrelevance (Young et al., 2005a), which we tested for thefirst time with model-based fMRI. We derived predictions from reinforcement learning theory, in combination with the prominent‘switching model of latent inhibition’, which is based on decades of animal research (Weiner, 2003). In brief, the switching model posits that associations learned during pre-exposure (i.e. ‘CS-no event’ associations) compete for expression with associations learned during subsequent conditioning (i.e.
‘CS-US’associations). The expression of‘CS-US’associations is delayed for pre-exposed CS, as compared to a novel CS, because the‘CS-no event’
association is initially stronger than the‘CS-US’association. Thus, the
‘CS-US’association can only be expressed after sufficient evidence fa- voring the‘CS-US’over the‘CS-no event’association has accumulated. At the neuronal level, the switching model postulates that ‘CS-no event’
associations are encoded by the entorhinal cortex (EC), causing it to inhibit the nucleus accumbens core (NAcc) which is responsible for initiating the expression of the‘CS-US’association. This model is sup- ported by animal studies showing that lesions to the EC disrupt both latent inhibition and learned irrelevance (Allen et al., 2002a;Coutureau et al., 1999;Jeanblanc et al., 2004;Shohamy et al., 2000), while lesions to the NAcc core cause latent inhibition to be abnormally persistent, i.e.
the inhibition of CRs is prolonged in lesioned animals as compared to control animals (Gal et al., 2005b;Schiller et al., 2006;Weiner, 2003;
Weiner et al., 1996). One limitation of the switching model is that it does not account for how exactly‘CS-no event’and‘CS-US’associations are acquired and regulated. Yet, the dynamic interaction between the two types of associations is critical for the emergence of learned irrelevance.
According to reinforcement learning theory, behavioral change is driven by the mismatch between actual and predicted outcomes, the so-called prediction error (PE). For example, behaviors causing better or worse than predicted outcomes are more likely to be repeated or suppressed, respectively (Aberg et al., 2015, 2016; Schultz and Dickinson, 2000;
Sutton and Barto, 1998). The NAcc and the basolateral amygdala (BLA) encode PEs (Averbeck and Costa, 2017;Roesch et al., 2012), and alter- ations of NAcc and BLA functionality may modulate the expression of latent inhibition. Thus, given that the NAcc might be particularly important for reducing the effects induced by prior stimulus pre-exposure (Young et al., 2005b), the NAcc might encode the PE signal needed to acquire‘CS-US’associations. By contrast, the contribution of the BLA to pre-exposure effects is less clear. Because disrupted BLA function was reported to prevent the impact of stimulus pre-exposure (Coutureau
et al., 2001;Schauz and Koch, 2000), the BLA might encode PEs that enable the acquisition of pre-exposure effects. However, other studies implicate the BLA in the conditioning phase, but not the pre-exposure phase (Barak and Weiner, 2010; Schiller and Weiner, 2004; Schiller et al., 2006), therefore suggesting that the BLA contributes to reversing, rather than enabling, pre-exposure effects. While the above mentioned studies focused on latent inhibition in animals, here we set out to test for similar involvements of the BLA and the NAcc in human learned irrele- vance. To this end, we applied a model-based fMRI approach that allows defining separate PE signals that are respectively involved in the acqui- sition and reversal of learned irrelevance.
Based on these observations and our results, here we propose that learned irrelevance can be understood in the framework of the switching model of latent inhibition. Specifically, using a computational approach heavily inspired by the switching model, we found that the expression of learned irrelevance can be explained by the relative weighting of state- action values pertaining to ‘CS-US’ and ‘CS-no US’ associations.
Furthermore, the balance between these state-action values was respec- tively negatively and positively correlated with activity in the NAcc and the EC. Additionally, individuals showing a stronger behavioral learned irrelevance effect showed a greater neuronal learned irrelevance effect in the bilateral EC, but not in the NAcc. This latter result may be related to the notion that the NAcc acts as a general behavioral switch that allows the expression of‘CS-US’ associations only when input from the EC surpasses a certain threshold (Weiner, 1990;Weiner and Feldon, 1997).
Supporting this notion, the NAcc was more engaged by decisions to start expressing‘CS-US’ associations (vs. continuing to express‘CS-no US’ associations), independent on whether the CS was novel or pre-exposed.
Finally, our modeling approach also suggests that the expression of learned irrelevance is up- and down-regulated following‘no US’and‘US’ outcomes via changes in the relative weighting of their respective state-action values. As such, ‘no US’outcomes increased the relative weight of ‘CS-no US’ associations via model-derived PE signals that correlated with activity in the BLA, while the relative weight of‘CS-US’ associations increased following US outcomes via PEs that correlated with activity in the NAcc. The obtained results thus uncover neuro- computational correlates that may contribute to our ability to assign an
‘irrelevance value’to any stimulus that is not associated with significant
behavioral consequences. Our study also decisively connects research on pre-exposure effects in humans and animals.
2. Materials and methods 2.1. Participants
After having provided written consent according to the ethical reg- ulations of the Geneva University Hospital, thirty-four participants joined the experiment. All participants were right-handed, native French speakers, and without any previous history of psychiatric or neurological disorders. The study was performed in accordance with the Declaration of Helsinki. One participant fell asleep in the scanner, thus data from 33 participants were included in the subsequent analyses (15 females;
average ageSEM: 25.2120.867).
2.2. Learned irrelevance task 2.2.1. Motivation for task-selection
Learned irrelevance is caused by the uncorrelated presentation of the CS and the US in a pre-exposure phase, and is measured as a delay in the expression of a CS-US association in a subsequent conditioning phase (Mackintosh, 1973). In humans, learned irrelevance experiments are usually limited to one pre-exposure phase and one conditioning phase.
This is because, in the subsequent conditioning phase, participants quickly figure out that the CS now predicts the US, thus effectively eliminating the effect of learned irrelevance. This limitation has pre- vented the combination of traditional learned irrelevance paradigms (i.e.
used in animal research) with neuroimaging methods, such as EEG and fMRI, which require many trial repetitions (Young et al., 2005a).
To alleviate this limitation, a paradigm was developed in which different letters (i.e. CS’s) were presented sequentially together with a target letter X (i.e. the US;Young et al., 2005a). Participants performed a speeded response whenever the letter X appeared in the sequence. During some parts of the sequence (i.e. pre-exposure phases), learned irrelevance was induced by presenting all letters randomly interleaved with the X (i.e. all CS’s became pre-exposed because no CS predicted the US). In subsequent parts of the sequence (i.e. conditioning phases), one letter reliably predicted the X and therefore allowed the formation of a CS-US association. Learned irrelevance was estimated by comparing how much slower participants responded to the US when preceded by a
pre-exposed, as compared to a novel letter during the conditioning phases. This type of experimental design allows many trial repetitions without attenuating the learned irrelevance effect (Gal et al., 2005a;
Orosz et al., 2007,2008,2011;Schmidt-Hansen et al., 2009;Young et al., 2005a), and can therefore be combined with fMRI. Up to now, only one fMRI study used this type of task and reported very preliminary results from only four participants (Young et al., 2005a). Besides the small sample size, the fMRI data were analyzed using a block design in which many different types of trials (e.g. pre-exposure/conditioning trials) and phases (e.g. decision/learning phases) were collapsed, and no activation in the EC or the NAcc was reported. This outcome is partly due to the low number of participants, but also because performance and fMRI data was averaged across trials and task phases, thus disregarding trial-by-trial
Fig. 1. Stimuli and procedure. A. When an animal was presented,‘Follow’was selected if it was believed that the animal would lead to the treasure (i.e. the next picture contains the crown;Fig. 1B), but‘Leave’otherwise. B. When the treasure was presented, the‘Take!’option had to be selected as quickly as possible. C. In total, participants performed sixteen blocks. Each block consisted of a succession of four sequences. Each sequence containedfive trials each including one treasure (US). The first block always served as a pre-exposure condition in which different animals (CS) preceded the treasure, so as to establish learned irrelevance (i.e.‘Pre-exposed Random’block,Fig. 1D). The four different types of blocks were then presented in a pseudorandom order. D. In‘Pre-exposed Predictable’blocks, one pre-exposed animal predicted the treasure in all four sequences (here, the goat). In‘Pre-exposed Random’blocks, all animals were randomly followed by the treasure, i.e.
none of these animals reliably predicted treasure. In this example of a‘Pre-exposed Random’block, the goat precedes the treasure in thefirst sequence, but does not precede the treasure in the following three sequences. In‘Novel Predictable’blocks, one novel animal predicted treasure in all four sequences (here, the giraffe). In
‘Novel Random’blocks, all animals, including a novel one (here, the penguin) were randomly followed by the treasure. In this example of a‘Novel Random’block, the penguin precedes the treasure in thefirst sequence, but not in the three subsequent sequences. For description purposes, the treasure is on trial 3 in thefirst sequence and trial 4 in the fourth sequence in all blocks, but was pseudorandomly distributed in each sequence during the experiment. Similarly, colored symbols indicate which animal precedes the treasure in each sequence and was not shown during the experiment. Light-blue cross¼Pre-exposed Predictable, Pink cross¼Pre-exposed Random, Blue circle¼Novel Predictable, Red circle¼Novel Random.
variations that may be critical for understanding the learning mecha- nisms that drive changes in behavior (Aberg et al., 2017). Here we developed a treasure hunt game, based on the‘letter task’used in pre- vious studies, but modified to be compatible with a trial-wise computa- tional modeling of behavior and event-related fMRI.
2.2.2. Task description
Participants performed a task framed in the context of a treasure hunt game where pictures of animals (i.e. representing the CS’s) and a treasure (i.e. the US) were presented sequentially, i.e. in each trial one picture of an animal or of the treasure was presented (Fig. 1A and B). Whenever the treasure picture was presented, participants were instructed to select
‘Take!’as quickly as possible in order to keep the treasure (participants
were told that whenever a treasure was discovered, bandits tried to steal it and the treasure could be retained only by responding ‘Take!’ as quickly as possible). By contrast, when an animal picture was presented, participants indicated whether to follow or to leave that animal (Fig. 1A).
Participants were instructed that if they believed the animal would lead them to the treasure (i.e. the next picture is the treasure;Fig. 1B), they should follow it by selecting‘Follow’, while they should ignore animals believed not to lead to treasure by selecting ‘Leave’. Previous studies using similar designs show that learned irrelevance caused increased reaction times to the US and impaired the ability to predict the US (Schmidt-Hansen et al., 2009;Young et al., 2005a).
As an incentive to perform the task well, participants earned points converted into monetary reward at the end of the experiment. Four different action-outcomes were possible: 1) following animals leading to the treasure was worthþ5 points (treasure was discovered), 2) following animals leading to other animals was penalized with 5 points, 3) leaving animals leading to other animals earned 0 points (the treasure was not found but no time was lost), 4) leaving animals leading to the treasure earnedþ1 point (participants were instructed that they could steal part of the treasure from the bandits by quickly responding‘Take’
when the treasure picture was presented). A post-experimental debrief- ing indicated that participants were motivated by trying to find the treasure and to respond accurately, but were not influenced by the points associated with the different action-outcomes. Consistent with such subjective reports, computational models incorporating the actual action-outcome values provided significantly inferiorfits to behavior, as compared to using binary action-outcome representations (see Section 3.2). Thus, similar to previous learned irrelevance designs without monetary rewards, performance in the present study could be attributed solely to the presence and the absence of the US. Note that each partic- ipant was exposed to the whole sequence of stimuli, including all oc- currences of the US, independently of whether he/she decided to follow or leave an animal. This procedure eliminates the need to make explor- atory‘Follow’decisions in order to learn whether an animal predicts the treasure or not, and this was also explicitly instructed to the participants.
Finally, to eliminate neuronal activity related to the preparation and execution of motor responses, participants were forced to make a button press both when the US was predicted and when it was not predicted, unlike previous behavioral studies in which a button press was made only when the US was predicted to occur in the subsequent trial (Schmid- t-Hansen et al., 2009).
Each participant performed sixteen experimental blocks, each block consisting of four sequences offive trials each (Fig. 1C). Specifically, each sequence consisted of four animals and one treasure display, for a total of 20 trials per block. The position of the treasure within each sequence was pseudorandomized to ensure that it was presented once at positions two, three, four, andfive within one block. To assess associative learning and learned irrelevance, four different types of blocks were presented (Fig. 1D). In each‘Pre-exposed Predictable’block, one pre-exposed ani- mal predicted the treasure across all four sequences. In each‘Pre-exposed Random’block, pre-exposed animals randomly preceded the treasure such that an animal preceding the treasure in one sequence did not precede the treasure in any of the other three sequences of the block.
‘Novel Predictable’ blocks were similar to ‘Pre-exposed Predictable’ blocks, except that one novel animal predicted the treasure across all four sequences. Finally,‘Novel Random’blocks were similar to‘Pre-exposed Random’blocks, with the exception that one novel animal replaced one pre-exposed animal. Critically, each of the eight novel animal was pre- sented four times, and each in one block only. Accordingly, each of the
‘Novel Predictable’and‘Novel Random’blocks (eight blocks in total)
contained one single and different novel animal. Participants observed a continuous stream of pictures and were therefore unaware that the sequence of stimuli was divided into separate blocks (i.e. the concept of blocks is only relevant for the data analysis). Accordingly, participants needed to track animals that predicted the treasure, as well as animals that did not, and to re-learn these contingencies throughout the experi- ment (i.e. while one animal might initially not predict the treasure, it might start doing so later in the experiment).
Previous learned irrelevance paradigms used separate categories of letters as stimuli, with vowels used as pre-exposed stimuli and conso- nants as novel stimuli (Orosz et al., 2007; Young et al., 2005a). The present study used pictures of four common animals (i.e. pig, goat, swan, and cow) as pre-exposed stimuli and eight pictures of more exotic ani- mals (i.e. giraffe, zebra, camel, rhino, penguin, elephant, monkey, and ostrich) as novel stimuli. Each picture was displayed for 2.5 s with a jittered interstimulus interval of 1.0 s (range 0.5–1.5 s). If participants failed to respond during the picture display (average proportion of missed trialsSEM: 0.0130.004), the words“No response”appeared on the screen for 1.0 s and participants lost 10 points.
2.2.3. Procedure
To get familiarized with the task, participants first performed a training session outside the MRI scanner in which three blocks were presented. First a‘Pre-exposed Random’block was presented, followed
by a‘Pre-exposed Predictable’block, and then by another‘Pre-exposed
Random’block. Participants proceeded to perform the experiment inside the MRI scanner after having indicated that they understood the task, i.e.
by verbally stating which animal predicted the treasure in the ‘Pre- exposed Predictable’ block. Animal stimuli used during the training session were not used during the MRI session.
Inside the MRI scanner, sixteen blocks were performed. All partici- pants started by performing one‘Pre-exposed Random’block, followed in a pseudorandom order byfive ‘Pre-exposed Predictable’ blocks, five
‘Novel Predictable’blocks, two‘Pre-exposed Random’blocks, and three
‘Novel Random’blocks. The pseudorandomization included two limita-
tions. First, to ensure that‘Predictable’blocks contained a large pro- portion of animals that had been recently pre-exposed, each‘Random’ block was preceded and followed by a‘Predictable’block, and second, no block of the same type was presented in immediate succession. Each block contained four sequences offive trials each, so that 320 trials were performed for a total duration of about 20 min (including two breaks of 30 s each).
2.2.4. Statistical analysis
Our main performance measure of interest was the hit rate, defined as the proportion of‘Follow’responses for animals preceding treasure. As in previous studies that used the‘letter task’(Orosz et al., 2007,2008, 2011;Young et al., 2005a), we also analyzed how quickly participants responded when the US was presented, i.e. the reaction time to select
‘Take!’when the treasure was presented. Additionally, we calculated and analyzed the unbiased sensitivity index d’ from the hit rates. Using repeated measures ANOVAs, performance differences between block types were tested using factors Exposure (Pre-exposed, Novel), Predict- ability (Predictable, Random), and Sequence number in a block (1,2,3,4).
Significant effects were further analyzed using pairwise t-tests, and Bonferroni correction was applied where needed. Correlations were tested using Pearson correlation coefficients.
Because some data was not normally distributed (as indicated by the Lilliefors test;Lilliefors, 1967), we performed complementary analyses
using non-parametric approaches. In brief, instead of repeated measures ANOVAs we used an ANOVA-type statistic (ATS) and estimated p-values using a wild bootstrap procedure (as implemented in the freely available R package MANOVA.RM; Friedrich et al., 2018). Instead of pairwise t-tests, we used Monte-Carlo (MC) permutation tests (Moore and McCabe, 2005). The results of the parametric methods together with the p-values from MC permutation tests are presented in the main text, while the p-values from the ATS approach are presented in Supplementary mate- rials. In terms of p-values, the conclusions drawn from using parametric and non-parametric tests were identical (Blanca et al., 2017;Lumley et al., 2002;Schmider et al., 2010).
2.3. Computational approach
The switching model of latent inhibition posits that associations learned during the pre-exposure phase (i.e. CS-no event associations) competes with associations learned during the subsequent conditioning phase (i.e. CS-US associations;Weiner, 1990,2003;Weiner and Feldon, 1997). Thus, following the pre-exposure phase the‘CS-no event’asso- ciation is initially stronger than the ‘CS-US’ association, ultimately causing a delay in the expression of the‘CS-US’association during the conditioning phase. We propose that learned irrelevance can be un- derstood in the same framework, by considering that learned irrele- vance results from the learning of a‘CS-no US’association during the pre-exposure phase.
To this end, we applied a computational approach that allows the separate investigation of the parameters that determine theexpressionof learned irrelevance (i.e. what causes one behavior to be expressed over another), from thelearningmechanisms that dynamically up- and down- regulate these parameters (i.e. which events lead to behavioral change).
Reinforcement learning theory posits that behavior is governed by two inter-linked phases. In a decision phase, given a particular state (here, a presented animal), the selected action (here, ‘Follow’and ‘Leave’) is determined by weighting the expected value of available actions (here, denoted by the state-action values Qfollow and Qleave). In a learning phase, the expected value of the elicited action is updated based on the mismatch between the action’s predicted outcome and the actual outcome, the so-called prediction error (PE;Sutton and Barto, 1998).
The separation of the decision phase from the learning phase cannot easily be accomplished by research that relies on lesion methods or pharmacological manipulations. For example, even though a brain lesion may alter the effect of pre-exposure on subsequent behavior, it remains unknown whether this effect is caused by the lesion’s impact on mechanisms determining which behavior to express (i.e. a decision phase) and/or mechanisms that process the behavioral outcomes (i.e. a learning phase). A formal description of the considered models is described next.
2.3.1. Q-learning
In Q-learning, state-action values are updated based on the PE scaled by a learning rate parameter. The state-action valueQðtÞs;adenotes the expected outcome of trialt, states, and actiona.
Qðtþ1Þs;a¼QðtÞs;aþαδðtÞ
δðtÞ ¼rðtÞ QðtÞs;a
For each trialt,δðtÞis the PE andαdenotes the learning rate. In the current experiment, statesrefers to a specific animal,ais the action (i.e.
‘Leave’or‘Follow’), and the outcomer(t)is determined by the picture
presented in the subsequent trial. The probability of performing a specific action over another (for example‘Follow’vs.‘Leave’) in a given trial can be estimated by a soft-max choice probability function (Sutton and Barto, 1998).
pðtÞs;Follow¼eQðtÞs;Followβ
ðeQðtÞs;FollowβþeQðtÞs;LeaveβÞ
The trade-off between exploration and exploitation is estimated by theβparameter.
2.3.2. Q-learning version of the switching model
Inspired by the switching model (Weiner, 2003), we created a Q-learning model in which two associations compete for behavioral expression in each trial. Specifically, we considered that the selection of the‘Follow’action represents a relatively stronger‘CS-US’(vs.‘CS-no US’) association, while the selection of the ‘Leave’ action indexes a weaker‘CS-US’(vs.‘CS-no US’) association. Accordingly, whenever the presentation of an animalsis followed by the treasure outcome, the probability of selecting‘Follow’over‘Leave’should increase for that animal. In other words, the relative strength of that‘CS-US’(vs.‘CS-no US’) association rises. Given a treasure outcome, this can be accom- plished by increasingQðtÞs;Followif the selected action was‘Follow’or by decreasingQðtÞs;Leavefor a response to‘Leave’.
Qðtþ1Þs;a¼QðtÞs;aþαþδþðtÞ δþðtÞ ¼ f1QðtÞs;a; if a¼Follow
0QðtÞs;a; if a¼Leave
δþðtÞdetermines the change in associative strength andαþis the corre- sponding learning rate.
Conversely, when an animal is followed by an animal picture, the change in strength of the ‘CS-US’ (vs.‘CS-no US’) association should favor the‘CS-no US’association. Thus, following an animal outcome, this is accomplished by either decreasing QðtÞs;Follow or by increasing QðtÞs;Leave.
Qðtþ1Þs;a¼QðtÞs;aþαδðtÞ δðtÞ ¼ f0QðtÞs;a; if a¼Follow
1QðtÞs;a; if a¼Leave
δðtÞand α is the corresponding prediction error and learning rate, respectively.
Initially, we created and confronted four different models. Thefirst model, outlined above, uses two separate learning rates for the scaling of δþðtÞand δðtÞ. A second model uses separate learning rates for pre- exposed and novel animals in an attempt to accommodate evidence of slower learning speeds following learned irrelevance (Schmidt-Hansen et al., 2009). A third model assumes only one learning rate across PE types. A fourth model is identical to thefirst model, but uses the actual points awarded for each action according to the action-outcome bonus system. Afifth model, designed following suggestions of one anonymous reviewer, presumes one state-action value only for each animal, i.e.QðtÞs. The value ofQðtÞsincreases and decreases following treasure and animal outcomes, respectively, and the selected action is determined by a threshold parameterθ, whereQðtÞs>θis more likely to elicit a‘Follow’
action while‘Leave’actions are more likely whenQðtÞs<θ. As in the previous models, the tendency to select a‘Follow’action is increased and decreased following treasure and animal outcomes respectively:
δþðtÞ ¼1QðtÞsif outcome¼Treasure
δðtÞ ¼0QðtÞsif outcome¼Animal
The probability of selecting a‘Follow’action depends on the value of QðtÞs, in combination with the threshold parameterθ:
pFollow¼ f1ε; if QðtÞs>θ ε; otherwise
In other words, whenQðtÞsis larger or smaller than the thresholdθ,
‘Follow’ or ‘Leave’actions are elicited with a probability 1-ϵ. Theϵ
parameter determines the average level of decision noise/exploration.
2.3.3. Modelfitting
The free parameters, i.e. learning rates, exploration/exploitation parameterβ, thresholdθ, and decision noise/explorationϵ, werefitted individually to each participant’s behavior by minimizing the negative log-likelihood estimate:
LLE¼–lnðYn
1
pðtÞs;aÞ
Givenntrials,pðtÞs;ais the soft-max choice probability of selecting actionafor animalsin trialt. To avoid getting stuck in local minima, each fit was repeated 100 times with different random starting points for each free parameter. All modelfits were compared using the Akaike Infor- mation Criterion (AIC) corrected for small sample sizes, i.e. (AICc;Wa- genmakers and Farrell, 2004), which controls for different numbers of fitted parameters:
AICc¼ 2*LLEþ2*kþ2*k2þ2*k nk1
Additionally, a random choice“null”model was used to compute a standardized metric of model fit, a pseudo-R2 statistic defined as the improvement from a null model to thefitted model, i.e. pseudo-R2¼1 - LLEfitted/LLErandom, whereLLErandomis the log-likelihood estimate under the random choice model andLLEfittedis log-likelihood estimate under the fit model (Gershman et al., 2009;McFadden, 1974).
2.3.4. Model simulation of behavior
To ensure that the most parsimonious model can account for the behavioral phenomena of interest (Palminteri et al., 2017), we used individuallyfitted model-parameters to derive a trial-by-trial probability of selecting the‘Follow’action in trials where one animal predicted the treasure (i.e. pCorrect). Hence, a model capable of predicting a learned irrelevance effect should predict a lower pCorrect for pre-exposed, as compared to novel animals. Additionally, the observed learned irrele- vance effect (as defined by the difference in hit rates between ‘Novel Predictable and ‘Pre-exposed Predictable’conditions) should correlate with the learned irrelevance effect predicted by the model (i.e. the dif- ference in pCorrect between‘Novel Predictable and‘Pre-exposed Pre- dictable’conditions). Moreover, to provide a visual indication of the similarity between actual and simulated behavior, we included the average estimates of pCorrect along actual performance in all thefigures featuring hit rates.
2.4. MRI data 2.4.1. Image acquisition
MRI images were acquired using a 3T whole body MRI scanner (Trio TIM, Siemens, Germany) with a 12-channel head coil. Standard struc- tural images were acquired with a T1 weighted 3D sequence (MPRAGE, Repetition time (TR)/Inversion delay time (TI)/Echo time (TE)¼1900/
900/2.27 ms,flip angle¼9, voxel dimensions¼1 mm isotropic, matrix size¼256256192). Functional images were acquired with a sus- ceptibility weighted EPI sequence (TR/TE¼2100/30 ms,flip angle¼ 80, voxel dimensions¼3.2 mm isotropic, matrix size¼646436).
2.4.2. Data analysis
Functional MRI data were preprocessed and then analyzed using the general linear model (GLM) for event-related designs in SPM8 (Welcome Department of Imaging Neuroscience, London, UK;http://www.fil.ion.
ucl.ac.uk/spm). During preprocessing, all functional volumes were real- igned to the mean image, co-registered to the structural T1 image, cor- rected for slice timing, normalized to the MNI EPI-template, and smoothed using an 8 mm FWHM Gaussian kernel. Statistical analyses were performed on a voxel wise basis across the whole-brain. At thefirst- level analysis, individual events were modeled by a standard synthetic hemodynamic response function (HRF) and six rigid-body realignment
parameters were included as nuisance covariates when estimating sta- tistical maps. Three types of analyses were then conducted to investigate the neuronal correlates of learned irrelevance.
2.4.2.1. Action selections. First, we wanted to confirm a prediction derived from the switching model of latent inhibition, namely that the NAcc is more engaged by decisions to start expressing‘CS-US’associa- tions following CS-US pairings vs. failures to do so (i.e. continuing to express‘CS-no US’associations). To this end, we considered trials in which the animal in the preceding trial had elicited a‘Leave’response that was subsequently followed by the treasure outcome (i.e. a CS-US pairing). These trials were then categorized based on whether partici- pants predicted that the animal would lead to treasure (i.e. a‘Correct switch-to-follow’decision), or that the animal was not relevant for pre- dicting the treasure (i.e. an‘Incorrect repeat-leave’decision).
An event-related design was created with three different event-types (Correct switch-to-follow for Novel animals, Correct switch-to-follow for Pre-exposed animals, and Incorrect repeat-leave for Pre-exposed ani- mals) time-locked to the onset of stimulus presentation (i.e. the picture of an animal). Treasure pictures were included in the model as a regressor of no interest, as were trials in which no response was made during the picture display, and trials not falling in any of the above categories.
Please observe that very few trials were classified as‘Incorrect repeat- leave’for novel animals [median number of such trials¼3; number of participants displaying one or less of such trials¼10]. Thus, because of the unreliability of estimating the beta weight of this category, it was omitted in the subsequent analyses. The corresponding numbers (median number of such trials/number of participants displaying one or less of such trials) for‘Correct switch-to-follow’trials for novel and pre-exposed animals were 8/0 and 12/0 respectively, while for trials classified as
‘Incorrect repeat-leave’for pre-exposed animals the numbers were 13/0.
2.4.2.2. Model-based. The output of the computational model was used to provide a more fine-grained and trial-by-trial representation of neuronal activation related to specific computational parameters involved in learned irrelevance. To this end, an event-related design was created which included two event-types, respectively time-locked to the onset of animal and treasure pictures. To assess the decision phase, the state-action values Qfollow and Qleave were added as linear parametric modulators to the onset times for animals, while the learning phase was assessed by adding the model-derived PEs (i.e. δþand δ) as linear parametric modulators to the onset times for treasures and animals, respectively. Trials in which no response was made were included as a regressor of no interest. To detect variance uniquely associated with each of the different parametric modulators, the default serial orthogonali- zation procedure implemented in SPM8 was disabled.
2.4.2.3. Neuronal learned irrelevance.At a behavioral level, learned irrelevance is defined by the difference in hit rates between novel and pre-exposed animals that predict the treasure in‘Predictable’blocks. To test for the neuronal correlates of this effect, we created an event-related design withfive event-types time-locked to the onset of animal pictures, and one event-type time-locked to the onset of treasure pictures. Four of the animal event-types were animals followed by the treasure picture, i.e.
in‘Predictable’novel,‘Predictable’pre-exposed,‘Random’novel, and
‘Random’ pre-exposed blocks, while the fifth event-type included all
animals that were not followed by the treasure. All presentations of an- imal pictures were included in the analysis (including thefirst presen- tation of a novel animal), except for trials in which no response was made which were included as a regressor of no interest. The neuronal learned irrelevance effect was calculated as the difference in beta weights be- tween the‘Predictable’novel and‘Predictable’pre-exposed contrasts.
2.4.3. Regions of interest (ROIs)
A priori ROIs were used for small volume corrections. Left and right
ROIs for the entorhinal cortex (EC) and the basolateral amygdala (BLA) were obtained from the Anatomy toolbox (Amunts et al., 2005;Eickhoff et al., 2005). The ROI for the right NAcc was obtained from a recent study that delineated the shell and core subregions of the NAcc (Baliki et al., 2013). Note that the left NAcc ROI corresponded to the symmetrical projection of the right NAcc ROI onto the left hemisphere.
2.4.4. Statistical analyses
For both the action selections and model-based analyses, second-level group analyses were performed usingt-tests implemented in SPM. For group-level analyses, small volume corrections (SVC) on peak-voxel ac- tivity, using a threshold of p<0.05 Family-Wise Error rate (FWE) for multiple comparisons, were applied using the a priori ROIs defined above. Moreover, for display purposes and follow-up analyses, we extracted beta parameter estimates using 3 mm radius spheres centered on the peak-coordinates for each significant cluster of activation. For correlations between learned irrelevance effects in behavior and brain activity, we followed recent recommendations by correlating behavioral measures with the average activity within ROIs obtained from group- level statistical maps rather than single-voxel activity (Vul et al., 2009). Bonferroni-correction was applied where appropriate.
3. Results 3.1. Behavior
We developed a treasure hunt game to estimate learned irrelevance in humans. In this game, participants learned that some animals (i.e. CS) could help themfind a treasure (i.e. US). Pictures of animals were pre- sented sequentially one-by-one and participants selected a‘Follow’op- tion if they predicted that the animal would lead to the treasure (i.e. that the next picture presented is the treasure), while they selected a‘Leave’
option otherwise (Fig. 1A). Whenever the treasure was presented, par- ticipants were instructed to select‘Take!’as quickly as possible (Fig. 1B).
Thus, to optimize performance participants needed to track which ani- mals led to the treasure, and this was possible in blocks where one spe- cific animal always preceded the treasure (i.e.‘Predictable’blocks), but not in‘Random’blocks (where animals randomly preceded the treasure;
Fig. 1D). Learned irrelevance was assessed by comparing performance in blocks where novel animals predicted the treasure (i.e.‘Novel Predict- able’blocks), as compared to‘Pre-exposed Predictable’blocks in which animals that had previously been presented in an uncorrelated fashion with the treasure predicted the treasure.
3.1.1. Hit rates
Hit rates (i.e. the proportion of‘Follow’responses) for animals pre- ceding treasure in the different types of blocks are displayed inFig. 2A.
Differences between block types were initially investigated by an ANOVA with three factors: Exposure (Pre-exposed, Novel), Predictability (Pre- dictable, Random), and Sequence number in a block (1, 2, 3, 4).
The main effects of Predictability [F (1,32)¼70.46, p<0.001,η2p¼ 0.69, ANOVA] and Sequence [F (3,96)¼4.64, p¼0.005,η2p ¼0.13, ANOVA] were significant. These two results can be explained by learning occurring progressively across successive sequences in ‘Predictable’ blocks, but not in the ‘Random’blocks, as indicated by a significant PredictabilitySequence interaction [F (3,96)¼16.48, p<0.001,η2p¼ 0.34, ANOVA] and separate ANOVAs with factor Sequence for‘Predict- able’and‘Random’blocks [Predictable: F (3, 96)¼19.22, p<0.001,η2p
¼0.38; Random: F (3, 96)¼2.15, p¼0.100,η2p¼0.06, ANOVA]. The main effect of Exposure was also significant [F (1,32)¼14.61, p<0.001, η2p ¼0.31, ANOVA], with lower hit rates for‘Pre-exposed’vs.‘Novel’ blocks. Specifically, hit rates were lower for‘Pre-exposed Predictable’as compared to the ‘Novel Predictable’blocks, as would be predicted if learned irrelevance had occurred. This notion was confirmed by a
significant PredictabilityExposure interaction [F (1,32)¼33.50, p<
0.001, η2p ¼ 0.51, ANOVA] (see also additional analyses in the next paragraph). Noteworthy, the non-significant Exposure x Predictability Sequence interaction [F (3,96)¼1.00, p¼0.399,η2p ¼0.03, ANOVA]
indicates that performance differences between the‘Novel Predictable’ and‘Pre-exposed Predictable’blocks were not due to a lower learning speed for pre-exposed, as compared to novel animals. This bias continued to be expressed until enough evidence had been accumulated to allow a switch to the‘Follow’response. By contrast, because novel animals did not suffer from learned irrelevance,‘Follow’responses were expressed earlier in a block. No other interactions were significant [all p-values>
0.165, seeSupplementary Table 1for a full ANOVA table and p-values estimated using a non-parametric approach].
To more clearly illustrate the effect of learned irrelevance, hit rates for the different block types were collapsed across factor Sequence (see Fig. 2B). The collapsed hit rates did not differ between the two‘Random’
blocks [mean HRSEM:‘Pre-exposed Random’¼0.2830.040;‘Novel Random’¼0.2700.036;t(32)¼0.450, pairedt-test: p¼0.656, MC permutation test: p¼0.655], and were therefore averaged. Hit rates for
‘Pre-exposed Predictable’[mean HRSEM: 0.5310.038] and‘Novel
Predictable’[mean HRSEM: 0.7830.038] blocks differed from those of the averaged‘Random’blocks [mean HRSEM: 0.2770.035; both p-values<0.001 for both paired t-tests and MC permutation tests], thus confirming that learning occurred in both‘Predictable’blocks. Critically, hit rates in the‘Pre-exposed Predictable’block were significantly smaller than for‘Novel Predictable’[t(32)¼ 5.348, pairedt-test: p<0.001, MC permutation test: p<0.001], consistent with a learned irrelevance effect. Complementary analyses on response times and the unbiased sensitivity index d’mirrored the hit rate results (see Supplementary in- formation 1.1, 1.2;Supplementary Fig. 1andSupplementary Tables 2 and 3). In sum, these behavioral results confirm the presence of learned irrelevance.
One potential confound to the claim that learned irrelevance had occurred is that participants may have felt more inclined to make an exploratory‘Follow’response whenever a novel animal was encountered, despite our best efforts to prevent the need for exploratory decisions (see Section 2.2.2). However, participants did not exhibit a significant response bias for novel animals, nor did such a bias develop throughout the course of the experiment (see Supplementary information 1.3;Sup- plementary Fig. 2A).
3.2. Computational approach
Inspired by the switching model of latent inhibition, we considered that selections of the‘Leave’action might reflect a stronger‘CS-no US’
(vs.‘CS-US’) association, and vice versa for the‘Follow’action. More- over, the tendency to favor‘Follow’over‘Leave’actions should increase whenever an animal is followed by the treasure (i.e. following a CS-US pairing), and decrease when the animal is followed by an animal outcome (i.e. a‘CS-no US’pairing). For a full formulation of the corre- sponding computational approach, see Section 3.2. Fitted model pa- rameters are reported inTable 1. In summary, the most parsimonious model assumes different learning rates for‘US’and‘no US’outcomes, and provided a significantly betterfit to behavior [mean AICcSEM¼ 235.870 14.055], as compared to the model that assumes separate learning rates for novel and pre-exposed animals [mean AICcSEM¼ 239.76914.092,t(32)¼2.352, pairedt-test: p¼0.025, MC permu- tation test: p¼0.001], and the model with one learning rate [mean AICc SEM¼238.75914.188;t(32)¼2.526, pairedt-test: p¼0.010, MC permutation test: p¼0.002]. Thefit of the most parsimonious model to behavior is displayed in Fig. 2A (dotted lines) and Fig. 2B (small symbols).
The most parsimonious model posits that behavior is determined via the combination of two phases. In a decision phase, the action with the largest corresponding state-action value (Qleave, Qfollow) is selected. In
Fig. 2. Behavior and model (MeanSEM). A. Hit rates (large symbols and solid lines) and model-derived probabilities (pCorrect; small symbols and dotted lines) for selecting the‘Follow’option for animals preceding the treasure as a function of Sequence and block type. Significant learning occurred in the two‘Predictable’blocks as indicated by gradually increased hit rates as a function of an increased sequence number. B. Hit rates (large symbols) and model-derived probabilities (pCorrect;
small symbols) for selecting the‘Follow’option collapsed across sequences. Pre-exposure caused lower hit rates for‘Pre-exposed Predictable’blocks as compared to the
‘Novel Predictable’blocks, i.e. the effect of learned irrelevance. Please observe that hit rates for Novel Random blocks in A and B refer to‘Follow’responses for any animal that precede the treasure, and not exclusively to‘Follow’responses for novel animals. Indeed, this condition is designed so that novel animals randomly predict the treasure. C. Average state-action values Qleave (dashed lines) and Qfollow (solid lines) as a function of Sequence for treasure-predicting animals in‘Novel Predictable’(blue lines) and‘Pre-exposed Predictable’blocks (light-blue lines). Pre-exposure caused an initial increase in Qleave for animals in the‘Pre-exposed Predictable’condition, while the levels of Qfollow were largely similar between the two conditions across sequences. D. Average levels of Qleave and Qfollow collapsed across sequences. E. Pre-exposure caused an initial negative difference between Qfollow and Qleave for animals in the‘Pre-exposed Predictable’condition. F.
Average difference between Qleave and Qfollow across sequences. H. The observed learned irrelevance effect (Hit rate in‘Novel Predictable’versus‘Pre-exposed Predictable’blocks) correlated with the learned irrelevance effect predicted by the model (pCorrect in‘Novel Predictable’versus‘Pre-exposed Predictable’blocks). I.
The observed learned irrelevance effect (Hit rate in‘Novel Predictable’versus‘Pre-exposed Predictable’blocks) correlated with the average difference in state-action values (Qfollow-Qleave) between‘Novel Predictable’and‘Pre-exposed Predictable’blocks. **p<0.001, ns.¼not significant (p>0.05).
a subsequent learning phase, the state-action value of the selected action is updated based on the mismatch between the actual and predicted outcome, i.e. the prediction error (PE). Accordingly, an animal frequently paired with the treasure is more likely to yield a subsequent‘Follow’
response because treasure outcomes increase the expected value of Qfollow (relative Qleave). By contrast, animals exposed to an uncorre- lated presentation with the treasure (as occurs during the pre-exposure phase) are more likely to elicit a ‘Leave’ response because Qleave is gradually becoming larger than Qfollow. Importantly, the state-action values are updated following treasure and animal outcomes via two separate PE signals. δþ increases Qfollow (relative Qleave) following treasure outcomes, while δ increases Qleave (relative Qfollow) following animal outcomes.
To illustrate how this model accounts for learned irrelevance, the state-action values of Qleave and Qfollow for animals that predict the treasure within‘Novel Predictable’and‘Pre-exposed Predictable’blocks are shown inFig. 2C–D. Critically, at the start of‘Pre-exposed Predict- able’blocks (i.e. Sequence 1), the treasure-predicting animal is associ- ated with high levels of Qleave, as compared to Qfollow, and as compared to the treasure-predicting animal in‘Novel Predictable’blocks (Fig. 2C). As mentioned, this initial bias is caused by the uncorrelated presentation of the animal and the treasure in preceding blocks. Thus, because decisions are determined by the differential weighting of Qleave and Qfollow (which is initially negative for pre-exposed animals;
Fig. 2D), the selection of the‘Follow’action is initially suppressed, as compared to novel animals (Fig. 2D). This is how the model accounts for the learned irrelevance effect.
To validate the model’s ability to account for learned irrelevance, we correlated the observed learned irrelevance effect (i.e. the difference in hit rates between ‘Novel Predictable’ and ‘Pre-exposed Predictable’
conditions) and the model-derived learned irrelevance effect (i.e. the difference in predicted hit rates, pCorrect, between‘Novel Predictable’
and ‘Pre-exposed Predictable’ conditions). The significant correlation
indicates that the model captures relevant aspects of learned irrelevance [Fig. 2H; r (32)¼0.549, p<0.001, Pearson correlation]. Furthermore, the significant correlation between the observed learned irrelevance ef- fect and the average difference in state-action values (i.e. Qfollow- Qleave) for treasure-predicting animals in‘Novel Predictable’vs.‘Pre- exposed Predictable’blocks suggests that the learned irrelevance effect can be explained by the relative weighting of the state-action values Qleave and Qfollow [Fig. 2I; r (32)¼0.430, p¼0.012, Pearson corre- lation]. Together, these results support that the most parsimonious model indeed provides a plausible mechanistic explanation for human learned irrelevance.
Of note, the post-experimental debriefing showed that participants did not consider the actual points awarded for the different action-
outcomes. To verify this notion, wefitted the most parsimonious model to behavioral data using actual action-outcome points. In accordance with the debriefing results, the model fitted with the actual action- outcome points provided a significantly worse fit to behavior [mean AICcSEM¼271.6429.595], as compared to thefit using binary values [t(32)¼4.713, pairedt-test: p<0.001, MC permutation test: p<
0.001]. Next, we tested neuronal predictions derived from the switching model of latent inhibition.
3.3. fMRI
3.3.1. Correct switch-to-follow versus incorrect repeat-leave decisions Pre-exposure effects cause a delayed expression of CS-US associations during subsequent CS-US pairings. The switching model of latent inhi- bition attributes this delay to a suppressed switching mechanism in the NAcc (Weiner, 2003). Here, we tested whether failures to start expressing
‘CS-US’associations could be related to a reduced engagement of the NAcc in human learned irrelevance. Accordingly, we compared the BOLD signal when participants made a correct switch to start expressing the
‘CS-US’ association (as indicated by a ‘Follow’ action; Correct switch-to-follow) to when participants incorrectly continued to express
the‘CS-no US’association (as indicated by a‘Leave’action; Incorrect
repeat-leave). We predicted a stronger engagement of the NAcc in Cor- rect switch-to-follow (vs. Incorrect repeat-leave) trials, and a stronger engagement of the EC in Incorrect repeat-leave (vs. Correct switch-to-follow) trials. The latter prediction is derived from the notion that the EC suppresses the NAcc switching mechanism via its encoding of
‘CS-no event’associations (Weiner, 2003).
In accordance with our predictions, the NAcc showed stronger ac- tivity in Correct switch-to-follow trials for both novel and pre-exposed animals, as compared to Incorrect repeat-leave actions for pre-exposed animals [novel animals:Fig. 3A,C; MNI Right NAcc: x¼9 y¼5 z¼ 8, t (32) ¼ 3.846, p ¼ 0.007 (pFWE, SVC); pre-exposed animals:
Fig. 3B,C; MNI Right NAcc: x¼9 y¼5 z¼ 8,t(32)¼3.664, p¼0.012 (pFWE, SVC)]. Please note that because of a very limited number of such trials, the‘Incorrect repeat-leave action’contrast for novel animals was omitted in this analysis.
By contrast, there was no difference in NAcc activity between novel and pre-exposed animals. This latter result suggests that the mechanism involved in starting to express‘CS-US’associations is shared between novel and pre-exposed CS’s. Indeed, the switching model of latent inhi- bition posits that the activation of the NAcc switching mechanism de- pends on the balance between‘CS-US’and‘CS-no event’associations, and makes no distinction between novel and pre-exposed stimuli beyond this notion (Weiner, 1990;Weiner and Feldon, 1997). Of note, supple- mentary analyses ensured that these results were not confounded by the Table 1
Modelfits. Mean (SEM).
Model Negative
LLE
AICc PseudoR2 α α- αþ αNOV αPEX β θ ϵ
Random choice 175.18 (0.73)
350.37 (1.45) – – – – – – – – –
α-/αþ 114.88
(7.03)
235.87 (14.06)
0.34 (0.04)
– 0.60
(0.06)
0.56 (0.07)
– – 0.61
(0.14)
– –
αNOV/αPEX 116.84 (7.05)
239.77 (14.09)
0.33 (0.04)
– – – 0.50
(0.07)
0.63 (0.07)
0.48 (0.10)
– –
α 117.36
(7.09)
238.76 (14.19)
0.33 (0.04)
0.50 (0.07)
– – – – 0.56
(0.14)
– –
α-/αþ(real points)
132.78 (4.98)
271.64 (9.60) 0.24 (0.04)
– 0.44
(0.07)
0.53 (0.07)
– – 3.67
(0.49)
– –
α-/αþ (threshold)
127.59 (6.51)
263.17 (13.02)
0.27 (0.04)
– 0.62
(0.06)
0.53 (0.05)
– – – 0.59
(0.05)
0.30 (0.03) LLE is the log-likelihood estimate. AICc is the Akaike Information Criterion corrected for small sample sizes.αdenotes the learning rate in the model with one learning rate only,α- andαþare the learning rates for animal and treasure outcomes, respectively, whileαNOVandαPEXare the learning rates for novel and pre-exposed animals, respectively.βdetermines the trade-off between exploration and exploitation.θis a threshold parameter that determines whether to elicit a Leave or a Follow action.ϵ reflects decision noise/exploration.
processing of the treasure outcome in subsequent trials (see Supple- mentary information, Section3.1).
BOLD signal in the EC did not differ significantly for the same con- trasts as above (no voxels within the bilateral EC ROI showed significant activation at an uncorrected threshold of p¼0.001). One reason for the lack of significant results in the EC could be that the EC is not involved in the decision to switch actions per se, or it could be that collapsing trials within a category ignored subtle and important trial-by-trial variations in BOLD signal. For example, it has been suggested that the EC encodes the strength of‘CS-no event’associations (Weiner and Feldon, 1997). This hypothesis is formally tested in the next section using model-based fMRI
analysis, which allows trial-by-trial investigation of BOLD signal related to subtle changes in specific computational parameters.
3.3.2. Model-based fMRI
To specify the neuronal mechanisms underlying the expression and learning aspects involved in learned irrelevance, we created a compu- tational model inspired by the switching model of latent inhibition. In brief, this model assumes that, during a decision phase, actions to‘Leave’
and‘Follow’respectively reflect relatively stronger‘CS-no US’and‘CS-
US’associations, as represented via the relative weighting of the state- action values Qleave and Qfollow. Importantly, while neuronal Fig. 3. A-C. BOLD activity of the decision phase following an animal-treasure pairing (i.e. CS-US pairing; MeanSEM). A. Contrasting BOLD activity for Correct switch-to-follow actions for pre-exposed animals versus Incorrect repeat-leave actions for pre-exposed animals show increased BOLD signal in the right NAcc (pFWE, SVC<0.05). B. Contrasting BOLD activity for Correct switch-to-follow actions for novel animals versus Incorrect repeat-leave actions for pre-exposed animals show increased BOLD signal in the right NAcc (pFWE, SVC<0.05). C. Beta weights extracted from the R NAcc for the three different conditions involved in A,B are shown here for visualization purposes. D-G. BOLD activity correlating with state-action values of ‘Leave’ (Qleave) and ‘Follow’ (Qfollow) actions derived from the computational model (MeanSEM). Contrasting Qfollow and Qleave showed larger activity in the right NAcc (D,E; pFWE, SVC<0.05), while activity in the bilateral EC was larger when contrasting Qleave vs Qfollow (F-G; pFWE, SVC<0.05). E. As illustrated by the beta weights extracted from the R NAcc, BOLD signal correlated negatively with Qleave but remained unaffected by changes in Qfollow. G. As illustrated by the beta weights extracted from the EC, BOLD signal correlated positively and negatively with Qleave and Qfollow, respectively. H. The behavioral learned irrelevance effect did not correlate significantly with the neuronal learned irrelevance effect in the right NAcc cluster. I. The behavioral learned irrelevance effect correlated significantly with the neuronal learned irrelevance effect in the bilateral EC clusters. The activations for the right NAcc and bilateral EC clusters were calculated as the average beta weights of all voxels activated at a threshold of p¼0.001 in the contrasts Qfollow>Qleave (see D, same Figure) and Qleave>Qfollow (see F, same Figure). Only voxels that were part of the NAcc and EC clusters were included in the analyses. For display purposes, brain activations in A, B, D, and F, are displayed at an uncorrected threshold of p¼0.001, and the beta parameters in C, E, and G, were extracted from 3 mm spheres centered on respective peak voxels.
responses associated with a specific action (for example those analyzed in section3.3.1above) could arise from many different combinations of Qleave and Qfollow, computational modeling allows the investigation of their separate neuronal correlates. Further, the model posits that, during a learning phase, the relative difference between Qleave and Qfollow, is regulated via separate PE signals that occur after animal and treasure outcomes. Based on previous animal research, we predicted that Qleave and Qfollow should be reflected in the activity of the EC and the NAcc.
Moreover, we predicted that the PE signals should correlate with activity in the NAcc and the BLA.
3.3.2.1. Decision phase: the neuronal correlates of Qleave and Qfol- low. During the decision stage, and in accordance with our predictions, activity in the right NAcc increased as Qfollow (vs. Qleave) increased [Fig. 3D and E; MNI Right NAcc: x¼9 y¼5 z¼ 11, T (32)¼3.565, p<
0.014 (pFWE, SVC)], while the EC showed a stronger activation by the reverse contrast, i.e. Qleave (vs. Qfollow) [Fig. 3F–H; MNI Right EC: x¼ 18 y¼ 10 z¼ 29, T (32)¼3.524, p<0.048 (pFWE, SVC); MNI Left EC: x¼ 18 y¼ 7 z¼ 29, T (32)¼4.147, p¼0.009 (pFWE, SVC)].
Notably, while significant activation was detected only in the right NAcc, strong activation was also observed in the left hemisphere just outside the pre-defined NAcc ROI (seeFig. 3D). The differential activation of the right NAcc was caused by a negative correlation with Qleave, while there was no significant modulation by Qfollow (Fig. 3E). By contrast, activity in the EC correlated positively and negatively with Qleave and Qfollow, respectively (Fig. 3G and H).
3.3.2.2. Decision phase: the behavioral learned irrelevance effect predicts a neuronal learned irrelevance effect in the EC, but not in the NAcc. According to the computational model, the difference between Qfollow and Qleave should be more negative for pre-exposed, as compared to novel animals (Fig. 2F). The magnitude of Qfollow-Qleave correlated with the
magnitude of the behavioral learned irrelevance effect (i.e. the hit rate between novel and pre-exposed animals that predicted the treasure in Predictable blocks;Fig. 2I). Moreover, the magnitude of Qfollow-Qleave was tracked by the EC and the NAcc (Fig. 3D, F). Given these results, a straightforward prediction is that individuals displaying a stronger behavioral learned irrelevance effect should show a stronger neuronal learned irrelevance effect in the EC and the NAcc.
We tested this prediction by defining a neuronal learned irrelevance effect as the difference in BOLD signal between pre-exposed and novel animals that predicted the treasure in Predictable blocks. In accordance with recommendations on how to correlate inter-individual performance measures with fMRI data (Vul et al., 2009), correlations between the behavioral and neuronal learned irrelevance effects were performed after first averaging the activation of all voxels within pre-defined ROIs.
Because our predictions were derived from the computational model, we defined two EC clusters via the contrast between Qleave > Qfollow (thresholded at p¼0.001;Fig. 3D) and one right NAcc cluster via the contrast between Qfollow>Qleave (thresholded at p¼0.001;Fig. 3F).
As predicted, individuals showing a stronger learned irrelevance effect showed stronger activation of the bilateral EC for pre-exposed (vs. novel) animals [Fig. 3I; left EC cluster: r¼0.411, p¼0.009; right EC cluster: r¼ 0.404, p¼0.010, one-tailed tests]. By contrast, activation in the right NAcc did not correlate with the learned irrelevance effect [Fig. 3H; r¼ 0.215, p¼0.770, one-tailed test]. In other words, the behavioral learned irrelevance effect correlated with the neuronal learned irrelevance effect in the EC, but not in the NAcc.
3.3.2.3. Learning phase: neuronal correlates of prediction errors following treasure (δþ) and animal outcomes (δ). During the learning phase, cor- relations between brain activity (in the NAcc and the BLA) and model- derived PEs (δþ and δ; i.e. the mismatch between the state-action value of the selected action and a treasure/animal outcome,
Fig. 4. BOLD activity of the learning phase (MeanSEM). Prediction errors of treasure (δþ) and animal outcomes (δ) designed to respectively increase and decrease the subsequent frequency of‘Follow’(vs. Leave) actions (MeanSEM). A. BOLD signal in the bilateral NAcc correlated positively withδþ(pFWE, SVC<0.05). For display purposes, beta parameter estimates from the left (B) and the right (C) NAcc are shown for bothδþandδ. D. Beta parameter estimates from the right BLA correlated positively withδ(pFWE, SVC<0.05). E. For display purposes, beta parameter estimates from the right BLA are shown for bothδþandδ. Brain activations in A and D are displayed at an uncorrected threshold of p¼0.001, and the beta parameters in B, C, and E, were extracted from 3 mm spheres centered on respective peak voxels.