What Is the Relationship between Dopamine and Effort?

HAL Id: hal-02394833

https://hal.archives-ouvertes.fr/hal-02394833

Submitted on 5 Dec 2019

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

What Is the Relationship between Dopamine and Effort?

Mark Walton, Sébastien Bouret

To cite this version:

Mark Walton, Sébastien Bouret. What Is the Relationship between Dopamine and Effort?. Trends in

Neurosciences, Elsevier, 2019, 42 (2), pp.79-91. �10.1016/j.tins.2018.10.001�. �hal-02394833�

Opinion

What

Is

the

Relationship

between

Dopamine

and

Effort?

Mark

E.

Walton

and

Sebastien

Bouret

*

The trade-off between reward and effort is at the heart of most behavioral theories, from ecology to economics. Compared to reward, however, effort remainspoorlyunderstood,bothatthebehavioralandneurophysiologicallevels. Thisisimportantbecauseunwillingnesstoovercomeefforttogainrewardisa commonfeatureofmanyneuropsychiatricandneurologicaldisorders.Arecent surge in interest in the neurobiological basis of effort has led to seemingly conflicting resultsregarding the roleofdopamine. We arguehere that, upon closer examination, there is actually striking consensus across studies: dopamineprimarilycodesforfuturerewardbutislesssensitivetoanticipated effortcost.Thisstrongassociationbetweendopamineandtheincentiveeffects ofrewardsplacesdopamineinakeypositiontopromotereward-directedaction.

Dopamine,Benefits,andCosts

Dopamineplaysacentralroleinreward-guidedlearningandmotivation.Ingeneralterms,there isabroadagreementonitsroleinappetitivemotivation,namelythatdopaminemediatesthe positiveinfluenceofpotentialfuturerewardsonbehavior(actionintensity,approach,learning, anddecisionmaking)[1–9].Inparticular,arecurringthemeisthatnormaldopamine transmis-sionis necessaryto activateorganisms,whichinturnallowsthem to exerteffortandgain positiveoutcomes[10].

However,beyondthisinitial consensus,thereismuchdebate overthe preciserelationship between dopamine – particularly the rapid and transient (‘phasic’) changes in dopamine observedwhen animals are presented with orapproach responseoptions – and effortful choices.Partofthereasonisthat,todate,experimentsthatdirectlyinvestigatedeffort-based decisionmakinganddopaminehavenotalwaysmeasuredtheseparableinfluencesofeffort andreward.Forinstance,inmanysituations,efforthasbeenstudiedinparadigmswherethe amountofforceproducedconjointlyscaledwiththeamountofreward(e.g.,[11]).In other experiments,subjectscouldearnmoreorbetterrewardsbyexertingmoreeffort(e.g.,[12,13]). Intheseexperimentsonecanonlyassesstherelativesensitivitytoeffort/reward;effortcannot bedisentangledfromrewardbecauseincreasesindifficultycanbecompensatedbyincreasein reward.Moreover, to be able to draw conclusions about the effectof different economic parametersondopamine,itisvitalthatparametersareconjointlyquantifiedandcontrolled[14–

17].Thisisparticularlyimportantwhendifferentstudiesrequireanimalstoovercomedifferent effortdemands(thevarietiesofeffortcostarediscussedinBox1).

Anotherreasonfortheconflictinginterpretationsabouttherelationshipbetweendopamineand effortisthatithasnotalwaysbeenclearlyspecifiedatwhichpointsduringaneffort-based decisiondopaminemightbeacting.Thereductioninwillingnesstoovercomeeffortconstraints observedfollowingpharmacologicalmanipulationscouldresultfromatleastthreefactors:(i) changesinthewaycostsandbenefitsarevaluedorcompared,(ii)thewillingnesstoinitiatea

Highlights

Comparedtoreward, effortremains poorlyunderstood,bothatthe beha-vioralandneurophysiologicallevels.

Dopaminehasbeenproposedas cen-traltoeffort-relateddecisionmaking, butitsroleisnotclearlydefined. In fact, the activity of midbrain dopamine neurons and mesolimbic dopamine levels are consistently modulated by anticipated future rewardmorestronglyandconsistently thaneffort,evenwhentheweightof reward and effort on behavior are equated.

Thesesignalsmaypromotedecisions toactbasedonthepotentialgainfrom afuturereward.

1

DepartmentofExperimental Psychology,UniversityofOxford, TinsleyBuilding,MansfieldRoad, OxfordOX13SR,UK

2

InstitutduCerveauetdelaMoelle Épinière(ICM),CentreNationaldela RechercheScientifique(CNRS), HôpitalPitiéSalpêtrière,75013Paris, France

*Correspondence:

mark.walton@psy.ox.ac.uk

(M.E.Walton)and

sebastien.bouret@icm-institute.org

(S.Bouret).

choice,and/or(i)themotivationtoovercomethecost(ofnote,theremightalsobechangesin howsubjectslearnabouteffort,butgiventhatdataonthisissuearecurrentlylacking,itwillnot bediscussedfurtherhere).

Tomakemattersevenmorecomplicated,therehavebeenmisunderstandingsabouthowto interpretthose existingdatathatdodirectlyspeaktotheissueoftherelationshipbetween dopamineandeffort.Wehaveundertakenstudiestosystemicallyexaminehowbothreward andeffortinfluencedopamineactivity[18]andrelease[19,20].Despiteusingdifferent techni-quesindifferentspeciesatdifferentlevelsofthedopaminesystem,wehaveobtained–inour opinion– strikinglyconvergentresults.Nevertheless, we have foundour databeingused, respectively,toargueforeffortcodingandeffortinsensitivityofdopamine,sometimesevenin thesamearticle[21,22].

Motivatedbythesediscrepancies, controversies, andconflictinginterpretations,we review herethe evidencefora rolefor phasicchangesindopamine duringeffort-related decision making.Weoutlineareaswherethereseemstobeaconsensusandareaswhere disagree-mentsremain.Finally,weaddresstheopenquestionsthatweseeasbeingcrucialforfurther progressinthisarea.

TheInfluence ofUpcomingEffortonDopamineNeuronalActivity

Thereisanextensiveliterature,bothinprimatesandrodents,showingthattheresponsesof manymidbraindopamineneuronselicitedbypredictivecuesreflecttheexpectedvalueofa futurereward.Suchsignalsareconsistentlylargerwhenthebenefitswillbegreater[18,23–25],

Box1.NotAllEffortIsEqual

Inneuroscientificexperimentsinvestigatingeffort-relatedchoice,effortisusuallytreatedasacostwhichcandiminish preferenceforrewardsorothergoalsthatanorganismmayfindmoredesirable[75].Perhapsnotsurprisingly,animals havebeenshowntobeverysensitivetoenergydemandsandmetabolicstatewhenforaging[52,76,77].Inrodent studiesofeffort-relateddecisionmaking,thetwomostcommonwaysofloadinganeffortcosthavebeen,onamaze,to placeaphysicalbarrierbetweenthestartpointoftherodentandarewardor,inanoperantchamber,toimposea lever-pressrequirementtoobtainthereward[78,79].Inmonkeysandhumans,studieshavetendedtouseeitherrepeated actions,asinsomerodentstudies,orahand-heldgripthatrequiresaparticularforcetobeproduced[11,18,30].More recently,therehavealsobeenattemptstoexamineotherformsofeffortthatareindependentofchangesinrewardrate, andwhichtendtofallunderthegeneralrubricof‘cognitive’effort[80–82].Allthesetypesofmanipulationcan systematicallyalterthechoicesoftheanimals,withhighercostsresultinginlowerpreferencefortherespectiveoption overalower-effortalternative.

However,theprecisenatureoftheseeffortcostsisnotstraightforwardtoisolate,quantify,andcompareacrossstudies. Forinstance,preferencesforscalingaphysicalbarriercanbetemporarilyreducedbyinducingphysicalfatigue[83], suggestingthatthiscostmayberelatedtoexertion.Perhapsmitigatingthiscaveat,however,isthefactthattheenergy requirementsofclimbingandjumpinginsmall animalssuchas rodentsarelikelynegligiblecomparedtoother homeostaticneeds([84]andA.Kacelnik,personalcommunication),suggestingthatabarrierimposesmorethana simplephysiologicalcost.Theeffortinvolvedinrepeatedrespondingisevenlessclear,particularlybecausethisusually involvesrespondingoverlongerperiodsoftime(note,however,thatseveralstudieshavedemonstratedthatchangesin effort-relatedchoicecannotbesimplyexplainedbyanincreaseddelaytorewardorchangeinrewardrate[85,86]). Moreover,effortcostsmaydifferentiallyinfluencedistinctelementsofmotivation.Forinstance, ahighresponse schedulecanchangethelikelihoodofinitiatinganactionbutnotinfluencesimplePavlovianresponsessuchas appetitivelippingbyamonkeyinanticipationoftheupcomingdeliveryofwater[36,87].Equally,pharmacological manipulationscanleavepatternsofschedulelength-relatedchoicerelativelyunalteredwhilemodulatingtheamountof forcethatisproduced[88].

Thesedistinctionsareimportantnotonlywhenconsideringthenormativedecision-makingstrategyindifferentmodel organismsbutalsobecausethereisincreasingevidencethattheneuralcircuitsnecessarytoassessandovercome differenttypesofeffortcostarepartiallydistinct.Forinstance,whereasdepletionofnucleusaccumbensdopaminecan causeanimalstobeeffort-aversewhenchoosingwhetherornottoscaleabarrierorwhenneedingtomakemultiple leverpresses,theyhavelittleeffectonchoicesbetweenoptionswithdifferentforcerequirements[89].

butcanalso benegatively influenced bycosts thatdirectly relatetoa decreaseinreward availability,suchasthedelayinthatfuturerewardoritsprobability[24,26–29].Thiscodingalso incorporatesindividualpreferencesfordifferentrewardtypes,andcanbemodulatedbyrisk inclination,fatigue,andsatiety[16,18,30,31].Notsurprisingly,thisrichsetoffindingshasbeen usedtosupportapositionthatdopamineactivityencodesasubjectivevalue,orutility,signal thatisappropriatefordrivingdecisionmaking[32].Ofnote,thesedatawereobtainedin well-trainedanimalsbecausevariabilityinbehaviorinearlyphasesoftrainingmakesthe interpreta-tionoftherelationshipbetweenbrainactivityandbehaviormoredifficult.Crucially,however,the animalsintheseexperimentsshowedsufficientbehavioralflexibilitytoruleoutaninterpretation intermsofhabit orovertraining.Instead,they masteredthe taskrules wellenoughto use informationappropriatelytomakedecisionsthatminimizecostsandmaximizebenefits. However,becauseweallknowfromdailylifesituations(forinstance,whetherornottorunto catchatrain)thatanydecisionreliesnotonlyontheanticipatedvalueoffuturerewardbutalso onpredictionsoftheeffortthatmustbeexertedtogainthisbenefit(Box2forconsiderationof howto defineacost).Therefore,if dopamineisto beconsidered asuitablesignalto drive economicchoice–inotherwords,ifdopaminerepresentsanetutilitysignal–itmustalsobe modulatedbyanticipatedeffortcosts.However,todate,theevidenceforthisislacking. Oneexamplecomesfromarewardscheduletaskwheremonkeyshavetocompleteaseriesof between1 and4 correct trialsto gain reward,signaled bya predictivecue indicatingthe numberoftrialsremainingintheschedule.Intheseexperiments,the majorityofdopamine neuronsshowednosensitivitytocuedinformationaboutschedulelength,eventhoughthis information had a strong influence on the motivation of the monkey to perform the task (i.e., monkeys were much more likely to engage in the task for short compared to long schedules)[33]. Notably, the strong but equivalent levels of dopamine activity to the first cueineachsequencewasmirroredbyanequivalentlevelofappetitivelipping,anappetitive Pavlovianresponsethathasbeenshowntoreflectinformationaboutoutcomevalue[34,35], followingpresentationofthatcue[36].Inotherwords,theintensityofthedopamineresponses appearsdrivenbyrewardinformation(indexedbylipping)ratherthanbychangesineffortlevel.

Box2.Effort:ASpecialTypeofCost?

Thetrade-offbetweencostsandbenefitsiscentraltomanydisciplinesinterestedinbehavior.Ineconomics,decision makingiscapturedbyacomparisonbetweentheutilityofpotentialitems(goodsorservices)onoffer([90]forreview). Theutilityofanitemincreaseswiththeexpectedbenefits(thesizeofareward,forinstance),anddecreaseswiththe expectedcosts.Inthatframe,anyfeaturethatleadstoadecreaseinutilitycanbeconsideredasacost:theeffort requiredtoobtainit,thedelayuntilitisobtained,ortheriskofnotobtainingit(ifitischosen).Optimaldecisionmaking occursherewhenthechoiceisbasedonaperfectevaluationofthe(expected)utilityoftheitemstobechosenfrom.In otherwords,whatismaximizedistheprecisionwithwhichthefunctionthatrelatestheobjectivemeasures(rewardsize, effort,delay,risk)mapsontothesubjectivevalueofanindividual,asmeasuredusingchoices.

Inbiology,thetrade-offbetweencostsandbenefitsisalsocentral,butwhatanimalsarethoughttooptimizeintheirbehavior is the allocationofenergy,ratherthanutility.This has beencapturedbythetheory ofoptimalforaging,whichpredictsthatthe energyspenttoobtainfoodshouldnotexceedtheenergyprovidedbythefoodobtained[77,91].Notethatanimalsmust alsosolvelife-historytrade-offs,forinstancebetweenreproductivecosts(mating,parenting)andsomaticefforts(growth, storage,maintenance)[92,93].Overtime,animalsarethoughttomaximizetherateofenergyintakeandminimizetheir energyexpendituretomaintainapositiveenergybalance.Crucially,thisbalancebetweenenergycostsandbenefitscould beachievedthroughtwoformallydistinct(butnotexclusive)strategies:(i)aselectionofappropriatebehaviorsthrough evolution,whichimpliesthatsuchbehaviorswouldbebothrelativelyfixedforagivenspeciesandadaptiveintheirnatural environment,and(ii)acognitiveevaluationofpotentialcostsandbenefitsassociatedwithagivenaction,whichwouldallow fineradaptationtoindividualchoices.Inthelattercase,effortcostscorrespondtotheanticipatedenergyexpenditureperunit timeofthepotentialactions,allowingplanningofupcomingbehavior.Variousspeciespresumablydisplaydistinct combinationsofthesestrategies,andthereforedifferentwaystoadjusttheirenergybalanceasafunctionoftheecological constraintsinwhichtheyevolved([94–100]andLouailetal.,unpublished).

Inlinewiththesedata,PasquereauandTurner[30]alsofoundlittleeffectofupcomingenergy expenditure on the activity of dopamine response, with only 13% of dopamine neurons encodingeffort(comparedto47%forreward)(Figure1A).

Althougheffortcostsdidinfluenceperformanceparameterssuchasreactiontimes,itcouldbe arguedthatthisdiscrepancybetweenrewardandeffortcodingcamefromthefactthatthe weightsofthecostandbenefitparametersonvaluewerenotcompletelyequated,particularly becausethewillingnessofthemonkeystoworkwasonlyinfluencedbytheeffortcostsona smallminorityofsessions.However,asubsequentstudybyVarazzanietal.[18]alsofoundthat dopamineneuronsweresignificantlymoresensitivetoinformationaboutupcomingrewardsize thanabout upcomingenergyexpenditure(Figure1B).Thisoccurredeven thoughherethe weightofrewardandeffortonthewillingnesstoworkofthemonkeyswassimilarinmagnitude (althoughopposite indirection),rulingoutthe possibilitythatthe differenceinsensitivityto rewardversuseffortindopamineneuronsisrelatedtoalowerbehavioralsensitivitytoeffort. Becauseofthisreducedsensitivitytoeffortcostsrelativetorewardbenefits,theresponsesof dopamineneuronsinthistask,inisolation,didnotpredictupcomingchoices,incontrastto othertaskswheretheoutcomevalueonlydependeduponrewardinformation[16,29].

TheInfluence ofUpcomingEffortonDopamineRelease

Onepossiblereasonforthelimitedevidenceforeffortencodingbydopamine,outlinedabove,is thatthestudies to date have mostlytargeted neurons in the substantianigra pars compacta, which projectstodorsalpartsofthestriatum.Bycontrast,manipulationsofdopaminetransmissionin rodentshasstronglyimplicatedmesolimbicpathwaysfromtheventraltegmentalarea(VTA), particularlytothecoreofthenucleusaccumbens(NAcC),asbeingnecessarytoallowanimalsto selectoptionsrequiringadditionaleffortforgreaterrewardoveralower-rewardalternative[10].At

0.3 0.2 0.1 0 0 0.5 1 0.1 −0.05 0 −0.5 0.5 1 0 (A)

R

e

gr

ession c

o

e

ffi

cien

ts

Time from cue onset (s)

(B)

Reward Effort

Figure1.RelativeSensitivityofDopamine(DA)NeuronstorewardBenefitsandEffortCosts.Sensitivityof

substantianigraparscompactadopamineneuronsinmonkeystoinformationaboutupcomingrewardbenefits(blue)and effortcosts(red)intworecentneurophysiologicalstudies(A,B)inbehavingmonkeys.Inbothstudies,monkeyswere requiredtoperformagivenactiontoobtainagivenreward.Rewardsizesandphysicaldifficulty(effortcost)were manipulatedindependentlyacrosstrials,andeachtrialstartedwithavisualcueindicatingtheupcomingeffortandreward. Regressioncoefficientswerecalculatedusingasliding-windowproceduretoevaluatethedifferenceinfiringacrossreward (blue)andeffort(redconditions)ateachtime-pointaroundstimulusonset.Thefiringofdopamineneuronsshowsreliable positiveencodingofrewardsize(firingratesaregreaterforcuesindicatinglargeversussmallrewards)within200msafter cueonset.Atthesametime,dopamineneuronsalsodisplaynegativemodulationbyeffortlevel(firingratesaresmallerfor largereffortlevels).Crucially,themagnitudeoftherewardmodulationisgreaterthantheeffortmodulationinbothstudies, eventhoughtheyclearlydifferinthewayanimalsneededtocopewiththeexpecteddifficulty.Thedifferenceinsensitivity cannotbesimplyduetoadifferenceinsubjectivesensitivitytoreward,ascomparedtoeffort,becausethesetwovariables hadanequivalentweightonthewillingnesstoworkoftheanimal,atleastin[18].Panel(A)reproduced,withpermission, from[30];panel(B)adapted,withpermission,from[18].

firstglance,thismightsuggestthatdopaminelevelsintheNAcCwouldbestronglysensitiveto upcomingeffortcosts.However,analternative,inaccordancewiththeelectrophysiologicaldata, wouldbethatdopaminelevelsmightprincipallyencodepredictionsoffuturerewardtoallow animalstodeterminewhatwouldbeanappropriatelevelofefforttoexert[5].

Totestthis,Ganandcolleagues[19]usedfast-scancyclicvoltammetrytomeasuresubsecond fluctuationsinNAcCdopaminelevelsinresponsetocuessignalingtheavailabilityofoneoftwo options.Oneoptionalwaysrequiredanimalstopaya‘reference’effortcostforreward(16lever pressesfor1foodpellet),andtheothereitherhadadifferentassociatedeffortcost(2or32lever presses)oradifferentrewardvalue(4or0foodpellets).Importantly,thereducedcostand increasedrewardlevelswere selectedto conferonaverageequal utility,andthereforeany differencesindopaminereleasebetweenthecostorrewardconditionscannotbeattributedto differencesinnetvalue.Exactlyliketheactivityofdopamineneurons,cue-eliciteddopamine release consistently scaled with anticipated future reward. Importantly, despite having an equivalentinfluence onresponselatency,learningrates,andoverallpreference,changesin effortcostshadsubstantiallylessimpactondopaminelevels(Figure2A).

However,itisnotthecase,ashasbeensometimesdescribed,thattherewasnoinfluenceof effortcostsondopaminelevels;themodulationbyeffortwaspresent,butwasweakerand transient(itwasonlypresentforcoststhatwerelessthanthereferenceandinanimalswhohad not undergone extensive training) (Figure 2B). As with neuronal activity, this meant that dopaminelevelsinisolationcouldnotbeusedtopredicteffort-basedchoices.

Acomplementarypicture emerges fromacompanionstudy by Hollonand colleagues[20].Instead ofoneoptionbeingobjectivelyofhighervaluethanthealternative,hereratswerefacedwitha decisionbetweenonelow-effortcost/low-rewardoptionandanotherhigher-effort cost/high-rewardalternative.Inseparateconditions,theeffortcostassociatedwiththehighrewardwas alteredsuchthatinoneconditiontheanimalspreferredovercomingthehighereffort/highreward, whereasinthe otherthe costwas increaseduntiltheaveragepreferencewasforthe low-cost/low-rewardoption.Averagecue-eliciteddopaminelevelsweresignificantlygreateronhigh-reward thanonlow-rewardtrials.Asbefore,therewasalsoasmall,consistentinfluenceofeffortcostson cue-evokeddopamine,particularlytheinitialcue-evokedresponse.Importantly,thisinfluence wasagainmuchweakerthantheexpectationoffuturereward,meaningthat,eveninconditions wheretheratspreferredandrespondedfastertothelow-rewardoption,averagedopaminelevels werestillgreaterfollowingpresentationofthehigh-effort/high-rewardoption(Figure2C).

WhatRoleMightRapidChangesinDopaminePlayinEffort-Related

Choice?

Theemergingpicturefromthestudiesdiscussedaboveisoneofconvergence:dopamineis reliablymodulatedbyexpectationsoffuturereward,whereasthenegativeinfluenceofeffort costsismuchmorelimitedandisonlyobservedinrestrictedtaskconditions(Figure3,Key Figure).Whatisremarkableisthatsuchaconsistentpictureemergesdespitedifferencesin species,task,technique,anddopaminesubsystem(nigrostriataldopamineneuronsversus mesolimbic dopamine release) (Box 3 for discussion of homogeneity and diversity of dopamine). Moreover,these results align with several other studies implicating dopamine morestronglyinrewardthanineffortprocessing[37–40].Thus,webelievethatthisfeature isneitheranecdotalnorduetoanexperimentalartifact.

Crucially,thisholdstrueeventhough,behaviorally,thechoicesoftheanimalswerestrongly modulatedbybotheffortandrewardtoadegreethatwasquantitativelyequivalent.Inother

Benefit condion Cue-elicit ed Δ [D A] 1 pellet 4 pellets 5nM 6s Cue onset 7s 6s 5nM 16 presses 2 presses Cue onset Cost condions 5nM 32 presses 16 presses 6s Cue onset LR / LC HR / MC

High reward opon preferred

Benefit (4 vs 1) 0 3 6 9 12 15 18 -10 -5 0 5 10 15 20 25 [DA] HR/LC - [DA] REF (nM) Cost (2 vs 16) 0 3 6 9 12 15 18

Extended training (≥10 sessions) Standard training (<10 sessions)

LR / LC HR / HC

Low reward opon preferred

Choice index HR > LR LR > HR

Dopamine discriminability index

HR > LR LR > HR LR/LC v HR/MC LR/LC v HR/HC Best linear fit (A) (C)

Number of sessions’ experience with conngencies

7s -1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1 5nM Cue onset 5nM Cue onset Cue-elicit ed Δ [D A] 0 50 100 Choice (%) 1 2 3 0 R e sponse la te ncy (s) 0 50 100 Choice (%) 1 2 3 0 R e sponse la te ncy (s) 0 50 100 Choice (%) 1 2 3 0 R e sponse la te ncy (s) (B) Figure2.

(Figurelegendcontinuedonthebottomofthenextpage.)

Cue-ElicitedDopamine(DA)LevelsArePrimarilyModulatedbyExpectedFutureBenefitsandNot

byAnticipated Costs.(A)Behavior(choiceperformanceandresponselatencies,upperpanels)andcue-elicited

dopaminelevels(lowerpanels),recordedwithfast-scancyclicvoltammetryinratnucleusaccumbenscore,inconditions

words,there isa clearuncouplingbetweencue-elicited dopamineactivity andtrial-by-trial effort-relateddecisions.Therefore,wewouldcontendthatdopaminecannotactasa‘common currency’thatintegratesacrossalleconomicvariablestosignalthenetutilityofanavailable option(or,forthatmatter,deviationsfromaverageexpectation,i.e.,autilitypredictionerror). Howthencanonereconcilethesedatawiththecareful,elegantexperimentsshowingastrong relationshipbetweentheactivityofdopamineneuronsandquantitativeutilitypredictionerror

[32,41]?Onewaywouldbesimplytoassumethatchoicesaboutwhetherornottoinvesteffort form a fundamentallydistinct class ofdecisions that are separate from other cost–benefit scenariossuchasintertemporalorriskychoice(Box2).Incontrasttocostssuchasdelayor probability,effortismuchmorecloselyalignedwithactionsystems.Indeed,severalstudies haveshownthattheevaluationofeffortcostsinvolvecortical–striatalcircuitscloselyrelatedto responseselection,suchasdorsomedialfrontalcortex(anteriorcingulateandsupplementary motorcortex)anddorsalstriatum/putamen[42–48],andnotcircuitscenteredonorbitaland ventromedial prefrontal cortices and parts of ventral striatum, which are more commonly implicatedinbenefits-baseddecisions(e.g.,[49,50]).Therefore,decisionsrelyingona cost-–benefitanalysiswouldresultfromajointinfluenceofthesetwosystemsonmotoroutput. However,althoughitisindisputablethatdopamineencodesparametersaboutexpectedfuture outcomesrelevanttoadecision,thereisalsoincreasingevidencethatitsfunctionmaynotbeto directlyguideactionselectionbetweensimultaneouslyavailableoptions.Indeed,evenwhen choicesaredefinedsolelybydifferencesinoutcomes,midbraindopamineactivityandNAcC dopaminelevelsinresponseto individualoptionsarenotnecessarilyareliable predictorof preferencebecausetheyoften signalthevalueofanupcomingchoiceeven whenit isnot reward-maximizing[29,51]. Fromthis perspective,dopamineseemsto signal thepotential benefitsofadecisionthathasalreadybeenfinalized.

whereratswerepresentedwithoptionssignalingavailabilityofafuturereward(foodpellets)afterpayinganeffortcost (repeatedleverpresses).Ineachconditiontherewasonereferenceoption(16presses/1pellet,bluebar/line)andone alternativethatwasassociatedwithahigherbenefit(16presses/4pellets,purplebar/line,leftpanels),lowercost(2 presses/1pellet,redbar/line,midpanels),orhighercost(32presses/1pellet,burgundybar/line,rightpanels).Filledlines correspondtothepreferredoptionofeachpair.(B)Differenceinpeakcue-eliciteddopamineonhigherbenefit/lowercost trials([DA]HR/LC)comparedtoreferencetrials([DA]REF)inindividualratsasafunctionoftheamountofexperiencetheyhad

withthoseparticularcost–benefitcontingencies(standardtraining,<10sessionsofexperience,lightercoloreddots; extendedtraining,10sessionsofexperience,darkercoloreddots).Althoughagreaterpeakdopaminewasconsistently recordedonhigher-benefittrialsregardlessoftrainingexperience(leftpanel),thedifferenceindopaminebetween lower-costandreferencetrialsreducedwithincreasingexperienceofthiscondition(rightpanel).Specifically,afterextended trainingwiththelower-costcontingencies,therewasnoreliabledifferenceincue-eliciteddopamineonlowercostand referencetrials,eventhoughratsstillexhibitedastrongpreferenceforthelower-costoptionandrespondedfasteron lower-costtrials.Notethat,unlikeinsomestudies(e.g.,[60]),thecost/benefitcontingenciesintheexperimentsdepicted herereversedeachday,andthatthesedatacomeonlyfromtrialsafteranimalshadachievedastablepreferenceforthe higher-benefitorlower-costoptionineachsession.Therefore,theseresultsneitherreflectlearning(inearlysessions)nor habitualresponding(inlatersessions).Panelsadapted,withpermission,from[19].(C)(Upperpanel,left)dopamine responsestocuessignalingavailabilityofeitheralow-reward/low-costoption(LR/LC:1reward/4presses)ora high-reward/mid-costoption(HR/MC,8presses/4rewards;leftpanel).(Upperpanel,right)Asintheleftpanel,butcomparing responsestotheLR/LCoptiontoaHR/high-costoption(HR/HC,32presses/4rewards).Thecost–benefit contin-gencieswereagainreversedeveryfewsessions,anddopaminedatawerecollectedfromtrialsafteranimalshadachieved astablepreferencefortheHR/MC(versusLR/LC)orLR/LC(versusHR/HC).(Lowerpanel)A‘dopaminediscriminability index’plottedagainstaveragepreferenceacrossasession,quantifiedusinga‘choiceindex’(HR LRchoices).The dopaminediscriminabilityindexwasbasedontheareaunderthereceiveroperatingcharacteristic(auROC)classifying dopaminereleaseasdiscriminableonHRfromLRtrialsineachsession.Ascanbeobserved,althoughthechoiceindex spansthefullrangeofvalues,thedopamineindexisstronglypositivelyskewed,showingthatitwasmorecommonto classifydopaminereleaseasgreateronHRthanLRtrialsirrespectiveofpreference(bluedots:LR/LCversusHR/MC;red dots:LR/LCversusHR/HC).Panelsadapted,withpermission,from[20].

What then might bethe functionsof transientincreases indopamine beforeeffort-related actions?Onepossibilityisthatdopaminedoesnotsignalpredictionsoffuturerewardtoguide whatactiontotake,butinsteadprovidesasignaltoshapewhether(andpossiblyalsowhenand howfast)toactgiventhepotentialbenefitsoftakingapresentedopportunityinaparticular environment. In naturalistic settings, potential rewards are often encountered sequentially ratherthan simultaneously.Thisimplies that akey computation, recurringacross species, iswhetherornotto engagewitha presentedopportunity[52].Thus,wewouldarguethat dopamineactivationreflectstheincentiveinfluenceofapotentialrewardonbehaviorthatcould leadtoobtainingit(Figure3).Whilesuchsignalswilltendtobeelicitedbyexternalstimuli,they cannonethelessbecontextuallyregulatedbyafferentinput[53,54],allowingcontroloverwhen itisbeneficialtoengageversuswhenitisbettertodisplayrestraint.

Forinstance,NAcCdopaminelevelsaredynamicallymodulatednotonlybyrewardprediction errorsbutalsobywhetherornotanactionshouldbemadetogainthereward[55,56].Similarly, acomparablephenomenonhasbeenobservedinthefiringofdopamineneuronsrecorded fromsubstantianigraparscompactainmonkeys[18,36],indicatingthatthisphenomenonis again reliable across species, task, recording technique, and dopamine subsystem. Put another way, dopamineactivity would reflect an instantaneous estimate of the change in averagerewardrate–orpotentialnetenergygain,basedontheinternalstateoftheanimal KeyFigure

Influence

of

Expected

Reward

and

Effort

on

Behavior

+

+

–

+/?

Effort

Incenve

Benefits

Costs

Expected mental and

physical resources

for acon Expected reward (nature, size, ming

DA

Figure3.Informationaboutupcomingbenefitsandcostsareseparatelyintegratedintoincentivesandeffort,respectively.

Incentiveshaveadualeffectonbehavior:first,theyhaveapositiveinfluenceonactionselection(animalsselectthemost beneficial options)and,second, incentive processesstimulate action execution.Effort,definedas theamount of anticipatedresourcesnecessaryforaction,negativelyaffectsdecisions:animalstendtoselectactionsthatminimize energyexpenditure(Box2).Nonetheless,theinfluenceofthisinformationonactionexecutionmightbedualistic:although anticipatedgreaterdemandcanretardactioninitiation,animalsoncecommittedmayneedtoboosttheirmotivationto overcomeeffortcosts.However,todate,theprocessesinvolvedinsurmountingeffortfulchallengesremainlittleexplored. Abbreviation:DA,dopamine.

[57,58]–thatwouldbeachievedbypursuinganoption.Thissignalcould,inturn,motivatea newcourseofactiontobeinitiated[59].Indeed,becauseeffortcostscancauseanimalstoact moreslowlyorperformlessreliably, itisworthconsideringthatwhatmay appearinsome experimentalsituationsas aninfluenceofeffortondopaminemay,without carefulcontrols (suchasequatingtrialratesirrespectiveofhigh-orlow-effortchoices:compare[19]with[60]), actuallyrelatetochangesinrewardavailability.

Wewouldspeculatethatthissignalincorporateseffortcostsonlyifthepursuitoftherewardhas aninfluenceonthenetincomeorrewardrateoftheanimal.Thiscloseconnectionwithratesof rewardmightalsoexplaintheimportantlinkbetweendopamineandtimejudgments[61],and betweendopamineandactionvigor[37,62,63],giventhatdecisionsofwhenandhowfastto actboth willinfluence therate ofevents.Ofnote, effortcosts inlaboratoryparadigmsare typicallyofrelativelylowenergydemandandalsotendnottoinducelargefluctuationsofreward rate within an individual session. According to the framework outlined above, a natural consequence of these characteristics would bethat recorded dopamine signalscome to bedominatedbypredictionsoffuturereward.

LinkingRecording andManipulationStudies

Theideasdiscussedinearliersectionsabouttherelationshipbetweendopamineandeffort emergedprimarilyfromrecentcorrelativerecordingstudiesinbehavinganimals.Inaddition, there is a history of studies examining the effects of pharmacological manipulations of dopaminetransmissionduringeffort-baseddecisionmaking[2,8,10].However,itisnotalways straightforwardtoalignthefindingsfromrecordingstudieswiththepharmacologicalliterature onthistopicgiventhedifferencesintimescaleandapproach.Animportantfuturestepwillbeto combinerecordingand manipulation techniques,andto usemanipulation techniques with

Box3.IsDopamineSignalingaSingleFunctionalEntity?

Forthesakeofsimplicity,wehavechosentotreatthedopaminesystemasafunctionalunit.Indeed,ourdeliberateaim hasbeentofocusontheconcordanceintheliteratureregardingdopamineandeffort,highlightingthecommonfeatures thatwerereliableenoughtobedescribedregardlessofthespecies,thetask,orthemethodusedtomeasureactivity. Nonetheless,wedonotwishtounderplaythewidespreadevidencefordiversityofmolecularfeatures,anatomical connections,andcodingacrossdopamineneurons.Forinstance:

(i)Attheanatomicallevel,theprojectionsofdopamineneuronsvaryasafunctionoftheirventromedialtodorsolateral locationinthemidbrainandoftheirmolecularproperties[101–103].Note,however,thatthesearegradeddistinctions

[104],andseveralstudiespointtointeractionsamongthesechannels[105].

(ii)Ataphysiologicallevel,asignificantproportionofdopamineneuronswithinparticularmidbrainregionsrespondina stereotypedwaytorewardsandtheirpredictors[6,28].However,dopamineneuronsrecordedinthedistinctregionsof themidbraincanalsodisplaydistinctfunctionalcorrelates,particularlyconcerningcodingofrewardandaversion ([71,106];butseealso[107]).Moreover,therecanevenbeheterogeneousactivitypatternsinsubpopulationsof dopamineneuronswithinmidbrainregions[18,30,70].Suchdiversityisalsoapparentinterminalregions[72,73,108]. (iii)Atafunctionallevel,experimentshaveshownthatartificialactivationofdopamineneuronsortheirprojections originatingfromdistinctmidbrainregionscanhavedistinctfunctionalconsequences[109,110],evenifstimulationof bothSNcandVTAcaninduceappetitiveeffects[111].

Althoughthisevidenceconvergestosuggestthatthedopaminesystemcanbebrokendownintoseveralfunctional units,theexactboundariesbetweentheseunitsandtheexactnatureoftheirfunctionsremaintobeelucidated. Crucially,thelevelofdescriptionremainsstronglydependentuponthequestionofinterestandthescaleatwhichitis studied(e.g.,theinformationencodedbysingledopamineneuronsversusthatcommunicatedbydiffusion-based volumetransmissioninterminalregions).Ausefulcomparisonmaybewithprimaryvisualcortexwhich,forsome purposes,canbeconsideredasafunctionalentityeventhoughitcanbebrokendownintoseveralpatchesbasedon oculardominanceorthepositionofthevisualinput,andisalsopartofaninterconnectedsetofcorticalregionsinvolved invisualprocessing.

bettertemporalresolution,tounderstandbetterhowdisruptingneurochemistrycaninturn influence dopamine coding. Nonetheless, even given these caveats, we believe that the proposed role for dopamine outlined here can fit well alongside the existing literature on theeffectsofdisruptingdopaminetransmission.

Afrequentobservation,forinstance,isthatblockingdopaminereceptorsreducesthelikelihood andspeedofengagementasafunctionoffuturereward[9,64,65].Thisobservationfitsnicely intoapicturewhererapidincreasesindopamineprovideaunitarysignaltodifferentterminal regionsaboutthepotentialnetgainofapresentedopportunity,andcanthushelptoauthorize decisionsaboutwhentoactandwhennottoact.Thefactthatwehaveobservedsuchsignals particularly clearly in situations when animals are required to switch from an inactive or disengagedstatetoanactiveonemayalsoberelatedtoaparticularrolefordopaminewhen needingtoinitiateanon-stereotypedresponse[3].

Wehaveemphasizedwhatweseeasaconsensusintheliteratureonthepointthateffortcosts havealimitedinfluenceondopamineactivity.Thismayseem,atfirstglance,contradictoryto thefactthatpharmacologicalmanipulationsofdopamineoftencausepronouncedchangesin allocationofeffort.Therearesomesubtletieshere,however,thatcanperhapseasilybemissed, particularlywhencomparingcorrelativeandinterventionalapproaches.Onesubtledistinction isthat,evenifdopaminecanpromoteenergyexpenditure,itonlydoessoasafunctionofthe upcomingreward,andnotasafunctionoftheupcoming(energy)costitself.Crucially,thefew recentexperimentsthatexaminedtheinfluenceofdopaminetreatmentsintaskswhereefforts costsandrewardbenefitsweredissociated,alsofoundastrongerinfluenceonreward-based overeffort-baseddecisions[37,40].Therefore,evenifthesemanipulationslackthetemporal and anatomical precision of recording studies, they reinforce the idea that the dopamine systemasawholeismuchmoresensitivetopotentialbenefitsthantopotentialeffortcosts,and demonstratethegeneralityofthisrelation,aswellasitscausalnature.

Ifeffortcostsare notdirectly encodedbydopaminesignals,anobviousquestionremains: whichcircuitssignaleffortcosts,andhowthesemightinteractwithdopamine?Asdescribed earlier,thereisgeneralagreementthatanteriorcingulatecortexanddorsomedialmotorareas playsomeroleinrepresentingeffort,andtheformerhasrecentlybeenshowntomodulateVTA activityduringanefforttask[53].Whenitcomestothequestionofwhatneuralpathwaysallow effortcosts to be overcome, one intriguing possibility is that, despite all the attention on dopamine,itmaybethatotherneurochemicalsystemsarecrucialinthiscontext.Forinstance, locuscoeruleusneuronsarestronglyactivebothimmediatelybeforeandafteraneffortfulaction is initiated [18,36], suggesting that noradrenaline might be crucial to mobilize energy to overcomean effort cost.Some striatalcholinergic interneurons also increase their activity whenengaginginahigh-effortorsmall-rewardtrial[66].Althoughserotoninhastendedtobe linkedmorecloselytodelay-baseddecisions,recentevidencesuggestsitmayalsoaffecthow effortcosts accumulateovertimeaswellas thevigorofresponding(thelatterpossiblyvia dopamineinteractions)[67,68].

ConcludingRemarks

Thefunctionofdopaminehaslonggeneratedagreatdealofdebate,andwilllikelycontinueto doso.Wehighlightherewhatwebelievetobeaconspicuouspointofconsensus.Crucially,this consensusisonlyarrivedatwhenthedataareconsideredforwhat theyspecifically show, puttingasideanyattempttofitthemintooneorotherestablishedposition.Tous,itseemsthat thedataconvergetoapicturewheredopaminesignalsdisplaystrongandconsistentreward encoding,butlimitedandsometimestransienteffortencoding.

OutstandingQuestions What is the timescale of dopamine action?Doestheaveragebackground (‘tonic’)dopaminelevelhavearole dis-tinctfromthat ofthetransient cue-evokedsignalsinmotivatinganimals toovercomeeffort,orhasthe distinc-tionbetweenitsactionatfastandslow timescalesbeenoverstated?

Doesdopaminealsoinfluence move-ment parameters? NAcC dopamine levelsdonotstraightforwardly corre-latewith movementtimes.However, severalstudieshavesuggestedalink between nigrostriatal dopamine, at least, and the vigor with which an actionisperformed.

What is the relationship between dopamineandotherstructures,such asanteriorcingulatecortex,andother neurochemicalsystemsduring effort-relatedchoices?

Clearly,therearemanyissuesthatremaintobeaddressed(seeOutstandingQuestions)andwe wouldbesurprisediftherearenonewpointsofdivergenceoncevariousaspectsofeffortand differentdopamine pathwaysare examinedinmoredetail.For instance,our starting positionis that theconvergenceweobservesuggeststheremaybeaunitary,oratleastgeneral,functionfor dopamineinpromotingactionbasedonpredictionsoffuturebenefits(cf[69]and[9]for comple-mentaryideas).However,givenongoingdebatesconcerningthedegreeofdiversityofcodingin midbrain dopamineneurons (e.g.,[18,30,70,71]), as well asclearvariations inpatternsof release in terminalregions[72,73],we are cognizant thatthere may befiner-graineddistinctions to bedrawn. Inparticular,itwillbeimportanttounderstandbetterhowtheroleofdopamineinspontaneous movementandtimingdovetailswithsomeofourviewpoints[61,63,70,74].Nonetheless, we believethatthepotentialtofindcorrespondencesacrossspecies,technique,andtaskshowsthat itshouldbepossibletoharnesstheadvantagesthatthedifferentapproachesconfertouncover commonandconservedfunctionsofdopamine.

Acknowledgments

M.E.W.wassupportedbyaWellcomeTrustSeniorResearchFellowship(202831/Z/16/Z).S.B.wassupportedbythe CNRSandaresearchgrantfromtheFondationdeFrance.WewouldliketothankRobTurnerandBenjaminPasquereau forallowingustoreproducepartoftheirdata,andNickHollonforhelpwiththeadaptationofpartsofFigure2.M.E.W. wouldliketoacknowledgetheinfluenceofPaulPhillipsonshapingseveraloftheideasexpressedhere,aswellasuseful discussionswithAlexKacelnik.

References

1. Berridge,K.C. (2007)The debateoverdopamine’srolein

reward:thecaseforincentivesalience.Psychopharmacology

(Berl.)191,391–431

2. Chong,T.T.andHusain,M.(2016)Theroleofdopamineinthe

pathophysiologyandtreatmentofapathy.Prog.BrainRes.229,

389–426

3. Nicola,S.M.(2010)Theflexibleapproachhypothesis:unification

ofeffortandcue-respondinghypothesesfortheroleofnucleus

accumbens dopamine in the activation of reward-seeking

behavior.J.Neurosci.30,16585–16600

4. Robbins,T.W.andEveritt,B.J.(1992)Functionsofdopaminein

thedorsalandventralstriatum.Semin.Neurosci.4,119–127

5. Phillips,P.E.etal.(2007)Calculatingutility:preclinicalevidence

forcost–benefitanalysisbymesolimbicdopamine.

Psychophar-macology(Berl.)191,483–495

6. Schultz,W.(2002)Gettingformalwithdopamineandreward.

Neuron36,241–263

7. Guitart-Masip,M.etal.(2014)Actionversusvalenceindecision

making.TrendsCogn.Sci.18,194–202

8. Pessiglione,M.etal.(2017)Whynottryharder?Computational

approachtomotivationdeficitsinneuro-psychiatricdiseases.

BrainPublishedonlineNovember29,2017.http://dx.doi.org/

10.1093/brain/awx278

9. Berke,J.D.(2018)Whatdoesdopaminemean?Nat.Neurosci.

21,787–793

10. Salamone,J.D.andCorrea,M.(2012)Themysterious

motiva-tionalfunctionsofmesolimbicdopamine.Neuron76,470–485

11. Schmidt,L. etal.(2008) Disconnectingforce frommoney:

effectsofbasalgangliadamageonincentivemotivation.Brain

131,1303–1310

12. Cousins,M.S.andSalamone,J.D.(1994)Nucleusaccumbens

dopaminedepletionsinratsaffectrelativeresponseallocationin

anovelcost/benefitprocedure.Pharmacol.Biochem.Behav.

49,85–91

13. Cousins,M.S.etal.(1996)Nucleusaccumbensdopamine

depletionsalterrelativeresponseallocationinaT-mazecost/

benefittask.Behav.BrainRes.74,189–197

14. Fiorillo,C.D.etal.(2013)Multiphasictemporaldynamicsin

responsesofmidbraindopamineneuronstoappetitiveand

aversivestimuli.J.Neurosci.33,4710–4725

15. Fiorillo,C.D.(2013)Twodimensionsofvalue:dopamineneurons

representrewardbutnotaversiveness.Science341,546–549

16. Stauffer,W.R.etal.(2014)Dopaminerewardpredictionerror

responsesreflectmarginalutility.Curr.Biol.24,2491–2500

17. Matsumoto,H.etal.(2016)Midbraindopamineneuronssignal

aversion in a reward-context-dependent manner. eLife 5,

e17328

18. Varazzani,C.etal.(2015)Noradrenalineanddopamineneurons

inthereward/efforttrade-off:adirectelectrophysiological

com-parisoninbehavingmonkeys.J.Neurosci.35,7866–7877

19. Gan,J.O.etal.(2010)Dissociablecostandbenefitencodingof

future rewards by mesolimbicdopamine. Nat. Neurosci.13, 25–27

20. Hollon,N.G.etal.(2014)Dopamine-associatedcachedvalues

arenotsufficientasthebasisforactionselection.Proc.Natl.

Acad.Sci.U.S.A.111,18357–18362

21. Schultz,W.etal.(2015)Phasicdopaminesignals:from

subjec-tiverewardvaluetoformaleconomicutility.Curr.Opin.Behav.

Sci.5,147–154

22. Shenhav,A.etal.(2017)Towardarationalandmechanistic

accountofmentaleffort.Annu.Rev.Neurosci.40,99–124

23. Tobler,P.N.etal.(2005)Adaptivecodingofrewardvalueby

dopamineneurons.Science307,1642–1645

24. Roesch,M.R.etal.(2007)Dopamineneuronsencodethebetter

optioninratsdecidingbetweendifferentlydelayedorsized

rewards.Nat.Neurosci.10,1615–1624

25. Nakamura,K.etal.(2008)Reward-dependentmodulationof

neuronalactivityintheprimatedorsalraphenucleus.J.

Neuro-sci.28,5331–5343

26. Kobayashi,S.andSchultz,W.(2008)Influenceofrewarddelays

on responses of dopamine neurons. J. Neurosci. 28,

7837–7846

27. Fiorillo,C.D.etal.(2003)Discretecodingofrewardprobability

and uncertainty by dopamine neurons. Science 299,

28. Eshel, N.etal. (2016)Dopamineneurons sharecommon

responsefunctionforrewardpredictionerror.Nat.Neurosci.

19,479–486

29. Morris,G.etal.(2006)Midbraindopamineneuronsencode

decisionsforfutureaction.Nat.Neurosci.9,1057–1063

30. Pasquereau,B.andTurner,R.S.(2013)Limitedencodingof

effortbydopamineneuronsinacost–benefittrade-offtask.J.

Neurosci.33,8288–8300

31. Satoh,T.etal.(2003)Correlatedcodingofmotivationand

outcomeofdecisionbydopamineneurons.J.Neurosci.23,

9913–9923

32. Stauffer,W.R.etal.(2016)Componentsandcharacteristicsof

thedopaminerewardutilitysignal.J.Comp. Neurol.524,

1699–1711

33. Ravel,S. and Richmond, B.J. (2006) Dopamineneuronal

responsesinmonkeysperformingvisuallycuedreward

sched-ules.Eur.J.Neurosci.24,277–290

34. Bouret,S.andRichmond,B.J.(2010)Ventromedialandorbital

prefrontalneuronsdifferentiallyencodeinternallyandexternally

driven motivational values in monkeys. J. Neurosci. 30,

8591–8601

35. San-Galli,A.andBouret,S.(2015)Assessingvalue

represen-tationinanimals.J.Physiol.Paris109,64–69

36. Bouret,S.etal.(2012)Complementaryneuralcorrelatesof

motivationindopaminergicandnoradrenergicneuronsof

mon-keys.Front.Behav.Neurosci.6,40

37. LeBouc,R.etal.(2016)Computationaldissectionofdopamine

motorandmotivationalfunctionsinhumans.J.Neurosci.36,

6623–6633

38. Wanat,M.J.etal.(2010)Delaysconferredbyescalatingcosts

modulatedopaminereleasetorewardsbutnottheirpredictors.

J.Neurosci.30,12020–12027

39. Skvortsova,V.etal.(2017)Aselectiverolefordopaminein

learningtomaximizerewardbutnottominimizeeffort:evidence

from patients with Parkinson’s disease. J. Neurosci. 37,

6087–6097

40. Chong,T.T.etal.(2018)Dissociationofrewardandeffort

sensitivityinmethcathinone-inducedParkinsonism.J.

Neuro-psychol.12,291–297

41. Schultz,W.etal.(2017)Thephasicdopaminesignalmaturing:

fromrewardviabehaviouralactivationtoformaleconomicutility.

Curr.Opin.Neurobiol.43,139–148

42. Croxson,P.L.etal.(2009)Effort-basedcost–benefitvaluation

andthehumanbrain.J.Neurosci.29,4531–4541

43. Kurniawan,I.T.etal.(2010)Choosingtomakeaneffort:therole

ofstriatuminsignalingphysicaleffortofachosenaction.J.

Neurophysiol.104,313–321

44. Klein-Flugge,M.C.etal.(2016)Neuralsignaturesofvalue

com-parisoninhumancingulatecortexduringdecisionsrequiringan

effort-rewardtrade-off.J.Neurosci.36,10002–10015

45. Walton,M.E.etal.(2002)Theroleofratmedialfrontalcortex

in effort-based decision making. J. Neurosci. 22,

10996–11003

46. Prevost,C.etal.(2010)Separatevaluationsubsystemsfordelay

andeffortdecisioncosts.J.Neurosci.30,14080–14090

47. Walton,M.E.etal.(2003)Functionalspecializationwithinmedial

frontalcortexoftheanteriorcingulateforevaluating

effort-relateddecisions.J.Neurosci.23,6475–6479

48. Hosokawa,T.etal.(2013)Single-neuronmechanisms

underly-ingcost–benefitanalysis infrontalcortex.J.Neurosci.33,

17385–17397

49. San-Galli,A.etal.(2018)Primateventromedialprefrontalcortex

neuronscontinuously encodethe willingnesstoengagein

reward-directedbehavior.Cereb.Cortex28,73–89

50. Strait,C.E.etal.(2015)Signaturesofvaluecomparisonin

ventralstriatumneurons.PLoSBiol.13,e1002173

51. Walton,M.E.etal.(2011)Theinfluenceofdopaminein

gener-atingactionfrommotivation.InNeuralBasisofMotivationaland

CognitiveControl(Mars,R.B.,ed.),pp.163–187,MITPress

52. Aw,J.M.etal.(2011)Howcostsaffectpreferences:

experi-mentsonstatedependence,hedonicstateandwithin-trial

contrastinstarlings.Anim.Behav.81,1117–1128

53. Elston,T.W.andBilkey,D.K.(2017)Anteriorcingulatecortex

modulationoftheventraltegmentalareainanefforttask.Cell

Rep.19,2220–2230

54. Jones,J.L.etal.(2010)Basolateralamygdalamodulates

termi-naldopaminereleaseinthenucleusaccumbensand

condi-tionedresponding.Biol.Psychiatry67,737–744

55. Syed,E.C.etal.(2016)Actioninitiationshapesmesolimbic

dopamineencodingoffuturerewards.Nat.Neurosci.19,34–36

56. Roitman,M.F.etal.(2004)Dopamineoperatesasasubsecond

modulatoroffoodseeking.J.Neurosci.24,1265–1271

57. McCutcheon,J.E.(2015)Theroleofdopamineinthepursuitof

nutritionalvalue.Physiol.Behav.152,408–415

58. Beeler,J.A.etal.(2012)Puttingdesireonabudget:dopamine

andenergyexpenditure,reconciling rewardandresources.

Front.Integr.Neurosci.6,49

59. Hamid,A.A.etal.(2016)Mesolimbicdopaminesignalsthevalue

ofwork.Nat.Neurosci.19,117–126

60. Day,J.J.etal.(2010)Phasicnucleusaccumbensdopamine

releaseencodeseffort-anddelay-relatedcosts.Biol.Psychiatry

68,306–309

61. Soares,S.etal.(2016)Midbraindopamineneuronscontrol

judgmentoftime.Science354,1273–1277

62. Panigrahi,B.etal.(2015)Dopamineisrequiredfortheneural

representation and control of movement vigor. Cell 162,

1418–1430

63. daSilva,J.A.etal.(2018)Dopamineneuronactivitybefore

actioninitiationgatesandinvigoratesfuturemovements.Nature

554,244–248

64. Nicola,S.M.etal.(2005)Nucleusaccumbensdopaminerelease

isnecessaryandsufficienttopromotethebehavioralresponse

toreward-predictivecues.Neuroscience135,1025–1033

65. Choi,W.Y.etal.(2009)DopamineD1andD2antagonisteffects

onresponselikelihoodandduration.Behav.Neurosci.123,

1279–1287

66. Nougaret,S.andRavel,S.(2015)Modulationoftonicallyactive

neuronsofthemonkeystriatumbyeventscarryingdifferent

forceandrewardinformation.J.Neurosci.35,15214–15226

67. Bailey,M.R.etal.(2018)Aninteractionbetweenserotonin

receptorsignalinganddopamineenhancesgoal-directedvigor

andpersistenceinmice.J.Neurosci.38,2149–2162

68. Meyniel,F.etal.(2016)Aspecificroleforserotoninin

overcom-ingeffortcost.eLife5,e17282

69. Lau,B.etal.(2017)Themanyworldshypothesisofdopamine

predictionerror:implicationsofaparallelcircuitarchitecturein

thebasalganglia.Curr.Opin.Neurobiol.46,241–247

70. Dodson,P.D.etal.(2016)Representation ofspontaneous

movementbydopaminergicneuronsiscell-typeselectiveand

disruptedinparkinsonism.Proc.Natl.Acad.Sci.U.S.A.113,

E2180–E2188

71. Matsumoto,M.andHikosaka,O.(2009)Twotypesofdopamine

neurondistinctlyconveypositiveandnegativemotivational

sig-nals.Nature459,837–841

72. Brown,H.D.etal.(2011)Primaryfoodrewardand

reward-predic-tivestimulievokedifferentpatternsofphasicdopaminesignaling

throughoutthestriatum.Eur.J.Neurosci.34,1997–2006

73. Saddoris,M.P. et al. (2015) Differential dopaminerelease

dynamicsinthenucleusaccumbenscoreandshellreveal

complementarysignalsforerrorpredictionandincentive

moti-vation.J.Neurosci.35,11572–11582

74. Howe,M.W.andDombeck,D.A.(2016)Rapidsignallingin

distinctdopaminergicaxonsduringlocomotionandreward.

Nature535,505–510

75. Hull,C.L.(1943)PrinciplesofBehavior,Appleton-Century

76. Bautista,L.M.etal.(2001)Towalkortofly?Howbirdschoose

amongforagingmodes.Proc.Natl.Acad.Sci.U.S.A.98,

1089–1094

77. Stephens,D.W.andKrebs,J.R.(1986)ForagingTheory,

Prince-tonUniversityPress

78. Salamone,J.D.etal.(1994)Nucleusaccumbensdopamine

releaseincreasesduringinstrumentalleverpressingforfood

butnotfreefoodconsumption.Pharmacol.Biochem.Behav.

49,25–31

79. Walton,M.E.etal.(2006)Weighingupthebenefitsofwork:

behavioralandneuralanalysesofeffort-relateddecisionmaking.

NeuralNetw.19,1302–1314

80. Cocker,P.J.etal.(2012)Sensitivitytocognitiveeffortmediates

psychostimulanteffectsonanovelrodentcost/benefit

decision-makingtask.Neuropsychopharmacology37,1825–1837

81. Kool,W.etal.(2010)Decisionmakingandtheavoidanceof

cognitivedemand.J.Exp.Psychol.Gen.139,665–682

82. Schmidt,L.etal.(2012)Neuralmechanismsunderlying

moti-vationofmentalversusphysicaleffort.PLoSBiol.10,e1001266

83. Iodice,P.etal.(2017)Fatiguemodulatesdopamineavailability

andpromotesflexiblechoicereversalsduringdecisionmaking.

Sci.Rep.7,535

84. Taylor,C.R.etal.(1970)Scalingofenergeticcostofrunningto

bodysizeinmammals.Am.J.Physiol.219,1104–1107

85. Floresco,S.B.etal.(2008)Dopaminergicandglutamatergic

regulationofeffort-anddelay-baseddecisionmaking.

Neuro-psychopharmacology33,1966–1979

86. Mingote,S.etal.(2005)Ratioandtimerequirementsonoperant

schedules:effort-relatedeffectsofnucleusaccumbens

dopa-minedepletions.Eur.J.Neurosci.21,1749–1757

87. Bouret,S.andRichmond,B.J.(2009)Relationoflocus

coeru-leusneuronsinmonkeystoPavlovianandoperantbehaviors.J.

Neurophysiol.101,898–911

88. Jahn,C.I.etal.(2018)Dualcontributionsofnoradrenalineto

behavioural flexibility and motivation. Psychopharmacology

(Berl.)235,2687–2702

89. Ishiwari,K.etal.(2004)Accumbensdopamineandthe

regula-tionofeffortinfood-seekingbehavior: modulationof work

outputbydifferentratioorforcerequirements.Behav.Brain

Res.151,83–91

90. Rangel,A.etal.(2008)Aframeworkforstudyingthe

neurobiol-ogyofvalue-baseddecisionmaking.Nat.Rev.Neurosci.9,

545–556

91. Milton,K.andMay,M.L.(1976)Bodyweight,dietandhome

rangeareainprimates.Nature259,459–462

92. Stearns,S.C.(1992)TheEvolutionofLifeHistories,Oxford

UniversityPress

93. Rosetta,L.etal.(2011)Energeticsduringreproduction:a

dou-blylabeledwaterstudyoflactatingbaboons.Am.J.Phys.

Anthropol.144,661–668

94. Karasov,W.H.(1985)Nutrientconstraintsinthefeedingecology

of anomnivorein aseasonal environment.Oecologia66,

280–290

95. vanMarkenLichtenbelt,W.D.(1993)Optimalforagingofa

herbivorouslizard,thegreeniguanainaseasonalenvironment.

Oecologia95,246–256

96. Hanya,G.(2004)Seasonalvariationsintheactivitybudgetof

JapanesemacaquesintheconiferousforestofYakushima:

effectsoffoodandtemperature.Am.J.Primatol.63,165–177

97. DeCasien,A.R.etal.(2017)Primatebrainsizeispredictedby

dietbutnotsociality.Nat.Ecol.Evol.1,112

98. Stevens,J.R.etal.(2005)Theecologyandevolutionofpatience

intwoNewWorldmonkeys.Biol.Lett.1,223–226

99. Stevens,J.R.etal.(2005)Willtravelforfood:spatialdiscounting

intwonewworldmonkeys.Curr.Biol.15,1855–1860

100.Evans,T.A.etal.(2014)Workingandwaitingforbetterrewards:

self-controlintwomonkeyspecies(CebusapellaandMacaca

mulatta).Behav.Process.103,236–242

101.Roeper,J.(2013)Dissectingthediversityofmidbraindopamine

neurons.TrendsNeurosci.36,336–342

102.Poulin,J.F.etal.(2018)Mappingprojectionsofmolecularly

defineddopamineneuronsubtypesusingintersectionalgenetic

approaches.Nat.Neurosci.21,1260–1271

103.Haber,S.N.etal.(1995)Subsetsofmidbraindopaminergic

neuronsinmonkeysaredistinguishedbydifferentlevelsof

mRNAfor thedopaminetransporter:comparisonwiththe

mRNAfortheD2receptor,tyrosinehydroxylaseandcalbindin

immunoreactivity.J.Comp.Neurol.362,400–410

104.Ikemoto,S.(2007)Dopaminerewardcircuitry:twoprojection

systemsfromtheventralmidbraintothenucleusaccumbens–

olfactorytuberclecomplex.BrainRes.Rev.56,27–78

105.Haber,S.N.etal.(2000)Striatonigrostriatalpathwaysin

pri-matesformanascendingspiralfromtheshelltothedorsolateral

striatum.J.Neurosci.20,2369–2382

106.Lammel,S.etal.(2011)Projection-specificmodulationof

dopa-mineneuronsynapsesbyaversiveandrewardingstimuli.

Neu-ron70,855–862

107.Fiorillo, C.D. et al. (2013) Diversity and homogeneity in

responsesofmidbraindopamineneurons.J.Neurosci.33,

4693–4709

108.Badrinarayan,A.etal.(2012)Aversivestimulidifferentially

mod-ulatereal-timedopaminetransmission dynamicswithinthe

nucleus accumbens core and shell. J. Neurosci. 32,

15779–15790

109.Saunders,B.T.etal.(2018)DopamineneuronscreatePavlovian

conditionedstimuliwithcircuit-definedmotivationalproperties.

Nat.Neurosci.21,1072–1083

110.Parker,N.F. etal.(2016)Rewardandchoiceencodingin

terminalsofmidbraindopamineneuronsdependsonstriatal

target.Nat.Neurosci.19,845–854

111.Ilango,A.etal.(2014)Similarrolesofsubstantianigraand

ventraltegmentaldopamineneuronsinrewardandaversion.