Crop model improvement reduces the uncertainty of the response to temperature of multi-model ensembles

(1)

Contents lists available atScienceDirect

Field

Crops

Research

j o u r n a l h o m e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / f c r

Crop

model

improvement

reduces

the

uncertainty

of

the

response

to

temperature

of

multi-model

ensembles

Andrea

Maiorano

a,∗

_,

_Pierre

_Martre

a,∗

_,

_Senthold

_Asseng

b

_,

_Frank

_Ewert

c,1

_,

Christoph

Müller

d

_,

_Reimund

_P.

_Rötter

e,2

_,

_Alex

_C.

_Ruane

f

_,

_Mikhail

_A.

_Semenov

g

_,

Daniel

Wallach

h

_,

_Enli

_Wang

i

_,

_Phillip

_D.

_Alderman

j,5,6

_,

_Belay

_T.

_Kassie

b

_,

Christian

Biernath

k

_,

_Bruno

_Basso

l

_,

_Davide

_Cammarano

b,3

_,

_Andrew

_J.

_Challinor

m,n

_,

Jordi

Doltra

o

_,

_Benjamin

_Dumont

l

_,

_Ehsan

_Eyshi

_Rezaei

c,y

_,

_Sebastian

_Gayler

p

_,

Kurt

Christian

Kersebaum

q

_,

_Bruce

_A.

_Kimball

r

_,

_Ann-Kristin

_Koehler

m

_,

_Bing

_Liu

s

_,

Garry

J. O’Leary

t

_,

_Jørgen

_E.

_Olesen

u

_,

_Michael

_J.

_Ottman

v

_,

_Eckart

_Priesack

k

_,

Matthew

Reynolds

j

_,

_Pierre

_{Stratonovitch}

g

_,

_Thilo

_Streck

w

_,

_Peter

_J.

_Thorburn

x

_,

Katharina

Waha

d,4

_,

_Gerard

_W.

_Wall

r

_,

_Jeffrey

_W.

_White

r

_,

_Zhigan

_Zhao

i,z

_,

_Yan

_Zhu

s

a_UMR_LEPSE,_INRA,_Montpellier_SupAgro,₂_Place_Viala,₃₄₀₆₀_Montpellier,_France

b_Agricultural_and_Biological_Engineering_Department,_University_of_Florida,_Gainesville,_FL-32611,_USA c_Institute_of_Crop_Science_and_Resource_{Conservation,}_University_of_Bonn,_D-53₁₁₅_Bonn,_Germany d_Potsdam_Institute_for_Climate_Impact_Research,_D-14₄₇₃_Potsdam,_Germany

e_Natural_Resources_Institute_Finland_(Luke),_FI-01301_Vantaa,_Finland f_NASA_Goddard_Institute_for_Space_Studies,_New_York,_NY-10025,_USA

g_{Computational}_and_Systems_Biology_Department,_Rothamsted_Research,_Harpenden,_Herts_AL5_2JQ,_UK h_INRA,_UMR₁₂₄₈_{Agrosystèmes}_et_{développement}_territorial,_F-31_326,_{Castanet-Tolosan,}_France i_CSIRO_Agriculture,_Black_Mountain,_ACT_2601,_Australia

j_CIMMYT_Int._AP_6-641,_D.F._Mexico_06600,_Mexico

k_Institute_of_Biochemical_Plant_Pathology,_Helmholtz_Zentrum_{München,German}_Research_Center_for_{Environmental}_Health,_D-85764_Neuherberg,_Germany l_Department_of_Geological_Sciences_and_W.K._Kellogg_Biological_Station,_Michigan_State_University,_East_Lansing,_MI-48_823,_USA

m_Institute_for_Climate_and_Atmospheric_Science,_School_of_Earth_and_Environment,_University_of_Leeds,_Leeds_LS2_9JT,_UK

n_CGIAR-ESSP_Program_on_Climate_Change,_Agriculture_and_Food_Security,_{International}_Centre_for_Tropical_Agriculture,_A.A._6713,_Cali,_Colombia o_Cantabrian_Agricultural_Research_and_Training_Centre,₃₉₆₀₀_Muriedas,_Spain

p_Institute_of_Soil_Science_and_Land_Evaluation,_University_of_Hohenheim,_D-70₅₉₉_Stuttgart,_Germany

q_Institute_of_Landscape_Systems_Analysis,_Leibniz_Centre_for_Agricultural_Landscape_Research,_D15₃₇₄_Müncheberg,_Germany r_USDA,_Agricultural_Research_Service,_US_Arid-Land_Agricultural_Research_Center,_Maricopa,_AZ_85138,_USA

s_College_of_Agriculture,_Nanjing_Agricultural_University,_Nanjing,_Jiangsu_210095,_China

t_Grains_Innovation_Park,_Department_of_Economic_Development_Jobs,_Transport_and_Resources,_Horsham_3400,_Australia u_Department_of_Agroecology,_Aarhus_University,₈₈₃₀_Tjele,_Denmark

v_The_School_of_Plant_Sciences,_University_of_Arizona,_Tucson,_AZ_85721,_USA

w_Institute_of_Soil_Science_and_Land_Evaluation,_University_of_Hohenheim,_D-70₅₉₉_Stuttgart,_Germany x_CSIRO_Agriculture,₃₀₆_Carmody_Road,_St_Lucia_Queensland_4067,_Australia

y_Center_for_Development_Research_(ZEF),_{Walter-Flex-Straße}_3,₅₃₁₁₃_Bonn,_Germany z_China_Agricultural_University,_Beijing_100193,_China

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received19November2015 Receivedinrevisedform4May2016 Accepted5May2016

Availableonlinexxx

a

b

s

t

r

a

c

t

Toimproveclimatechangeimpactestimatesandtoquantifytheiruncertainty,multi-modelensembles (MMEs)havebeensuggested.Modelimprovementscanimprovetheaccuracyofsimulationsandreduce theuncertaintyofclimatechangeimpactassessments.Furthermore,theycanreducethenumberof modelsneededinaMME.Herein,15wheatgrowthmodelsofalargerMMEwereimprovedthrough

∗ Correspondingauthorsat:UMRLEPSE,INRA,MontpellierSupAgro,2PlaceViala,34060Montpellier,France. E-mailaddresses:maiorano.andrea@gmail.com(A.Maiorano),pierre.martre@supagro.inra.fr(P.Martre). 1 _Present_address:_Leibniz_Centre_for_Agricultural₃₆_Landscape_Research_(ZALF),_D15374_Müncheberg,_Germany.

2 _Present_address:_Department_of_Crop_Sciences,_Division_Crop_Production_Systems_in_the_Tropics,_{Georg-August-Universität}_Göttingen,_D-37077_Göttingen,_Germany 3 _Present_address:_James_Hutton_Institute,_Invergowrie,_Dundee,_Scotland,_DD2_5DA,_UK.

4 _Present_address:_CSIRO_Agriculture,₃₀₆_Carmody_Rd,₄₀₆₇_St_Lucia,_Australia.

5 _Present_address:_Department_of_Plant_and_Soil_Sciences,_Oklahoma_State_University,_Stillwater,_OK,_74078-6028,_USA. 6 _Starting_from_P.D.A._the_author_list_is_in_alphabetical_order.

http://dx.doi.org/10.1016/j.fcr.2016.05.001

(2)

Keywords: Impactuncertainty Hightemperature Modelimprovement Multi-modelensemble Wheatcropmodel

re-parameterizationand/orincorporatingormodifyingheatstresseffectsonphenology,leafgrowthand senescence,biomassgrowth,andgrainnumberandsizeusingdetailedfieldexperimentaldatafrom theUSDAHotSerialCerealexperiment(calibrationdataset).Simulationresultsfrombeforeandafter modelimprovementwerethenevaluatedwithindependentfieldexperimentsfromaCIMMYT world-widefieldtrialnetwork(evaluationdataset).Modelimprovementsdecreasedthevariation(10thto 90thmodelensemblepercentilerange)ofgrainyieldssimulatedbytheMMEonaverageby39%inthe calibrationdatasetandby26%intheindependentevaluationdatasetforcropsgrowninmeanseasonal temperatures>24◦_C._MME_mean_squared_error_in_simulating_grain_yield_decreased_by_37%._A_reduction

inMMEuncertaintyrangeby27%increasedMMEpredictionskillsby47%.Resultssuggestthatthemean levelofvariationobservedinﬁeldexperimentsandusedasabenchmarkcanbereachedwithhalfthe numberofmodelsintheMME.Improvingcropmodelsisthereforeimportanttoincreasethecertaintyof model-basedimpactassessmentsandallowmorepractical,i.e.smallerMMEstobeusedeffectively.

1. Introduction

Wheatis themostwidelygrowncropintheworldand pro-videsmorethan20%ofthedailyproteinandfoodcaloriesforthe worldpopulation(Shiferawetal.,2013).Withapredictedworld populationof9 billionin 2050,the demandfor foodincluding wheatisexpectedtoincreasebythen(AlexandratosandBruinsma, 2012).Climatetrendsaresigniﬁcantlyaffectingagricultural pro-ductionsystems,includingwheat,inseveralregionsoftheworld, therebyposingriskstoglobalfoodsupplyandsecurity(Sundström etal.,2014).Therefore,quantifyingthepotentialimpactofclimate variabilityoncropshasbecomeapriorityinordertodevelop effec-tiveadaptationandmitigationstrategies(BurtonandLim,2005; Dentonetal.,2014).

Process-basedcropsimulationmodelsareusefultoolstoassess theimpactofclimateastheyconsidertheinteractionbetween climatevariablesandcropmanagementandtheireffectsoncrop productivity.Theiruseinclimateimpactstudiesandforanalyzing anddevelopingadaptationandmitigationstrategieshasincreased duringtherecentyears(Byjeshetal.,2010;Donatellietal.,2012; Moradietal.,2013;Rosenzweigetal.,2014).Nevertheless,most of thecurrent crop modelslack explicitdeﬁnitions of relevant physiologicalthresholdsandcropresponsestoextremeweather events,particularlyfortemperaturesexceedingthesethresholds (Rötteretal.,2011).Theseomissionsmightbeoneofthereasonfor theconsiderabledifferencesinestimatesofgrainyieldobserved amongmodelsespeciallyforhightemperatures,andbetween mod-elsandﬁeldobservations(Palosuoetal.,2011).Inaddition,sincea clearmethodologyislacking,mostclimatechangeimpact assess-mentsforagriculturehavenotaddressedcropmodeluncertainties (Müller,2011),whichhavebecomeamajorconcernrecentlyin cli-mateimpactassessments(Lobelletal.,2006;Ruaneetal.,2013; Zhangetal.,2015).

Whiteetal.(2011)reportedthatover40wheatcropmodels areinuseworldwide.Theydifferintheprocessestheyinclude, or in the modelling approaches used to simulatephysiological processes.Arecentworkcarried outbytheWheatTeamofthe Agricultural Model Inter-comparison and Improvement Project (AgMIP)(Rosenzweigetal.,2013)compared27wheatcrop mod-elsandshowedthatagreaterportionoftheuncertaintyinclimate changeimpactprojectionswasduetovariationsamongcrop mod-elsthantovariationsamongclimatemodels,andthatuncertainties in simulated yield increaseddramatically under high tempera-tureconditions (Asseng et al.,2013).Followingtheexample of theclimatemodellingcommunity,toincreasereliabilityofimpact estimatesandtogivebetterestimatesofuncertainty,useofcrop multi-modelensembles(MME)hasbeensuggested(Assengetal., 2015;Bassuetal.,2014;Lietal.,2015;Pirttiojaetal.,2015).Model improvementshavebeensuggestedforimprovingtheaccuracyof

simulationsandreducingtheuncertaintyofclimateimpact assess-ments(Assenget al., 2013;Challinor et al., 2014;Rötteret al., 2011).Martreetal.(2015)arguedthatoneoftheconsequencesof modelimprovementswillbethereductionofthenumberofmodels requiredforanacceptablelevelofsimulationuncertainty. Further-more,theimprovementofthemodelsinanensembleusinggood qualityﬁeld-basedexperimentaldatacouldsubstantiallywiden therangeofresearchquestionstobeaddressedandincreasethe conﬁdenceinsimulationresultsofapplicationsunderchanged cli-maticormanagementconditions(Martreetal.,2015).

Herein,weinvestigatedtheeffectsofmodelimprovementsin 15wheatcropmodelswithregardstoheatstressanditsimpacton modelperformances,uncertainty,andthenumberofcropmodels requiredinmulti-modelensemblesusedforimpactstudies.

2. Materialsandmethods

2.1. Experimentaldata

Detailedquality-assesseddatafromtheUSDA‘HotSerialCereal’ (HSC)experiment(Grantetal.,2011;Kimballetal.,2015;Ottman etal.,2012;Walletal.,2011)andfromthe‘InternationalHeatStress GenotypeExperiment’(IHSGE)coordinatedbyCIMMYT(Reynolds etal.,1994b)wereused.Bothexperimentswerewellwateredand fertilizedtoavoiddroughtandnutritionalstresstoassurethat tem-peraturewouldbethemainenvironmentalvariable.Dailyglobal solarradiation,maximum andminimum airtemperature, aver-agewindspeed, dewpoint temperatureandprecipitationwere recorded at weather stations near the experimental plots. The meandailyaverageairtemperatureforthegrowingseason (sow-ingtophysiologicalmaturity)wascalculatedfromminimumand maximum daily air temperatures as described in Asseng et al. (2015) and reportedin Supplementary InformationS2. In both experimentsphenologicaldevelopmentmeasurementsincluded: emergencedate(Zadockscale10),anthesisdate(Zadockscale65), andmaturitydate(Zadock scale89).Fromthesemeasurements thenumberofdaysfromsowingtoanthesis(days),fromanthesis tomaturity(days),andfromsowingtomaturity(days)were calcu-lated.Inbothexperiments,theplotswerekeptweed-free,andplant protectionmethodswereusedasnecessarytominimizedamage frompestanddiseases.Thetwodatasetsarefurtherdescribedin Assengetal.(2015).Followingisabriefdescriptionwithfocuson themeasurementdatathatwereavailableforthisstudy.

The HSC experiment was conducted at Maricopa, AZ, USA (33.07◦N,111.97◦W,361ma.s.l.):Thespringwheatcultivar‘Yecora RojO’wassownabouteverysixweeksfortwoyears,andinfrared heatersweredeployedonsixofthesowingdatesinaT-FACE (tem-peraturefree-aircontrolledenhancement)systemwhichwarmed thecanopiesoftheheatedplotsonaverageby1.3◦Cand2.7◦C

(3)

during theday and thenight, respectively (targets were 1.5◦C and 3.0◦C; modeswere 1.4◦C and 3.0◦C; Kimball etal., 2015). YecoraRojoisofshortstature,requireslittletonovernalization, isnotorlittlephotoperiodsensitive,andmaturesearly(Qualset etal.,1985).In-seasonmeasurementsincludedleafareaindex(LAI, m2_m−2_), _total_above_ground _dry_biomass,_dry_matter_weight_of

grainpersquare meter andnitrogen contentmeasuredatmilk stageand maturity. End-of-season(i.e.ripeness-maturity) mea-surementsweretotalabovegrounddrybiomass(tDMha−1),grain yield(t DMha−1), singlegraindry mass(mg DMgrain−1), and grainnumber(grainm−2).Biomassharvestindexwascalculated asHI=100×(grainyield)/(abovegroundbiomass)(%).

DatafromtheIHSGEexperimentsusedinthisstudyincludes twospringwheatcultivars(Bacanora88andNesser)grown dur-ing the 1990–1991 and 1991–1992 winter cropping cycles at hot,irrigated,andlowlatitudesitesinMexico(CiudadObregon, 27.34◦N,109.92◦W,38ma.s.l.;andTlatizapan,19.69◦N,99.13◦W, 940m a.s.l.), Egypt (Aswan, 24.1◦N, 32.9◦E, 200m a.s.l.), India (Dharwar, 15.49◦N, 74.98◦E, 940m a.s.l.), Sudan (Wad Medani, 14.40◦N, 33.49◦E, 411m a.s.l.), Bangladesh (Dinajpur, 25.65◦N, 88.68◦E,29ma.s.l.),andBrazil(Londrina,23.34◦S,51.16◦W,540m a.s.l.)(Reynolds, 1993;Reynoldsetal.,1994a,b).Experimentsin Mexico included normal (December) and late (March) sowing dates.Bacanora88 hasmoderatevernalizationrequirementand lowphotoperiodsensitivityandNesserhaslowtonovernalization requirementandphotoperiodsensitivity.Thesevensites(outofthe original12locations)werechosentorepresentarangeof temper-atureasdetailedinAssengetal.(2015).Bacanora88andNesser werechosen(outoftheoriginal16cultivars)fortheirlow pho-toperiodsensitivityandlowvernalizationrequirements.Variables measuredintheexperiment includedplantnumberper square meter,anthesisandﬁnalabovegroundbiomass,ﬁnalgrainyield andyieldcomponents(numberofearpersquaremeter,numberof grainperear,andsinglegraindrymass).Theseexperimentaldata werenotpubliclyavailableandcouldthereforebeusedinablind modelevaluation.

2.2. Modelinter-comparisonandimprovementprotocols

Ofthe30modelsthatparticipatedintheoriginalstudyusingthe HSCdata(Assengetal.,2015),15modelsacceptedtoparticipatein thisnewstudy.Therewasnoexplicitcriterionofinclusion,sothis wouldbean“ensembleofopportunity”asdeﬁnedintheclimate modelcommunity(TebaldiandKnutti,2007).Allofthemodels havebeendescribedinpublicationsandarecurrentlyinuse.For theevaluationdatasetmeasurements,abovegroundbiomassand grainyieldweresimulatedbyallthemodels.7outof15models didnotsimulatedsinglegraindrymassandgrainnumberbutused aharvestindexapproach.

Forbothexperiments,allmodelinggroupswereprovidedwith dailyweatherdata,cropmanagement,soil,andcultivar informa-tion.Qualitativeinformation onvernalization requirementsand daylengthresponseforeachcultivarwerealsoprovided.

TheHSCexperiment(calibrationdataset)wasusedtoimprove themodels.AllavailablemeasurementsfromtheHSCexperiment wereprovidedtomodelerstoimproveandreﬁnethe parameteri-zationandprocessesoftheirmodel.Theobjectivewastoimprove wheatmodelsforthesimulationoftheimpactofhigh tempera-tureandheatstressesoncropdevelopmentandgrowth.Modelling groupswereallowedtodecidehowtoimproveandimplementheat stressimpactintheirmodels.

TheIHSGEexperiment(evaluationdataset)wasusedas inde-pendent evaluation data set to test single models and model ensembleperformancesbeforeandafterimprovement.All mea-surementsoftheevaluationdatasetwerewithheldfrommodelers (blindtest)withtheexceptionofphenologyforalltreatmentsand

grainyieldforoneofthetreatments(oneyearatCiudadObregon, Mexico)whichwasusedtocalibrategenotypiccoefﬁcients.

Theexperimentaldatausedinthisstudywerenotpreviously usedtodeveloporcalibrateanyofthe15modelsusedinthisstudy. ExceptforthetwoExpert-Nmodelswhichwereexecutedbythe samegroup,allmodelsweresimulatedbydifferentgroupswithout communicationbetweenthegroupsregardingthe parameteriza-tionoftheinitialconditionsorcultivarspeciﬁcparameters.Inmost casesthemodeldevelopersexecutedtheirownmodels.

2.3. Evaluationofmodelimprovementeffectsonsinglemodels andonmulti-modelensembleaccuracy

Weevaluatedtheeffectofmodelimprovementontwodifferent performancecharacteristics,accuracyanduncertainty,andonthree modelentities:(i)singlemodels(accuracyonly);(ii)multi-model ensemble(MME,theensembleof15 modelsin thisexperiment exercise);and(iii)MMEmedian(e-median).

Accuracywasmeasuredusingthemeansquarederror(MSE), therootmeansquarederror(RMSE),andtherootmeansquared relativeerror(RMSRE).

Formeasuringsinglemodelerrorinreproducingthe calibra-tionandtheevaluationdatasetweconcentratedontherootmean squaredrelativeerror(RMSRE).Thiserrorindicatorhasthe advan-tageofgivingmoreequalweighttoeachmeasurement,andit’s meaningfulwhencomparingverydifferentenvironmentslikelyto giveabroadrangeofresponses(Martreetal.,2015).RMSREwas calculatedas: RMSREm=100×

1 N N

i=1

Yi− ˆYm,i Yi

2 (1)

whereRMSREmistheRMSREofmodelm,iisthesite/year/sowing

datescombinations(treatment),Nis thetotal number of treat-ments,Yiistheobservedvariablefortreatmenti, ˆYm,iisthevariable

simulatedbymodelmfortreatmenti.Sincethisindicatorisvery sensitivetoerrorswhenmeasuredvaluesaresmall,RMSEwasused asadditionalsupportinginformationforabetterunderstandingof RMSREwhenneeded.RMSEwascalculatedas:

RMSEm=

1 N N

i=1

Yi− ˆYm,i

2 (2)

where,RMSEmistheRMSEofmodelm.

Theaccuracyofthepopulationof15modelsbeforeandafter improvementwasanalyzedusingthemeansquarederror(MSE) anditstwocomponentssquaredbiasandvariance,averagedacross treatments: MSEMME= 1 N 1 M N

i=1 M

m=1

Yi− ˆYm,i

2 = _N1 N

i=1 varM( ˆYm,i)+ 1 N N

i=1

biasM( ˆYm,i,Yi)

2 (3)

where,MSEpmistheMSEofthepopulationofmodelsinthe

ensem-ble,Nisthenumberoftreatments,Misthetotalnumberofmodels intheensemble(i.e.15),varmandbiasMarethevarianceandthe

biasforthemodelpopulation,respectively.FromEq.(3)itis evi-dentthatwhilebiasisbasedonbothobservationsandsimulations, varianceonlytakesintoaccountsimulatedvalues.

(4)

2.4. EvaluationofmodelimprovementeffectsonMMEprediction uncertainty

To assess the MME prediction uncertainty we considered both thevariability inMME and thecomparison withhindcast (i.e.retrospectiveforecastsusingknowninputsandknownﬁeld measurements) (Wallach et al., 2015) using the two available measurementdatasets.Inordertoevaluatetheprediction uncer-taintyof the MME beforeand after improvement we usedthe HSCcalibration-datasettosimulatemodelhindcastinrespectto observeddata,andtheIHSGEexperimentasthe“unknown”data setusedtosimulatemodelprediction tounknowndata andto evaluatethepredictiveskillsofthemodelsin theensemble.As ameasureofuncertaintyweusedthemeansquarederrorof pre-diction(MSEP)anditsdecompositioninpredictionsquaredbias (bias2

prediction)andpredictionvariance(varprediction).Accordingto

Wallachetal.(2016)theaveragesquarederroracrosstreatments of MME-mean calculated using the known data set (hindcast)

(MSEhindcaste-mean)canbeusedasareferenceestimateofthemodel

pop-ulationsquaredbiaswhencalculatingpredictionestimates.This correspondstotheaveragesquaredbiasofhindcastsascalculated inEq.(4):

bias2_prediction=MSEhindcast_e-mean

= 1 Nhindcast Nhindcast

i=1

Yihindcast− 1 M M

m=1 ˆ Y_m,ihindcast

2 =bias2_hindcast (4)

where,Nhindcastisthenumberoftreatmentsintheknowndataset,

Yihindcastistheobservedvariablefortreatmentioftheknowndata

set, ˆYhindcast

m,i isthehindcastofthesimulatedvariablefortreatment

ibythemodelm.Thepredictionvariancevarpredictionisthe

vari-anceofthevaluessimulatedbythepopulationofmodelsforthe unknowndatasetaveragedacrosstreatments:

varprediction= 1 Nprediction Nprediction

i=1 varM( ˆY_iprediction) (5)

where,Npredictionisthenumberoftreatmentsintheunknowndata

set,Yipredictionisthesimulatedvariableforthetreatmentiofthe

unknowndataset.ThereforeanestimateofMSEPcanbecomposed as:

MSEP=bias2

prediction+varprediction (6)

2.5. EvaluationofmodelimprovementeffectsonMME-median

FollowingAssengetal.(2015)andMartreetal.(2015),weused themedianofthemodelsimulations(e-median)astheestimator oftheensemblemodelsimulations.Inordertoevaluatethe over-alle-medianaccuracywecalculatedthesamecriteriaasforthe individualmodels,namelyRMSRE(Eq(1)).

Toexplorehowthee-mediananditserror(RMSRE)variedwith thenumberofmodelsandwiththerandomselectionofmodelsin theensemble,weperformedabootstrapcalculation(i.e.random samplingwithreplacement)foreachvalueofM’(numberof mod-elsintheensemble)from1to15.For eachensembleofsizeM’ wedrew20×103_bootstrap_samples_{(substantially}_higher_than_the

3200samplesfoundbyMartreetal.(2015)asasufﬁcient num-berofsamplesfor27models)ofM’modelswithreplacement,so thesamemodelmightberepresentedmorethanonceina sam-ple.Thevariationofe-medianacrossthebootstrapsamplesdue

torandommodelselectionwasestimatedwiththecoefﬁcientof variation(CV): CV(ˆy_e-median,M)₌ 1 N N i=1

100_× sdB(ˆy M e−median,i) meanB(ˆyM e−median,i)

(7)

where,CV(ˆy_e-median,M)istheestimateofthecoefﬁcientof varia-tionofe-medianforthemodelensembleofsizeM’,sdB(ˆyM

e-median,i)

andmeanB arethestandarddeviationandthemeanofB

(num-berofbootstrapsamples)e-mediansofmodelensemblesofsizeM’ fortheithtreatment.AbenchmarkCVof13.5%,previously estab-lishedthroughameta-analysisofﬁeldtrials(Tayloretal.,1999)was usedtoevaluatetheminimumnumberofmodelsrequiredwithin aMME.

TheﬁnalestimateofRMSREfore-medianwascalculatedas:

RMSREM= 1 B B

b=1 100×

1 N N

i=1

yi− ˆybe−median,i yi

2 (8)

where,RMSREMistheRMSREofe-medianofthemodelensemble

ofM’size, ˆyb

e−median,iisthee-medianestimateinbootstrapsample

boftheithtreatment.

AllcalculationsandgraphsweremadeusingtheRstatistical softwareR3.1.3(RCoreTeam,2013)andthedevelopment envi-ronmentRStudio(RStudioTeam,2015).Bootstrapsamplingused theRfunctionsample.

3. Results

3.1. Individualmodelimprovements

The major draw backs in simulating the HSC experiment were related to the impact of the higher temperature range (Tmean>22◦C) on yield, biomass and phenology (Asseng et al.,

2015).Furthermoreitwasshownthatthefewmodelsthatalready includedheat stressroutinesaffectingcanopy senescencewere theonly onesabletoreproducethe impactofvery highmean seasonaltemperatures (Tmean≥29◦C)on grainyield and above

groundbiomass.Therefore,theprocessthatreceivedmost atten-tion was leaf senescence, followed by heat stress effects on processesrelatedtobiomassgrowthand/orphenological develop-ment,grainnumberand/orsize,leafdevelopment(Table1,Fig.1). Basedonexperimentalevidences(e.g.ParentandTardieu,2012; Porterand Gawith,1999), inseveralmodelslineartemperature responseswerereplacedbynon-linear(APSIM-Eand SiriusQual-ity)ortrapezoidal(APSIM-Wheat,GLAM-Wheat,Expert-N-SPASS, Expert-N-SUCROSS)responsefunctions.Thecardinaltemperatures forthese processeswerefixedusing valuesreportedin the lit-eratureor calibratedusing theHSC experimentaldataset.One model(APSIM-Nwheat)addedacanopytemperaturesub-routine. In additiontotheinclusion/modification of heat stressimpacts onphysiologicalprocesses, fivemodelsimprovedprocessesnot directlyrelatedtoheatstressusingtheHSCdatasetorother pub-lisheddatasets(Table1).Onemodel(GLAM-wheat)removedthe sub-routineforheatstresseffectongrainsetandpotentialharvest indexastheyobservednosubstantialimprovementanddecided nottoincreasethecomplexityoftheirmodel(Table1and Supple-mentaryMethods).

Inthecaseofheatstressimpactsonleafsenescence,asimilar approach,basedonAssengetal.(2011),wasadoptedinallmodels (Table1andSupplementaryMethods).Afactorforacceleratingleaf senescenceiscalculatedasalinearfunctionofairorcanopy tem-perature(dailymaximum,averageortri-hourlyaccordingtothe differentmodelimplementations)aboveathresholdtemperature value.Somemodelsincludedaplateautothesenescencefactor.

(5)

Table1

Outlineofindividualmodelimprovement.MoredetailsaregivenintheSupplementaryData.

Modelcode Modelname Reference Descriptionofmodelimprovements Introductionand/ormodiﬁcationof processrepresentation

Calibration AE APSIM-E (Chenetal.,2010;Keating

etal.,2003;Wangetal.,

2002;Zhaoetal.,2015)

Introductionofanonlinear temperatureresponsefunctionfor phenologicaldevelopmentand biomassgrowth.

Calibrationof14parametersrelatedto themodifiedtemperatureresponse functionsandtoradiationuse efficiencyandmaximumspecificleaf area.

AW APSIM-Wheat (Keatingetal.,2003) Modiﬁcationofthetemperature responsefunctionforthermaltime accumulationfromatriangulartoa trapezoidalfunction.

Calibrationofnineparametersrelated tothemodiﬁedtemperatureresponse functionforthermaltime

accumulation,canopysenescence, grainnumber,andgrainﬁllingrate. Modiﬁcationofheatstresseffecton

leafsenescencetoremove discontinuityaroundthethreshold temperature.

AN APSIM-Nwheat (Assengetal.,2004,1998;

Keatingetal.,2003)

Introductionofanempiricalmodelof canopytemperatureasafunctionof evapotranspirationanddailymeanair VPD(describedinWebberetal.,2015).

Calibrationofsevenparameters relatedtothenewcanopytemperature modelandthemodiﬁedleaf

senescenceheatstressresponse. Modiﬁcationofheatstresseffecton

leafsenescencetoremove discontinuityaroundthethreshold temperature.

FA FASSET (Berntsenetal.,2003;

Olesenetal.,2002)

Introductionofaheatstresseffecton leafsenescence.

Calibrationofsevenparametersrelated tothenewleafsenescenceresponse andtoLAI,DMallocationtoroots,N concentrationinstorageorgans. GL GLAM-Wheat (Challinoretal.,2004;Li

etal.,2010)

Introductionofatrapezoidal temperatureresponsefunctionforleaf growth.

Calibrationof26parametersrelated themodiﬁedornewtemperature responsefunctionsandtoLAI,HI, maximumpotentialleafgrowthand transpiration,transpirationefﬁciency, andVPDcalculation.

Modiﬁcationofthetemperature responsefunctionforphotosynthesis andtranspirationefﬁciencyfroma bi-linearfunctionwithnoreduction towardsthebasetemperaturetoa trapezoidalfunction.

Modiﬁcationofthetemperature responseofphenologicaldevelopment fromatrapezoidaltoatriangular function.

Modiﬁcationofthemagnitudeofthe responseofcanopysenescencetohigh temperature.

Removedheatstresseffectaround anthesisongrainsetandpotential harvestindex.

Modificationofthedefinitionof anthesis(frombeginningofflowering tomid-flowering).

HE HERMESS (Kersebaum,2007;

Kersebaumetal.,2011)

Correctionofanerrorinthecalculation ofthermaltimeaccumulation.

Calibrationofthermaltimefor phenologicaldevelopmentandofﬁve parametersrelatedtothecorrectionof thermaltimeaccumulation. Constantgrain-to-chaffdrymassratio

atmaturityreplacedbyafunction basedonthedurationofthe ﬂowering-to-maturityperiod. Ndilutioncurvesformaximumand criticalNconcentrationwereﬁxedtoa constantthermaltimefromemergence tomaturity,nowitisscaledtothe varietalthermaltimefromemergence tomaturity.

Simulationofsoilmoistureand mineralNstartsatthebeginningofthe yearforequilibrationbasedongiven weatherconditions.

LP LPJmL (Beringeretal.,2011;

Bondeauetal.,2007;Fader

etal.,2010;Gertenetal.,

2004;Mülleretal.,2007;

Rostetal.,2008)

Introductionofaheatstresseffecton leafsenescence.

Calibrationofﬁveparametersrelated tophenologicaldevelopment,the sensitivitytophotoperiodandLAI.

(6)

Table1(Continued)

Modelcode Modelname Reference Descriptionofmodelimprovements Introductionand/ormodiﬁcationof processrepresentation

Calibration

NP Expert-N-SPASS (Biernathetal.,2011;

Priesacketal.,2006;Wang

andEngel,2000)

Introductionofafunctiontocalculate hourlytemperature.

Calibrationofthreeparametersrelated toradiationuseefficiency,specificleaf drymassandgrainnumber. Modificationofthetemperature

responsefunctionsforphotosynthesis fromatriangulartoatrapezoidal function.

NS Expert-N-SUCROSS (Biernathetal.,2011;

Priesacketal.,2006)

Introductionofafunctiontocalculate hourlytemperature.

Calibrationofthreeparametersrelated toradiationuseefficiency,specificleaf drymassandgrainnumber. Modificationofthetemperature

responsefunctionsforphotosynthesis fromatriangulartoatrapezoidal function.

OL OLEARY (O’LearyandConnor,

1996a;O’LearyandConnor,

1996b;O’Learyetal.,1985)

Modiﬁcationofthetemperature responsefunctionsforphenological developmentandstemdevelopment fromalineartoatriangularorbi-linear withamaximumfunction.

Modiﬁcationoftheroutinesimulating transferofNtograinsfromgenericto cultivarspeciﬁc.

Introductionofadry-sowing emergencesubroutine.

Introductionofaneffectofelevation onthepsychometricconstantand radiationuseefﬁciency. SA SALUS (Bassoetal.,2010;

Senthilkumaretal.,2009)

Calibrationof35parametersrelatedto phyllochron,vernalization

requirement,sensitivityto photoperiod,LAI,cardinal temperaturesofthetemperature responsefunctionforradiationuse efﬁciency,leafexpansion,rootgrowth, grainﬁlling,grainnumber,grainN concentrationandDMpartitioning. SP SIMPLACE

<LINTUL2-CC-HEAT>

(Anguloetal.,2013) Introductionofaheatstresseffecton

leafsenescence.

Calibrationoffourparametersrelated toradiationuseefﬁciency,LAI,and criticalheatstressresponse. Reductionofyieldduetoheatstress

calculatedusingTmeaninsteadofTmax. Introductionofasub-routinefor post-anthesisbiomassre-translocation tograins.

S2 Sirius2010 (JamiesonandSemenov,

2000;Jamiesonetal.,

1998;Lawlessetal.,2005;

Stratonovitchand

Semenov,2015)

Introductionofaheatstresseffecton leafmaturationandsenescence.

Calibrationofsixparametersrelatedto thenewheatstressandfrost responses.

Introductionofaheatstressandfrost effectsongrainnumber

Introductionofaheatstresseffecton potentialgraindrymass.

SQ SiriusQuality (Ferriseetal.,2010;He

etal.,2012;Martreetal.,

2006)

Introductionofaheatstresseffecton leafmaturationandsenescence.

Calibrationof13parametersrelatedto heatstresseffectonleafmaturation andsenescence,thenon-linear temperatureresponsefunctionfor developmentandleafexpansnion, daylengthsensitivity,and vernalizationrequirement. Modiﬁcationofthetemperature

responsefunctionsforphenological developmentandleafexpansionfrom alineartoanon-linearfunction. WG WheatGrow (CaoandMoss,1997;Cao

etal.,2002;Huetal.,2004;

Lietal.,2002;Panetal.,

2007,2006)

Introductionofaheatstresseffecton phenologicaldevelopment.

Calibrationoffourparametersrelated totheheatstresseffecton

phenologicaldevelopmentandgrain ﬁllingduration.

Introductionoffunctiontocalculate hourlytemperature.

Inthecaseofimprovementsrelatedtoheatstressimpacton phenologicaland/orgrowthprocesses,theimpactofheatstress was modeled by introducing a temperature response function whichincludedadecreasingphase(triangular,trapezoidal,or

non-linear)athightemperaturesand which substitutedfora linear responsefunctionwithorwithoutaplateau.Onlyinonemodel (OLEARY)alinearresponseforphenologicaldevelopmentwas sub-stitutedforalinearwithaplateauforsomephenologicalstages.In

(7)

Fig.1.Numberofmodelsthatincludedormodiﬁed(ifalreadyincluded)keyprocessesrelatedtoheatstressduringthemodelimprovementexercise.

theAPSIM-wheatmodelthetemperatureeffectonthephenological developmentwaspreviouslymodeledusingafunctionwithasingle optimumtemperature(triangularfunction)thatwasnowchanged toafunctionwitharangeofoptimumtemperatures(trapezoidal function).The crop modelsthat did not introduce sucha type of response forphenological developmentand biomassgrowth alreadyincludedthistypeofresponseforbothprocesses (APSIM-NWheat,SIMPLACE),oralreadyhadafunctionwithadecreasing phaseaboveanoptimumtemperatureforbiomassgrowthandkept alineartemperatureresponsefunctionforphenological develop-ment(HERMESS,LPJmL,Sirius2010),orkeptalinearapproachfor bothprocesses(FASSET).

3.2. Effectsofmodelimprovementonsinglemodelsaccuracy

Fig.2illustratestheeffectsofmodelimprovementonthe sim-ulationsofthreetreatmentsoftheHSCcalibrationdatasetwhose meangrowingseasonaltemperaturesweredifferent.Inmostcases, measuredin-seasonandend-of-seasonLAI,abovegroundbiomass, andgrainyieldwereintherangeofmodelsimulationsforboth the un-improved and the improved models. Nevertheless, the improvedmodelsshowed a lowerlevel of variation (measured throughthe10thto90thpercentilerangeofthe15model sim-ulations).Forgrainyieldandabovegroundbiomass,theimproved MMEwasmorepreciseathightemperaturesthantheunimproved MME (meangrowingseasontemperature of 22◦C and 27◦C in Fig.2).Mostunimprovedandimprovedmodelsunderestimated theimpactofhightemperatureonLAI,butthiswastruetoalower extentfortheimprovedcompared totheun-improvedmodels. Consideringthee-medianofthemodelensemble,thesimulations oftheimprovedMMEappearedsimilartotheun-improved pop-ulationat15◦Cbutmoreaccurateat22◦Cand27◦CforLAIand abovegroundbiomass,andforgrainyieldat27◦C.

Inordertoexploreifthepopulationof15modelsusedinthis studyhadskillssimilartothatofthe30modelsthathadpreviously beenusedtosimulatedthecalibrationdataset(Assengetal.,2015), wecomparedtheRMSREdistributionofthesetwopopulationsof modelsforthecalibrationdataset(Fig.3).TheRMSREdistribution foralmostallthevariableswassimilarforthe30modelsandthe 15unimprovedmodelsincludedinthisstudy.Therefore,wecould

reasonablyexcludeany“modelsampling”effectsontheresultsof thiswork.ComparisonofRMSREdistributionofthe15unimproved andimprovedmodelsforthecalibrationdatasetshoweda reduc-tioninthemedianvaluesforRMSREofmostofthevariables:53% fordaysfromsowingtomaturity,36%forabovegroundbiomass, 31%forgrainyield,18%forHI,32%forgrainnumber,12.4%for sin-glegraindrymass.However,RMSRErangeforHI,grainnumber, andsinglegraindrymassremainedalmostunchanged.

Fig.4 shows theeffect ofmodel improvement onthe accu-racy(asmeasuredbyRMSRE)ofeachmodelforgrainyieldand forthekeyvariablesleadingtoﬁnalyieldforthecalibrationdata set.In general,modelswereimprovedfor almost allmeasured variables.Asexpected,modelsthathadlargeerrorsforaspeciﬁc variableweretheonesthatimprovedthemostforthatvariable. AllmodelshadlowerRMSREforsimulatingabovegroundbiomass andgrainyieldaftermodelimprovement.Theonlyvariablesfor whichmorethanonemodelworsenedaftermodelimprovements wereLAIandHI.Five models(APSIM-Nwheat,Expert-N—SPASS, Expert-N—SUCROSS, SALUS, and SIMPLACE<LINTUL2-CC-HEAT>) increasedtheerrorforLAIafterimprovements(Fig.4).

Twoofthesemodelswereamong theonesthatincludedor modiﬁedasub-routineforheatstressimpactonleafsenescence (APSIM-NwheatandSIMPLACE<LINTUL2-CC-HEAT>).Fourmodels hadhigherRMSREofHIafterimprovement(APSIM-Wheat, GLAM-Wheat, Expert-N− SUCROSS,and SiriusQuality), although they hadlowerRMSREforbothabovegroundbiomassandgrainyield aftermodelimprovement.Forboththecalibrationandevaluation datasets,modelimprovementdecreasedthevariation(measured throughthe10thto90thmodelensemblepercentilerange)ofmost simulatedvariablesathighmeanseasonaltemperatures(Fig.5). Forthecalibrationdatasetthereductionofthevariabilitybetween modelsandtheirconvergenceisanexpectedresultasalltheteams used thesame datasettoimprove and recalibrate theirmodel. For grainyield,an increase inprecisionwasobserved for tem-perature>24◦Cforboth thecalibrationandtheevaluationdata set:grainyieldvariationdecreasedby4%and21%consideringthe wholetemperaturerangeofthecalibrationandtheevaluationdata sets,respectively,andby39%and26%consideringonlymean sea-sonaltemperatures>24◦C.Fortheevaluationdataset,consistent reductionoftherangeofvariationamongmodelswasalsoobserved

(8)

A

B

C

D

E

F

G

H

I

Fig.2. Simulatedandmeasuredwheatgrowthdynamicsforthecalibrationdataset.(A–C)leafareaindex(LAI),(D–F)totalabovegroundbiomass,and(G–I)grainyieldversus daysaftersowingformeangrowingseasontemperatures15◦_C_(A,_D,_and_G),₂₂◦_C_(B,_E,_and_H)_and₂₇◦_C_(C,_F,_and_I)._Black_dotted_lines_and_dark_grey_areas_are_e-median (MMEmedian)andthe10thto90thpercentilerangeofthe15original(unimproved)models,respectively.Solidredlinesandlightgreyareasaree-medianandthe10thto 90thpercentilerangeofthe15improvedmodels,respectively.Areasaregreywhenimprovedandunimprovedrangesoverlap.Bluesymbolsaremeasuredmean±1s.d.for n=3independentreplicates(forinterpretationofthereferencestocolourinthisﬁgurelegend,thereaderisreferredtothewebversionofthisarticle.)

forHI(20%),grainnumber(71%),andsinglegraindrymass(44%) (Fig.5).

3.3. EffectsofindividualmodelimprovementonMMEaccuracy andpredictionskills

For both the calibration and evaluation data sets, model improvementsdecreased MSE ofmodels forgrainyield (Fig.6, panelA),phenology (Fig.6,panelsBand C),andabove ground biomass(Fig.6,panelD).Thisreductionwasmainlyduetoa reduc-tioninMMEvariance.Consideringthecalibrationdataset(Fig.6, panelA),MSEofgrainyielddecreasedonaverageby29%,equally duetodecreaseinsquaredbias(−33%)andvariance(−27%). Con-sideringtheevaluationdataset,MSEofgrainyieldwasreduced by37%, due toa 49% reduction in variance, while thesquared biasdidnotincreasedsigniﬁcantly(Fig.6,panelA).MSEofabove groundbiomasswasreducedby44%duetoa54%reductionin vari-ance,whilethesquaredbiasdidnotchangesigniﬁcantly(Fig.6, panelD).Analysisofthepredictionskillsofthemodelpopulation showedthatthelevelofpredictionerror(MSEP)whensimulating the“unknown”datasetwasreducedafterimprovementby47% (Fig.6,panelA).AstheMSEPisthesumofthesquaredbiasfor thecalibrationdatasetandthevariancefortheevaluationdataset

(Eq.(6)),changesinbiasandvarianceofMSEPfollowedthesame reductionpatterns.

3.4. EffectofindividualmodelimprovementonMMEe-median skill

TheRMSREofe-medianwasreducedby38%forgrainyieldand by46%forabovegroundbiomass,inthecalibrationdataset,and by2%forgrainyieldand11%forabovegroundbiomassinthe eval-uationdataset(Fig.3).Therelationshipbetweenthenumberof modelsinanensembleandtheCVandRMSREofe-median estima-tionofgrainyieldandabovegroundbiomasswasanalyzedthrough abootstrapapproachtocreatealargenumberofrandom ensem-blesof1–15models.Independentlyofthenumberofmodelsin theensembles,fortheevaluationdatasettheCVofe-medianwas about41%lowerforimprovedmodelscomparedwithunimproved models(Fig.7,panelAandB).

Therefore,modelimprovementdecreasedvariationofe-median inarangebetween15%forM’=1and7%forM’=15foraboveground biomassandbetween14%atM’=1,and9%forM’=15for grain yield.Asaconsequence,whilewiththeunimprovedmodelsthe benchmarkCV%of13.5%(Tayloretal.,1999)wasnotachievedfor grainyieldevenwiththemaximummodelensemblesize,withthe improvedmodelsthisthresholdwasreachedwitheightmodelsin

(9)

A

B

C

D

E

F

G

H

Fig.3. Effectofmodelimprovementonrootmeansquaredrelativeerror(RMSRE)distributionfordaysfromsowingtoanthesis(A),daysfromanthesistomaturity(B),leaf areaindex(LAI)(C),harvestindex(HI)(D),grainnumber(E),singlegraindrymass(F),finaltotalabovegroundbiomass(G),finalgrainyield(H),forthecalibrationdataset. RMSREwascalculatedforthe30modelsincludedinapreviousstudy(AgMIP-Wheat)(Assengetal.,2015)andthe15unimprovedandimprovedmodelsincludedinthe modelimprovementstudy.TheleftandtherightsideoftheboxarethefirstandthirdRMSREquartiles.ThelineinsidetheboxistheRMSREsecondquartileormedianof individualmodelerrors.TheendsofthewhiskersindicatetheRMSRE10thand90thpercentilerespectively.Theemptypointsaretheoutliers.Theredcrossesindicatethe e-medianRMSRE(forinterpretationofthereferencestocolourinthisfigurelegend,thereaderisreferredtothewebversionofthisarticle.)

theensemble.Modelimprovementsreducede-medianRMSREof grainyieldinarangebetween12%atM’=1,and2%atM’=15for grainyieldfortheevaluationdataset(Fig.8).

4. Discussion

For thefirsttime, using two uniqueexperimentalfield data setswithalargerangeoftemperature,weimprovedthe predic-tiveskillsofaMMEof15wheatmodels.Asaresultweincreased MME accuracy whilereducing model ensembleuncertainty. As aconsequence,thenumberofrequiredmodelsforMMEimpact assessmentsonyieldtoachieveobservedlevelsoffield experimen-talvariationwashalved.Thisisasignificantstepforwardforcrop modellingandfutureclimateimpactstudiesasuntilnowveryfew modelshave explicitlyconsideredheatstressimpactsonwheat developmentand growth(Assenget al., 2011;Moriondoet al., 2010).

4.1. Modelimprovements

Modelimprovementsincreasedtheaccuracyofsinglemodelsin reproducingheatstressimpactonwheatcrops.Asaconsequence,

theaccuracyofthemodelsandofthee-medianinsimulatingthe impactofhightemperaturesandheatstressincreasedandthe vari-anceamongmodelsinthepopulationwasreduced.

As we focused on the effects of model improvements on a MMEof15modelsandonthepossibleconsequencesforfuture MMEimpactassessmentsstudies,wedidnotanalyzeeachmodel improvement in detail. In this exercise, the concept of “model improvement”wasimplementedasanimprovementofthe appli-cability of models across diverse environments and climates includingclimateextremes.Eachcropmodelaimedtoimprove how high temperature effects were captured by incorporating and/or improving a range of different processes using a high-quality data set. The process descriptions in the models were mostly updated usingnewinformation from theliterature,e.g. a new approach to heat stress, or they accounted of a harm-fuleffectofhightemperaturesfortheﬁrsttime.Eachteamwas leftfreetodecidehowtoimplementheatstressintheirmodel. Thischoicewasmadeconsideringthediversityof implementa-tion of key physiological processes, and/or the diversity in the levelofempiricism/mechanismintheirapproaches(see supple-mentaryinformationinAssengetal.,2015,2013).Inmostcases, beingprimarilydevelopedtosimulate“standard”climate

(10)

condi-A

B

C

D

E

F

G

H

Fig.4.Log2differenceofRMSREforimprovedandunimprovedmodelsversusRMSREofunimprovedmodelsfordaysfromsowingtoanthesis(A),daysfromanthesisto maturity(B),leafareaindex(LAI)(C),harvestindex(HI)(D),grainnumber(E),singlegraindrymass(F),ﬁnaltotalabovegroundbiomass(G),ﬁnalgrainyield(H),forthe calibrationdataset.Apositivedifferenceofthelog2RMSRE’sindicateanimprovementinmodelperformance.TheextentofmodelimprovementintermsofRMSREdoubles foreachunitoflog2RMSREdifferencebetweentheun-improvedandtheimprovedpopulationofmodels.

tions,modelshadtoimprovehowhightemperatureeffectswere capturedbyincludingormodifyingsomekeybiologicalprocesses involvedincropheatstressresponse.Allthemodelsimprovedtheir skillsinsimulatingmostofthetestedvariables.However,in sev-eralmodelsHIsimulationwasnotimprovedandinthreemodels (APSIM-Wheat,GLAM-Wheat,Expert-N-SUCROSS)itwasslightly worsened,showingthatgrainyieldandabovegroundbiomassdid notimproveproportionallytoeachother.AsobservedbyChallinor etal.(2014)thismightindicatesomelevelofcompensationerror duringthecalibrationphasedespitetheimprovementofbothyield andbiomass.Furthermore,model improvementwasfocusedon heatstress,andmostoftheimprovementwasobservedformean growingseasontemperature>24◦Cwhichisalsotherangewhere mostofthedisagreementwasobservedbeforeimprovement.

Sevenmodelsincludedasub-routineforsimulatingthe accel-erationof leafsenescenceabove atemperature threshold.Heat stresswasreportedtoenhanceleafsenescencewithaconsequent reductionin thetotal amountofinterceptedlight, reduction of theaccumulationofassimilates,andshorteningofthegrain ﬁll-ingperiod (Chauhan et al., 2010; Wardlawand Moncur, 1995; Wardlaw,2002;Xuetal.,1995).

Mostbiological processesrespondexponentiallyto tempera-tureuntilanoptimumand thentheydecline (Dellet al.,2011;

ParentandTardieu,2012).Thedecliningphaseofatemperature functionhasbecomeparticularlyimportantwhenconsidering cli-matechangeimpacts(SchlenkerandRoberts,2009).Fivemodels modiﬁedtheirtemperaturesub-routinesbyincludingthis declin-ingphasewithincreasingtemperatures,and3modelsthatalready includedadecliningphaseusedtheHSCcalibrationdatasetto cali-bratetheimplementedfunctionortochangetheirshape(e.g.from trapezoidaltonon-linear).Regardingcardinaltemperaturesused for describing thetemperature responseof phenological devel-opmentand biomass growth(i.e. theminimum, optimum, and maximumtemperatures),therewasnoclearaccordanceamong models,withtheexceptionoftheoptimumtemperaturefor radi-ationuseefﬁciency(∼20◦_C) _and_the_minimum_temperature_for

bothphenologicaldevelopmentandbiomassgrowth(∼0◦_C)_(Wang

etal.,unpublished).Somemodelscalibratedtheoptimumandthe maximumtemperaturesusingthecalibrationdatasetandthebest matching values obtainedthrough calibrationmighthave been influencedbythespecificitiesofeachmodel(Eitzingeretal.,2012). Threemodelsaddedasub-routineforaccountingforheatstress impactongrainnumberand/orsize.Elevatedtemperaturesbefore anthesisacceleratedevelopmentofthespikeanddecreasegrain number(SainiandAspinall, 1982)andpotentialfinal grainsize (Ferriseetal.,2010).Temperaturesabove31◦C aroundanthesis

(11)

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Fig.5. Simulatedandmeasureddaysfromsowingtoanthesis(AandB),daysfromanthesistomaturity(CandD),leafareaindex(LAI)(EandF),harvestindex(HI)(Gand H),grainnumber(IandJ),singlegraindrymass(KandL),finaltotalabovegroundbiomass(MandN),finalgrainyield(OandP),versusmeangrowingseasontemperature forthecalibration(A,C,E,G,I,KM,O)andevaluation(B,D,F,H,J,L,N,P)datasets.Blackdottedlinesanddarkgreyareasaree-median(ensemblemedian)andthe10thto 90thpercentilerangeofthe15original(unimproved)models,respectively.Solidredlinesandlightgreyareasaree-medianandthe10thto90thpercentilerangeofthe15 improvedmodels,respectively.Symbolsaremeasuredmean±1s.d.forn=3independentreplicates.NotethatforLAI,therewerenoobservationsfortheevaluationdata set(forinterpretationofthereferencestocolourinthisfigurelegend,thereaderisreferredtothewebversionofthisarticle.)

were reported toreduce ear fertility and grainset and conse-quentlygrainnumber(Alghabarietal.,2014;Ferrisetal.,1998), andtemperaturesabove35◦Catthebeginningofgrainﬁllingwere reportedtoreducepotentialﬁnalgrainsize(HawkerandJenner, 1993;Keelingetal.,1994;Sainietal.,1984,1983).

Twomodelsconsideredheatstressimpactonleafdevelopment andexpansiongrowth,whichwasreportedtoslowdownunder heatstress(KempandBlacklow,1982).Somemodelsimproved theperformancesbyincludingormodifyingcanopytemperature routines.

However,modelling ofsuchtemperature responsesare cur-rentlylimitedbytheavailabilityofexperimentaldatasetswhere theseresponsescanbequantiﬁed.Furthermodelingand experi-mentalworkarealsoneededtoreachagreementamongmodels regarding the cardinal temperature of key physiological pro-cessesdeterminingwheatdevelopmentandgrowth.Furthermore, improvedmodelversionsshouldbefurthertestedthrough sensi-tivityanalysisinordertobetterunderstandtheimpactofnewand revisedprocessesandadditionalparametersinmodelstructures onsimulatedvariables.

4.2. Modelimprovementeffectsontheaccuracyandpredictive skillsofMME

Afterimprovement,thevariationrangeoftheMMEwasreduced athightemperaturesintheevaluationdataset.Thereductionof thevariationbetweenthemodelsathightemperaturesdoesnot eliminatethevalueofusingMMEasmodelstructuresremainstill differentanduncertaintywillcontinuetobepartofimpact assess-ment.Grainyieldpredictiveskill(quantiﬁedinthisstudybyMSE)of theMMEwasdoubled,andafterimprovementitwascomparable tothatofhindcasts,suggestingthattheimprovedmodel predic-tionsrelatedtotheimpactofheatstresscanbeconsideredreliable andconsistentinrelationtotheobservederror.

MME accuracy for grain yield and above ground biomass wasalso doubledafter improvement. Theunimproved and the improvedMMEhadsimilarsquaredbias,indicatingthatthemain sourceofvariationintheconsideredMMEwasduetodifferences betweenmodels.Theseresultssuggestthatthecurrentlevelofbias mightbeanintrinsicpropertyofcurrentsimulationsorofthe con-sideredMMEoralsopossiblylinkedtootheruncertaintyfactors

(12)

Unimproved Improved 0 1 2 3 4 5

MSE and MSEP

for g rain yield ([t DM ha −1 ] 2) Calibration

A

Unimproved Improved Evaluation Unimproved Improved Prediction 0 10 20 30 40 50 60 70

Unimproved Improved

MSE for Sowing to Anthesis

B

MSE for S o wing to Anthesis _(da ys) 2 0 10 20 30 40 50

Unimproved Improved

MSE for Anthesis to Maturity

C

MSE

for Anthesis to Matu

rity (d a ys) 2 0 5 10 15 20 25 30

Unimproved Improved

MSE for Above ground biomass

D

MSE for ab ov e ground biomass ([t DM ha −1 ] 2)

Fig.6. Meansquarederror(MSE)decompositionofgrainyieldsimulatedbythe15unimprovedandimprovedmodelsforthecalibrationandevaluation(comparisonwith hindcast)datasets,andthepredictiondataset(“unknown”dataset)(panelA).MSEdecompositionfordaysfromsowingtoanthesis(panelB),anthesistomaturity(panel C)andﬁnaltotalabovegroundbiomass(panelD)simulatedbythe15unimprovedandimprovedmodelsfortheevaluationdataset.InpanelA,thepredictiondatasetisthe sameastheevaluationdatasetbutisusedasan“unknown”datasettobepredicted.MSEwasdecomposedintosquaredbias(grey)andvariance(white).Dataaremean±1 s.e.for15(calibration)and14(evaluationandprediction)site/year/sowingdatescombinations.

thatarestillnotconsideredexplicitly.Duetothesimilarityofthe improvedandunimprovedMMEsquaredbiases,theresultsrelated totheanalysisofthepredictiveskillsoftheMMEweresimilarto theevaluationresults.Theagreementbetweentheevaluationand thepredictionresultsisanimportantresultandisrelatedtothe usefulnessofcropmodelsinexploringtheconsequencesonclimate change.Afundamentalquestionincropmodelimpactassessments isthequalityassessmentofestimatesofuncertainty(Wallachetal., 2015).Fortheﬁrsttime,thequalityofaMMEwasmeasured,and itshowedthatatthecurrentstateofcropmodeldevelopment, especiallyafterimprovement,predictionuncertaintiesand hind-casterrorsareatthesamelevel.Therefore,givenacertainlevelof squaredbiasmeasuredwithhindcastandappliedtopredictions,we canassumethatpredictionswiththesemodelsarereliable.Since inthisworkthelevelofpredictionuncertaintywasmeasuredusing thesquaredbiasforadatasetthatwasalsousedforcalibration,we suggestthatforfuturepredictionuncertainty assessmentsdone withthisMME,thesquaredbiasoftheimprovedmodels calcu-latedfortheevaluationdatasetsisusedasthereferenceprediction squaredbias.

4.3. Modelimprovementeffectsone-medianuncertainty

TwofundamentalquestionsinMMEuncertaintyarewhatisthe uncertaintyoftheMMEpredictorandhowdoesthequalityofthe uncertaintyestimatesvarywiththenumberofmodels(Wallach etal.,2015).

Asexpected,theCVandtheRMSREofe-mediandecreasedwith thenumberofmodels.OnaveragetheunimprovedversionofMME wasnotabletoreachthebenchmarkofCV≤13.5%forgrainyield (Tayloretal.,1999):evenwitharandommodelpopulationof15

modelstheaverageCVwas17%.Onthecontrary,theimproved MMEreachedCV≤13.5%with8modelsintheensembleandat this modelensemble sizetheRMSREof e-medianwasreduced by16%.MMEcanbeapowerfultoolfor climateimpact assess-mentsastheytakeadvantageofthepresenceofdifferentmodels intheensemble(Martreetal.,2015),buttheyarecostlyto exe-cute.ExecutionofMMEimplypublicavailabilityofcropmodels and/ortheinterestofmodelinggroups inparticipatingin coor-dinatedsimulationexercises,theiravailabilityoffundingand/or computationalresourcestodotherequestedsimulations(Tebaldi andKnutti,2007).Cropmodelsaredevelopedusingdifferent soft-warelanguagesand/orimplementationswhich makestheiruse bythirdpartiesdifﬁcult.Amodelframeworkthatisabletohost multiplecropmodelsmostprobablywillovercomethese limita-tionsinthefuture(Bergezetal.,2014;Davidetal.,2013;Donatelli etal.,2014;Holzworthetal.,2015Holzworthetal.,2015),butthe numberofcropmodelsincludedintheseplatformsisstilllimited and,evenwhenavailable,executingseveralcropmodelsrequiresat leastsomeknowledgeaboutthespeciﬁcsofeachmodelinorderto correctlyinterpretresults.Thereforethereductionoftherequired numberofmodelsinanensembleisafundamentalresultkey con-clusionofthisstudythatmakesmulti-modelimpactassessments morerealisticpracticalandlesscostlytobeexecuted.

UntilnowtheconstitutionofcropMMEshasbeenbasedonthe “ensembleofopportunity”approachwithoutanapriori speciﬁ-cationthatdeﬁnesthecharacteristicsofamodelthat shouldor shoudnotbepartofanensemble(SolazzoandGalmarini,2015). Inmostcases,theonlyrequirementforparticipationhasbeenthat theremustbeapublisheddescriptionofthemodel.However,one couldenvisageamorepro-activechoiceofmodels.Forexample, Solazzoand Galmarini(2015)proposedscreening modelstobe

(13)

0 20 40 60 80 Coefficient of va riation of e−median of g rain yield (%)

A

Unimproved models Improved models 0 2 4 6 8 10 12

_B

Coefficient of va riation of e−median of S o wing to Anthesis (%) 0 10 20 30 40 50 60 70

_C

1 3 5 7 9 11 13 15 Coefficient of va riation of e−median of ab ov e ground biomass (%) Number of models

Fig.7. Coefficientofvariationofmulti-modelensemblee-medianforfinalgrain yield(panelA),daysfromsowingtomaturity(panelB)andfinaltotalaboveground biomass(PanelC),versusnumberofmodelsinanensemble.Valueswerecalculated basedon20,000bootstrapsamplesof1–15original(unimproved)(bluecircles) andimproved(redtriangles)modelsfortheindependentevaluationdataset.The horizontalblackdashedlineinpanelAindicatesthemeancoefficientofvariationof GYcalculatedfromameta-analysisofagronomicfieldtrials(Tayloretal.,1999).For readability,resultsforunimprovedandimprovedmodelsareshownforoddand evennumberofmodels,respectively.Symbolsanderrorbarsindicatemeanand ±s.d.ofthe20,000samplee-medianvalues,respectively(forinterpretationofthe referencestocolourinthisfigurelegend,thereaderisreferredtothewebversion ofthisarticle.)

includedinaMMEinordertoreduceredundancy.Theypropose doingthisinthreesteps:i)determinationtowhatextentthe vari-abilitypresentintheobservationsisreproducedbytheMME,ii) determinationof theminimumnumber ofmodelsnecessaryto representtheobserved variabilityiii)identiﬁcationofthe mod-elstobeincludedinareduced MMEtobeusedforsubsequent analysis.Analternativeapproachtoexcludingsomemodelswould

Fig.8.Rootmeansquaredrelativeerror(RMSRE)ofmulti-modelensemble e-medianforfinalgrainyield(GY)versusnumberofmodelsintheensemblefor original,unimprovedmodels(bluecircles)andimprovedmodels(redtriangles)for theevaluationfielddataset.Valuesaremean±1s.d.for20,000bootstrapsamples. Forreadability,resultsforunimprovedandimprovedmodelsareshownforoddand evennumberofmodels,respectively(forinterpretationofthereferencestocolour inthisfigurelegend,thereaderisreferredtothewebversionofthisarticle.)

betodifferentiallyweightthedifferentmodelsinaMMEinorder toobtainaweightedaverageprediction.Intheclimatemodeling communityweightingmethodsbasedonmodelperformancehave beenreportedtoimproveperformanceofaMMEpredictor(Tebaldi andKnutti,2007).However,weightingbasedonfitofhindcastsis difficult,becauseitrequiresachoiceofwhichoutputvariablesto considerandhowtocombinetheminanoverallcriterion.Another openquestionisrelatedtothequantificationoftheglobal uncer-taintyinimpactassessments.Herewefocusedourattentionon theuncertaintyrelatedtomodelsimulationsandMMEassuminga fixed(non-varietal)parametersetforeachmodel.Furthermorewe didnotincludeuncertaintyrelatedtoweather,soil,and manage-mentinputs.Inthecaseofclimatechangeimpactassessmentsthe uncertaintyrelatedtoweatherinputsmayhaveahigher impor-tance.

5. Conclusions

Followingtheexampleoftheclimatesciencecommunity,the cropmodelcommunityhasrecentlyproposedtheuseofMMEas avalidapproachtoanalyzeimpactassessmentuncertaintiesfor currentandfutureclimateconditions.However,differentlyfrom climatemodels,the performance ofcropmodels canbe evalu-atedagainstcontrolledﬁeldexperimentsfromenvironmentsthat alreadyexperiencehigherthannormalgrowingseason tempera-turescreatingconditionsthatmightbecomecommoninthefuture. Usingauniquesetofexperimentsfortestingtheimpactofheat stressonwheatcrops,wedemonstratedthatcropmodel improve-mentscanincreasetheaccuracyofsimulations,increasepredictive skillsofMME’s,reduceMMEuncertainty,andreducethenumber ofmodelsneededforreliableimpactassessments.

Authorcontributions

AM,PM,SA,andFE,Conceivedanddesignedresearch.AM,PM, andDWanalyzedthesimulationresults.AMandPMwrotethe manuscript.SA,FW,DW,EW,CM,RPM,ACR,andMASrevisedthe manuscript.AM,PM,SA,FE,CM,RPR,MAS,EW,BTK,CB,BB,DC, AJC,JD,BD,EER,SG,KCK,AKK,BL,GJO,JEO,EP,PS,TS,PJT,KW,ZZ, andYZperformedsimulations,improvedindividualmodelsand

(14)

discussedtheresults.PDA,BAK,MJO,MPR,AR,GWW,andJWW providedexperimentaldata.

Acknowledgements

AMhasreceivedthesupportoftheEUintheframeworkofthe Marie-CurieFP7COFUNDPeopleProgramme,throughtheawardof anAgreenSkillsfellowshipundergrantagreementno. PCOFUND-GA-2010-267196. PM and DW acknowledge support from the FACCEJPIMACSURproject(031A103B)throughthemetaprogram AdaptationofAgricultureandForeststoClimateChange(AAFCC) oftheFrenchNationalInstituteforAgriculturalResearch(INRA). SAandDCreceivedfinancialsupportfromtheInternationalFood PolicyResearchInstitute(IFPRI)andtheInternationalMaizeand WheatImprovementCenter(CIMMYT).FEreceivedsupportfrom theFACCEMACSURproject(031A103B)fundedthroughthe Ger-manFederalMinistryofEducationandResearch(2812ERA115)and EERwasfundedthroughtheGermanFederalMinistryofEconomic CooperationandDevelopment(Project:PARI).EWwasfundedby thebyCSIROandtheChineseAcademyofSciencesthroughthe project‘Advancingcropyieldwhilereducingtheuseofwaterand nitrogen’.CMreceivedfinancialsupportfromtheKULUNDAproject (01LL0905L)andtheMACMITproject(01LN1317A)fundedthrough theGermanFederalMinistryofEducationandResearch(BMBF). RPRreceivedfinancialsupportfromFACCEMACSURprojectfunded throughtheFinnishMinistryofAgricultureandForestry.MPRand PDAreceivedfundingfromtheCGIARResearchProgramonClimate Change,Agriculture,and FoodSecurity(CCAFS).CB wasfunded throughtheHelmholtzproject‘REKLIM-RegionalClimateChange: Causes and Effects’ Topic 9: ‘Climate Change and Air Quality’. KCKandCNwerefundedbytheFACCEMACSURprojectthrough theGermanFederalOffice forAgricultureand Food(BLE).GO’L wasfundedthroughtheAustralianGrainsResearchand Develop-mentCorporationandtheDepartmentofEnvironmentandPrimary IndustriesVictoria,Australia.JEOwerefundedthroughtheFACCE MACSUR project by the Danish Strategic Research Innovation Foundation.ZZreceivedscholarshipfromtheChinaScholarship CouncilthroughtheCSIROandChineseMinistryofEducationPhD ResearchProgram.RothamstedResearchissupportedviathe20:20 WheatProgrammebytheUKBiotechnologyandBiologicalSciences ResearchCouncil.

AppendixA. Supplementarydata

Supplementarydataassociatedwiththisarticlecanbefound,in theonlineversion,athttp://dx.doi.org/10.1016/j.fcr.2016.05.001.

References

Alexandratos,N.,Bruinsma,J.,2012.WorldAgricultureTowards2030/2050:The

2012Revision.FAO,Rome.

Alghabari,F.,Lukac,M.,Jones,H.E.,Gooding,M.J.,2014.EffectofRhtallelesonthe toleranceofwheatgrainsettohightemperatureanddroughtstressduring bootingandanthesis.J.Agron.CropSci.200,36–45,http://dx.doi.org/10.1111/

jac.12038.

Angulo,C.,Rötter,R.,Lock,R.,Enders,A.,Fronzek,S.,Ewert,F.,2013.Implicationof cropmodelcalibrationstrategiesforassessingregionalimpactsofclimate changeinEurope.Agric.For.Meteorol.170,32–46,http://dx.doi.org/10.1016/j.

agrformet.2012.11.017.

Asseng,S.,Keating,B.A.,Fillery,I.R.P.,Gregory,P.J.,Bowden,J.W.,Turner,N.C., Palta,J.A.,Abrecht,D.G.,1998.PerformanceoftheAPSIM-wheatmodelin

WesternAustralia.FieldCropsRes.57,163–179.

Asseng,S.,Jamieson,P.,Kimball,B.,Pinter,P.,Sayre,K.,Bowden,J.,Howden,S., 2004.Simulatedwheatgrowthaffectedbyrisingtemperature,increasedwater deﬁcitandelevatedatmosphericCO2.FieldCropsRes.85,85–102,http://dx.

doi.org/10.1016/S0378-4290(03)00154-0.

Asseng,S.,Foster,I.,Turner,N.C.,2011.Theimpactoftemperaturevariabilityon wheatyields.Glob.ChangeBiol.17,997–1012,http://dx.doi.org/10.1111/j.

1365-2486.2010.02262.x.

Asseng,S.,Ewert,F.,Rosenzweig,C.,Jones,J.W.,Hatﬁeld,J.L.,Ruane,A.C.,Boote, K.J.,Thornburn,P.J.,Rötter,R.P.,Cammarano,D.,Brisson,N.,Basso,B.,Martre,

P.,Aggarwal,P.K.,Angulo,C.,Bertuzzi,P.,Biernath,C.,Challinor,A.J.,Doltra,J., Gayler,S.,Goldberg,R.,Grant,R.,Heng,L.,Hooker,J.,Hunt,L.A.,Ingwersen,J., Izaurralde,R.C.,Kersebaum,K.C.,Müller,C.,NareshKumar,S.,Nendel,C., O’Leary,G.,Olesen,J.E.,Osborne,T.M.,Palosuo,T.,Priesack,E.,Ripoche,D., Semenov,M.A.,Shcherbak,I.,Steduto,P.,Stöckle,S.,Stratonovitch,P.,Streck, T.,Supit,I.,Tao,F.,Travasso,M.,Waha,K.,Wallach,D.,White,J.W.,Williams, J.W.,Wolf,J.,2013.Uncertaintyinsimulatingwheatyieldsunderclimate change.Nat.Clim.Change,1–6,http://dx.doi.org/10.1038/ncliamte1916. Asseng,S.,Ewert,F.,Martre,P.,Rötter,R.P.,Lobell,D.B.,Cammarano,D.,Kimball,

B.A.,Ottman,M.J.,Wall,G.W.,White,J.W.,Reynolds,M.P.,Alderman,P.D., Prasad,P.V.V.,Aggarwal,P.K.,Anothai,J.,Basso,B.,Biernath,C.,Challinor,A.J., DeSanctis,G.,Doltra,J.,Fereres,E.,Garcia-Vila,M.,Gayler,S.,Hoogenboom,G., Hunt,L.A.,Izaurralde,R.C.,Jabloun,M.,Jones,C.D.,Kersebaum,K.C.,Koehler, A.-K.,Müller,C.,NareshKumar,S.,Nendel,C.,O’Leary,G.,Olesen,J.E.,Palosuo, T.,Priesack,E.,EyshiRezaei,E.,Ruane,A.C.,Semenov,M.A.,Shcherbak,I., Stöckle,C.,Stratonovitch,P.,Streck,T.,Supit,I.,Tao,F.,Thorburn,P.J.,Waha,K., Wang,E.,Wallach,D.,Wolf,J.,Zhao,Z.,Zhu,Y.,2015.Risingtemperatures reduceglobalwheatproduction.Nat.Clim.Change5,143–147,http://dx.doi.

org/10.1038/nclimate2470.

Basso,B.,Cammarano,D.,Troccoli,A.,Chen,D.,Ritchie,J.T.,2010.Long-termwheat responsetonitrogeninarainfedMediterraneanenvironment:ﬁelddataand simulationanalysis.Eur.J.Agron.33,132–138,http://dx.doi.org/10.1016/j.eja.

2010.04.004.

Bassu,S.,Brisson,N.,Durand,J.-L.,Boote,K.,Lizaso,J.,Jones,J.W.,Rosenzweig,C., Ruane,A.C.,Adam,M.,Baron,C.,Basso,B.,Biernath,C.,Boogaard,H.,Conijn,S., Corbeels,M.,Deryng,D.,DeSanctis,G.,Gayler,S.,Grassini,P.,Hatﬁeld,J.,Hoek, S.,Izaurralde,C.,Jongschaap,R.,Kemanian,A.R.,Kersebaum,K.C.,Kim,S.-H., Kumar,N.S.,Makowski,D.,Müller,C.,Nendel,C.,Priesack,E.,Pravia,M.V.,Sau, F.,Shcherbak,I.,Tao,F.,Teixeira,E.,Timlin,D.,Waha,K.,2014.Howdovarious maizecropmodelsvaryintheirresponsestoclimatechangefactors?Glob. ChangeBiol.20,2301–2320,http://dx.doi.org/10.1111/gcb.12520.

Bergez,J.E.,Raynal,H.,Launay,M.,Beaudoin,N.,Casellas,E.,Caubel,J.,Chabrier,P., Coucheney,E.,Dury,J.,GarciadeCortazar-Atauri,I.,Justes,E.,Mary,B., Ripoche,D.,Ruget,F.,2014.EvolutionoftheSTICScropmodeltotacklenew environmentalissues:newformalismsandintegrationinthemodellingand simulationplatformRECORD.Environ.Modell.Softw.62,370–384,http://dx.

doi.org/10.1016/j.envsoft.2014.07.010.

Beringer,T.,Lucht,W.,Schaphoff,S.,2011.Bioenergyproductionpotentialofglobal biomassplantationsunderenvironmentalandagriculturalconstraints.GCB Bioenergy3,299–312,http://dx.doi.org/10.1111/j.1757-1707.2010.01088.x. Berntsen,J.,Petersen,B.M.,Jacobsen,B.H.,Olesen,J.E.,Hutchings,N.J.,2003.

Evaluatingnitrogentaxationscenariosusingthedynamicwholefarm simulationmodelFASSET.Agric.Syst.76,817–839,http://dx.doi.org/10.1016/

S0308-521X(02)00111-7.

Biernath,C.,Gayler,S.,Bittner,S.,Klein,C.,Högy,P.,Fangmeier,A.,Priesack,E., 2011.Evaluatingtheabilityoffourcropmodelstopredictdifferent environmentalimpactsonspringwheatgrowninopen-topchambers.Eur.J. Agron.35,71–82,http://dx.doi.org/10.1016/j.eja.2011.04.001.

Bondeau,A.,Smith,P.C.,Zaehle,S.,Schaphoff,S.,Lucht,W.,Cramer,W.,Gerten,D., Lotze-campen,H.,Müller,C.,Reichstein,M.,Smith,B.,2007.Modellingtherole ofagricultureforthe20thcenturyglobalterrestrialcarbonbalance.Glob. ChangeBiol.13,679–706,http://dx.doi.org/10.1111/j.1365-2486.2006.01305.

x.

Burton,I.,Lim,B.,2005.Achievingadequateadaptationinagriculture.Clim. Change70,191–200,http://dx.doi.org/10.1007/s10584-005-5942-z. Byjesh,K.,Kumar,S.N.,Aggarwal,P.K.,2010.Simulatingimpacts,potential

adaptationandvulnerabilityofmaizetoclimatechangeinIndia.Mitig.Adapt. Strateg.Glob.Change15,413–431,

http://dx.doi.org/10.1007/s11027-010-9224-3.

Cao,W.,Moss,D.N.,1997.Modellingphasicdevelopmentinwheat:aconceptual integrationofphysiologicalcomponents.J.Agric.Sci.129,163–172,http://dx.

doi.org/10.1017/S0021859697004668.

Cao,W.,Liu,T.,Luo,W.,Wang,S.,Pan,J.,Guo,W.,2002.Simulatingorgangrowthin wheatbasedontheorgan-weightfractionconcept.PlantProd.Sci.,http://dx.

doi.org/10.1626/pps.5.248.

Challinor,A.J.,Wheeler,T.R.,Craufurd,P.Q.,Slingo,J.M.,Grimes,D.I.F.,2004.Design andoptimisationofalarge-areaprocess-basedmodelforannualcrops.Agric. For.Meteorol.124,99–120,http://dx.doi.org/10.1016/j.agrformet.2004.01.002. Challinor,A.,Martre,P.,Asseng,S.,Thornton,P.,Ewert,F.,2014.Makingthemostof climateimpactsensembles.Nat.Clim.Change4,77–80,http://dx.doi.org/10.

1038/nclimate2117.

Chauhan,S.,Srivalli,S.,Nautiyal,A.R.,Khanna-Chopra,R.,2010.Wheatcultivars differinginheattoleranceshowadifferentialresponsetomonocarpic senescenceunderhigh-temperaturestressandtheinvolvementofserine proteases.Photosynthetica47,536–547,

http://dx.doi.org/10.1007/s11099-009-0079-3.

Chen,C.,Wang,E.,Yu,Q.,2010.Modelingwheatandmaizeproductivityasaffected byclimatevariationandirrigationsupplyinNorthChinaPlain.Agron.J.102, 1037,http://dx.doi.org/10.2134/agronj2009.0505.

David,O.,Ascough,J.C.,Lloyd,W.,Green,T.R.,Rojas,K.W.,Leavesley,G.H.,Ahuja, L.R.,2013.Asoftwareengineeringperspectiveonenvironmentalmodeling frameworkdesign:theobjectmodelingsystem.Environ.Modell.Softw.39, 201–213,http://dx.doi.org/10.1016/j.envsoft.2012.03.006.

Dell,A.I.,Pawar,S.,Savage,V.M.,2011.Systematicvariationinthetemperature dependenceofphysiologicalandecologicaltraits.Proc.Natl.Acad.Sci.U.S.A. 108,10591–10596,http://dx.doi.org/10.1073/pnas.1015178108.