HAL Id: hal-02792689
https://hal.inrae.fr/hal-02792689
Submitted on 5 Jun 2020
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Evaluation and relevance of the modelling approach: application to climate change
Gianni Bellocchi
To cite this version:
Gianni Bellocchi. Evaluation and relevance of the modelling approach: application to climate change. Master. Master de Science (Evaluation de modèles), 2015, 34 p. �hal-02792689�
Evaluation and relevance of
the modelling approach:
application to climate change
Gianni BELLOCCHI
Senior scientist
Institut National de la Recherche Agronomique Unité de Recherche sur l'Écosystème Prairial
63100 CLERMONT-FERRAND Tel : 04 73624866 ; Fax : 04 73624457 Email : gianni.bellocchi@clermont.inra.fr
Model evaluation
Issues on model validation
Validation of models
Climate change studies
Research directions
MODEL EVALUATION
Model evaluation (validation + verification):
action in which the quality of a mathematical model for specific objectives is established
Jakeman et al., 2006, Env. Modell. Softw.
Model
(software)
validation
ISSUES ON MODEL VALIDATION
Validation purpose
•
Purposes for validation:
– establishment of overall credibility in the model
– assessment of how “right” or “wrong” a model is in a given application
– support that a specific configuration of input data, parameter sets and model structure are appropriate for a particular application
•
Reliability (ability of the calculations involved to
reproduce the conceptual model) versus usefulness
(ability to reflect the behaviour of actual systems)
Interpretation of phenomena
• Modelled and actual variables must have the same definition, e.g.
– Does leaf area expansion accounts for the expanded part of
a leaf (lamina) only, or for the stem-like structure that is attached to the stem (base and petiole) as well?
• Importance of meta-data associated with original
observations and development of model parameters and variables
Model comparison
•
When two or more models are constructed for the same
system or purpose, it is possible to make comparisons in
order to select the best one
•
When either field or reference modelled data are not
available, attempts can be made to determine the
proximity of one model to the other, also known as
co-validation
– “Extra” capabilities of one model compared to another should not
be used in co-validation, e.g. nitrogen stress effects to plant
growth should not be part of the comparison between two models where only one model includes nitrogen processes
Model complexity
• Process-based models reflect the same complexity as the system being represented
• The comparison of model results with measured data is usually
done at the system level (result of both the feedbacks within and between each of the sub-models making up the whole)
– ideal validation should take place both at the level of sub-models and
Data accuracy and quality
• The accuracy of a model is determined by
– the authenticity of the algorithms describing the processes of
the real world
– the quality of both its input data and data used to evaluate its
outputs
• Model validation must be accompanied by critical examination of the source and nature of the data used
Random errors
– if single samples do not take
into account temporal variability
– if samples taken at different
points do not represent the actual area of interest
– if the inputs have been
modified by unnoticed
factors (processing, transit, storage, misrecording …)
Systematic errors
– if instruments are miscalibrated
– measurements are taken at
inappropriate locations or seasons
– no measurements or estimates
are made of relevant factors
– the data are not representative
of the same output as the modelled one
Robustness of model results
• Model robustness is its reliability under different sets of experimental conditions
– Lack of robustness in model results may reflect absence of explicit
reference to physical processes in the mathematical relationships
– Indicators of model robustness, e.g. Confalonieri et al., 2010, Ecol.
Modell.: ratio between model performance and agrometeorological conditions
• The validation of a sugarcane model by Keating et al., 1999, Field Crop. Res. gives an example of sound test of robustness for the following reasons:
– Large number of data sets (19) for crop growth are used
– Broad range of conditions with differing locations, irrigation
treatments, and nitrogen fertility treatments are explored
– Seasonal evolution of individual components (leaf area index, green
biomass, millable stalk biomass, stalk sucrose, and nitrogen accumulation) is compared with observed results
– Deviations between simulated and observed results are discussed,
Time histories
• When simulating energy transfer or mass transformation in dynamic models, a time delay/anticipation frequently occurs if estimated versus measured values are compared: peak
synchronization between estimates and measurements most often will not occur
• If synchronous comparison between estimates and measurements is applied, models which produce no response with respect to a specific process can yield better results, compared to models which show a time mismatch in the response
Issues on model validation: summary
• Basic ideas about validation are virtually applicable to any model, and are equally consistent with modelling in a variety of fields
• Validation is purpose-dependent, based on equivalent definition of modelled and observed phenomena, to be substantiated over a
variety of conditions (robustness), and possibly run at the level of individual processes in complex models
• Data quality raises the need of a system for grading the relative quality of the input and the relative importance of the variables to be fit
• Concerns regard specific aspects such as predictions in the far future and synchronization of modelled and observed peak values • Model comparison is complementary to proper validation
VALIDATION OF MODELS
(Bellocchi et al., 2010, Agron. Sustain. Dev.; …)
• A range of statistical measures and visual techniques can be used to assess goodness-of-fit of a model and to compare the performances of more models, depending on the problem context
Indices
– difference-based indices (RMSE, EF, CRM, ...) – association-based indices (slope, intercept, r, r2)
– pattern indices
Test statistics
– t-test, chi square-test, …
Probability distributions
– cumulative, density functions
Time mismatch analysis Aggregation of statistics
Difference-based Association-based Pattern-based Aggregation … Integrated resources
Indices and procedures for validation
Indices and procedures for validation
Squared differences
Simple differences
Absolute differences
Difference-based performance indices
Association-based performance indices / 1
|a| 514.12 t = --- = --- = 0.94 p = 0.36 s.e.(a) 544.33 |1 - b| |1 - 0.9166| t = --- = --- = 0.42 p = 0.68 s.e.(b) 0.1967 |a| 17.31 t = --- = --- = 0.03 p = 0.98 s.e.(a) 544.33 |1 - b| |1 – 1.1590| t = --- = --- = 0.80 p = 0.43 s.e.(b) 0.1967 LS RMA
Disaggregated performance indices
(after Willmott, 1981, Phys. Geogr.)
Both model users and modellers will focus on reducing the systematic error, the formers by model re-calibration, the latter by better defining the basic equations
Taylor diagrams
(Taylor, 2001, J. Geophys. Res.)
The Taylor diagram characterizes the statistical relationship between simulated values and observations through three statistics simultaneously (centered Root Mean Squared Error, correlation coefficient, standard deviation). The diagram does not provide information about overall biases, but solely characterizes the centered error
Pattern-based performance indices
Fuzzy-based performance indicators / 1
Fuzzy-based performance indicators / 2
(Confalonieri et al., 2009, Ecol. Modell.)
(Confalonieri et al., 2006, Italian Journal of Agrometeorology) WARM • interactions • high uncertainty dangerous zone low-sensitivity zone • little interactions • limited uncertainty
Fuzzy-based performance indicators / 3
Fuzzy-based Time Mismatch Indicator
Probability of exceedence
(Stöckle et al., 1998, Proceedings ICCTA)
Validation of a weather generator
Validation of models: summary
• Each approach to model validation has its advantages and drawbacks:
– multiple approaches are complementary and generally used in combination
• What values for assessment metrics indicate satisfactory models remains a subjective issue and no definitive guidance exists
because of heterogeneity of approaches and application domains
• Combination of multiple metrics into synthetic indicators where subjective choices (expert decisions) are converted into explicit and transparent rules reveals a more comprehensive picture
CLIMATE CHANGE STUDIES
Regional variability Adaptation (management) Possible impacts Impacts model (simple, complex) Local conditions (soil, climate, management) Regional model(s) Regionalization method Global model(s) 150-300 kmHigh resolution scenarios
GHG emissions Socio-economic evolution GHG emission scenarios GHG concentrations Socio-economic scenarios (SRES)
Climate forecast evaluation
(Diodato and Bellocchi, submitted in 2011, Clim Dyn.)
ECP-ESD runs
Time periods (years)
Training (A1) Testing (a1) Training (A2) Testing (a2) Training (A3) Forecasting Official run R1 R2 R3 R4 R5 R6 R7 R8 R9 1698-1900 1701-1900 1704-1900 1707-1900 1710-1900 1713-1900 1716-1900 1719-1900 1722-1900 1725-1900 1901-1930 1901-1930 1901-1930 1901-1930 1901-1930 1901-1930 1901-1930 1901-1930 1901-1930 1901-1930 1698-1980 1701-1980 1704-1980 1707-1980 1710-1980 1713-1980 1716-1980 1719-1980 1722-1980 1725-1980 1981-2010 1981-2010 1981-2010 1981-2010 1981-2010 1981-2010 1981-2010 1981-2010 1981-2010 1981-2010 1698-2010 1701-2010 1704-2010 1707-2010 1710-2010 1713-2010 1716-2010 1719-2010 1722-2010 1725-2010 2011-2040 2011-2040 2011-2040 2011-2040 2011-2040 2011-2040 2011-2040 2011-2040 2011-2040 2011-2040
Climate change projection impacts
(Graux, 2011, PhD thesis)
Climate change projection impacts
(Bellocchi et al., 2002, Proceedings ESA)
Actual climate
Changed climate
Bellocchi et al., in preparation, Techniques for validation of biophysical models. A review data acquisition
