amber-package {amber}R Documentation

Overview of AMBER functions


The Canadian Land Surface Scheme Including Biogeochemical Cycles (CLASSIC) is the land surface component of the Canadian Earth System Model (CanESM) (Melton et al, 2020). The Automated Model Benchmarking R package (AMBER) evaluates the ability of CLASSIC to reproduce land surface processes by comparing model outputs against quasi-observational data sets derived from remote sensing products, eddy covariance flux tower measurements, and stream flow measurements. To summarize model performance across different statistical metrics AMBER employs a skill score system that was originally developed by the International Land Model Benchmarking (ILAMB) framework (Collier et al., 2018). AMBER was created to tailor the ILAMB skill score approach for CLASSIC model outputs. While AMBER was tested for CLASSIC only it may also work for other models.

amber functions

The functions provided by AMBER can be grouped into three categories:

Skill scores

The performance of a model is expressed through scores that range from zero to one, where increasing values imply better performance. These scores are usually computed in five steps:

The latter includes the bias, root-mean-square error, phase shift, inter-annual variability, and spatial distribution. The equations for computing the bias score (S_{bias}) are:

(i) \ bias(λ, φ)=\overline{v_{mod}}(λ, φ)-\overline{v_{ref}}(λ, φ)

(ii) \ \varepsilon_{bias}=|bias(λ, φ)|/σ_{ref}(λ, φ)

(iii) \ s_{bias}(λ, φ)=e^{-\varepsilon_{bias}(λ, φ)}

(iv) \ S_{bias}=\overline{\overline{s_{bias}}}

The equations for computing the root mean square error score (S_{rmse}) are:

(i) \ crmse(λ, φ) =√{\frac{1}{t_{f}-t_{0}}\int_{t_{0}}^{t_{f}}[(v_{mod}(t,λ, φ)-\overline{v_{mod}}(λ, φ))-(v_{ref}(t,λ, φ)-\overline{v_{ref}}(λ, φ))]^{2}dt}

(ii) \ \varepsilon_{rmse}(λ, φ)=crmse(λ, φ)/σ_{ref}(λ, φ)

(iii) \ s_{rmse}(λ, φ)=e^{-\varepsilon_{rmse}(λ, φ)}

(iv) \ S_{rmse}=\overline{\overline{s_{rmse}}}

The equations for computing the phase score (S_{phase}) are:

(i) \ θ(λ, φ)=\max(c_{mod}(t,λ, φ))-\max(c_{ref}(t,λ, φ))

(ii) \ \textrm{not applicable, as units are consistent across all variables}

(iii) \ s_{phase}(λ, φ)=\frac{1}{2}[1+\cos(\frac{2πθ(λ, φ)}{365})]

(iv) \ S_{phase}=\overline{\overline{s_{phase}}}

The equations for computing the inter-annual variability score (S_{iav}) are:

(i) \ iav_{ref}(λ, φ)=√{\frac{1}{t_{f}-t_{0}}\int_{t_{0}}^{t_{f}}(v_{ref}(t,λ, φ)-c_{ref}(t,λ, φ))^{2}dt}

(i) \ iav_{mod}(λ, φ)=√{\frac{1}{t_{f}-t_{0}}\int_{t_{0}}^{t_{f}}(v_{mod}(t,λ, φ)-c_{mod}(t,λ, φ))^{2}dt}

(ii) \ \varepsilon_{iav}=|(iav_{mod}(λ, φ)-iav_{ref}(λ, φ))|/iav_{ref}(λ, φ)

(iii) \ s_{iav}(λ, φ)=e^{-\varepsilon_{iav}(λ, φ)}

(iv) \ S_{iav}=\overline{\overline{s_{iav}}}

The equations for computing the spatial distribution score (S_{dist}) are:

(i) \ σ=σ_{\overline{v_{mod}}}/σ_{\overline{v_{ref}}}

(ii) and (iii) \ \textrm{not applicable}

(iv) \ S_{dist}=2(1+R)/(σ+\frac{1}{σ})^{2}

where \overline{v_{mod}}(λ, φ) and \overline{v_{ref}}(λ, φ) are the mean values in time t of a variable v as a function of longitude λ and latitude φ for model and reference data, respectively, t_0 and t_f are the initial and final time step, σ_{\overline{v_{mod}}} and σ_{\overline{v_{ref}}} are the standard deviation of the time mean values from the model and reference data, and R is the spatial correlation coefficient of \overline{v_{ref}}(λ, φ) and \overline{v_{mod}}(λ, φ).

Score values are then combined to derive a single overall score for each output variable:

(v) \ S_{overall}=\frac{S_{bias}+2S_{rmse}+S_{phase}+S_{iav}+S_{dist}}{1+2+1+1+1}.

Note that S_{rmse} is weighted by a factor of two, which emphasizes its importance.


Collier, Nathan, Forrest M. Hoffman, David M. Lawrence, Gretchen Keppel-Aleks, Charles D. Koven, William J. Riley, Mingquan Mu, and James T. Randerson. 2018. “The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation.” Journal of Advances in Modeling Earth Systems 10 (11): 2731–54.

Melton, Joe R., Vivek K. Arora, Eduard Wisernig-Cojoc, Christian Seiler, Matthew Fortier, Ed Chan, and Lina Teckentrup. 2020. “CLASSIC v1.0: The Open-Source Community Successor to the Canadian Land Surface Scheme (CLASS) and the Canadian Terrestrial Ecosystem Model (CTEM) - Part 1: Model Framework and Site-Level Performance.”

[Package amber version 1.0.3 Index]