R: Sensitivity Analysis Tools for LSD Simulations

LSDsensitivity-package {LSDsensitivity}

R Documentation

Sensitivity Analysis Tools for LSD Simulations

Description

Tools for sensitivity analysis of LSD simulation models. Reads object-oriented data produced by LSD simulation models and performs screening and global sensitivity analysis (Sobol decomposition method, Saltelli et al. (2008) ISBN:9780470725177). A Kriging or polynomial meta-model (Kleijnen (2009) <doi:10.1016/j.ejor.2007.10.013>) is estimated using the simulation data to provide the data required by the Sobol decomposition. LSD (Laboratory for Simulation Development) is free software developed by Marco Valente and Marcelo C. Pereira (documentation and downloads available at <https://www.labsimdev.org/>).

Details

The LSDsensitivity R package provides tools to analyze simulated experiments from LSD. LSD offers native tools to sample the configuration (factor) space of a simulation model using different design of experiments (DoE). The produced experimental design data can be transparently imported to R by using the function read.doe.lsd().

The package offers two sensitivity analysis (SA) methods (elementary.effects.lsd() and sobol.decomposition.lsd()) pre-configured for application on LSD simulations: Morris Elementary Effects (EE) and Sobol Variance Decomposition (SVD).

EE (elementary.effects.lsd()) employs a simple one-factor-at-a-time (OAT) SA and is usually applied as an initial screening method while selecting relevant factors to a SVD global SA. EE requires an appropriate set of sample points (the DoE) which can be generated in LSD when "EE Sampling" is selected in the "Data" menu. Please make sure to take note of the DoE parameters used for the sampling, as they will be required for the configuration of the R analysis script.

Due to its high computational cost, sobol.decomposition.lsd() (SVD) is performed over a meta-model fitted from the experimental data produced by the LSD original model. The meta-model can be fitted using different sampling strategies offered by LSD, being "NOLH Sampling" (Near Orthogonal Latin Hypercube) usually the most efficient. Additionally to the set of samples used to fit the meta-model, it is recommended to also generate another set for the (external) validation of the meta-model ("MC Range Sampling" is the recommended option).

The package offers two meta-modeling (MM) methods for using with SVD: Kriging and polynomial. Kriging (kriging.model.lsd()) is offered under five different variance kernels (Matern 5/2, Matern3/2, Gaussian, exponential and power exponential) and two trend models (constant or first order polynomial) to choose, including auto-selection to the best fitting alternative. Polynomial meta-models of first or second order, with or without interactions, and auto-selection are also offered (polynomial.model.lsd()). Kriging is the recommended option in most cases.

Additionally, the package offers tools for the graphical representation of the meta-models response surfaces (2D and 3D) (response.surface.lsd()), to predict meta-model response in specific points in the factor space (model.pred.lsd()), to identify maximum and minimum responses from a set of factors (model.limits.lsd()), and to find optimal parameter settings using the meta-model (model.optim.lsd()).

For a complete list of exported functions, use library(help = "LSDsensitivity").

LSD 7.0+ default installation provides example scripts for the usage of the LSDsensitivity package. LSD can be downloaded at https://github.com/marcov64/Lsd/. They can also be retrieved from the package itself using the commands:

EE example: file.show(system.file("examples", "elementary-effects-SA.R", package = "LSDsensitivity"))

Kriging SVD example: file.show(system.file("examples", "kriging-sobol-SA.R", package = "LSDsensitivity"))

Polynomial SVD example: file.show(system.file("examples", "poly-sobol-SA.R", package = "LSDsensitivity"))

Optimize MM example: file.show(system.file("examples", "optimize-MM.R", package = "LSDsensitivity"))

Note

Below are the minimum required steps to perform SA on a working LSD model using NOLH sampling, Kriging MM and SVD. The changes to perform an EE or to use a polynomial MM are also indicated, as options.

Define the parameters/initial values to be explored in the SA, their max/min ranges and the result variables over which the SA is to be done
In LMM create a no-window (command prompt) version of your model by selecting menu Model/Create 'No Window' Version
In LSD Browser make sure that all parameters/initial values are set with the correct calibration/default values (menu Data/Initial Values), the required result variables are being saved (menu Model/Change Element..., click on Save/OK or simply Save in the right mouse button context menu) and the number of MC runs for each SA sample (point) is defined (menu Run/Simulation Settings, Number of simulation runs field, typically set to 10)
Save your setup in a baseline .lsd configuration file (menu File/Save As...), preferably in a new folder inside your current model configuration folder (you can create a new folder while in the File/Save As... dialog box, if you do not, LSD will create a new folder when saving SA configuration and results files, named as your baseline configuration)
(Re)load your baseline .lsd configuration if it is not already loaded (menu File/Load...)
Choose the ranges (max/min) for each parameter/initial value in your SA exploration space by using the Sensitivity Analysis button in the menu Model/Change Element... window or the same option in the context menu (mouse right-button click on the parameter/variable name in the Variables & Parameters list box)
After choosing all ranges, save your exploration space definition as a .sa sensitivity analysis file using the same base name and folder as your .lsd baseline configuration (menu File/Save Sensitivity...)
With both the created .lsd and .sa files loaded (use menu File/Load... and File/Load Sensitivity... if required), select Data/Sensitivity Analysis/NOLH Sampling... and accept the defaults (several new .lsd files will be created in your just created baseline configuration folder (or one created by LSD, if you did not), those are the sample points for the meta-model estimation)
1. To perform Elementary Effects (EE) analysis instead of Sobol Variance Decomposition, in the step below select Data/Sensitivity Analysis/EE Sampling... instead (NOLH sampling cannot be used for EE)
2. If a polynomial meta-model (MM) is being estimated, sometimes it is preferred to use Data/Sensitivity Analysis/MC Range Sampling... despite not required
Immediately after the previous step, select menu Data/Sensitivity Analysis/MC Range Sampling... and accept the defaults (to create the external validation sample, more .lsd files will be created for the additional sampling points)
1. EE analysis does not uses external validation, so skip this step for EE
Immediately after the previous step select menu Run/Create/Run Parallel Batch, accept using the just created configuration, adjust the number of cores only if going to run in another machine (8 in a modern PC, 20 in a basic server), and decide if you want to start the (time-consuming) processing right now or later (in the current or in another machine)
If running later in the same machine, you just have to execute the created script file (.bat or .sh) inside the new folder your baseline .lsd file was created (or the one LSD created if you did not)
If running in another machine, you have to copy the entire model folder and sub-folders to the new machine (the remaining LSD folders are not required), recompile LSD for the new platform if required and execute the script file (.bat or .sh) in the same folder as your baseline .lsd file
Open R (or RStudio) and check you have the following packages installed and download them if required (if you install LSDsensitivity from CRAN or another online repository, and not from a file, all other dependency packages should be automatically downloaded):

LSDsensitivity, LSDinterface, abind, tseries, car, minqa, nloptr, Rcpp, RcppEigen, lme4, SparseM, MatrixModels, pbkrtest, quantreg, DiceKriging, kSamples, SuppDists, randtoolbox, rngWELL, rgenoud, sensitivity, xts, TTR, quadprog, zoo, quantmod
Open the kriging-sobol-SA.R example script (included in your LSD installation folder) in RStudio or your text editor
1. For EE analysis, open elementary-effects-SA.R instead
2. For the use of a polynomial MM for the SVD analysis, open poly-sobol-SA.R instead
Adjust the vector lsdVars to contain all the LSD saved variables you want to use in your analysis (do not include saved but unused variables, for performance reasons), replacing the dummies varX
Adjust the vector logVars to contain all LSD variables (included in lsdVars) that require to have the log value used in the analysis (let the vector empty, i.e. c( ), if no log variable is required)
Include in the vector newVars any new variable (not included in lsdVars) that has to be added to the dataset (let the vector empty, i.e. c( ), if no new variable is required)
Adapt the eval.vars() function to compute any new variable included in newVars (use the commented example as a reference)
Adjust the arguments to the function read.doe.lsd() for the relative folder of LSD data files (default is same as R working directory), the data files base name (the file name chosen for the baseline configuration in step 4 without the .lsd suffix) and the name of the variable to be used as reference for the sensitivity analysis (you have to run the script multiple times if there is more than one)
Save the modified script, renaming if necessary, and run it in R (or click the Source button in RStudio), redirecting output to a file first if required

Author(s)

Maintainer: NA

References

LSD documentation is available at https://www.labsimdev.org/

The latest LSD binaries and source code can be downloaded at https://github.com/marcov64/Lsd/.

Cioppa T, Lucas T (2007) Efficient nearly orthogonal and space-filling latin hypercubes. Technometrics 49(1):45-55

Kleijnen JP (2009) Kriging metamodeling in simulation: a review. Eur J Oper Res 192(3):707-716

Morris MD (1991) Factorial sampling plans for preliminary computational experiments.Technometrics 33(1):161-174

Rasmussen C, Williams C (2006) Gaussian processes for machine learning. MIT Press, Cambridge

Roustant O, Ginsbourger D, Deville Y (2012) Dicekriging, diceoptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J Stat Softw 51(1):1-55

Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, Saisana M, Tarantola S (2008) Global sensitivity analysis: the primer. Wiley, New York