plot_eval_by {simulator}

Plot a metric across multiple values of a model parameter

Description

This function is to be used on simulations in which generate_model was called using the vary_along parameter. When this is a single (scalar) numeric parameter, a single plot is created in which the x-axis is this parameter. Eventually, this function should handle one or two categorical variables (in which facets are used) and one categorical combined with one continuous variable.

Usage

plot_eval_by(
  sim,
  metric_name,
  varying,
  type = c("aggregated", "raw"),
  center_aggregator = NULL,
  spread_aggregator = NULL,
  use_ggplot2 = TRUE,
  main,
  xlab,
  ylab,
  xlim,
  ylim,
  include_zero = FALSE,
  legend_location = "topright",
  method_col = seq(num_methods),
  method_lty = rep(1, num_methods),
  method_lwd = rep(1, num_methods),
  method_pch = rep(1, num_methods),
  ...
)

Arguments

sim

an object of class Simulation

metric_name

the name of a metric to plot (ignored if custom aggregator is provided)

varying

character vector giving the name of a parameter that is varied across the models in evals. For now, this parameter must be numeric and there cannot be multiple models having the same value of this parameter.

type

if "aggregated" then shows line with error bars (line represents center_aggregator and error bars represent spread_aggregator; by default these are sample mean and estimated standard error); if type is "raw" then shows the raw data as points (with smoother overlayed)

center_aggregator

ignored if type is "raw". When NULL (which is default), the sample mean aggregator is used. User can write specialized aggregators (see definition of class Aggregator) as necessary, for example, when the evaluated metric is not scalar-valued.

spread_aggregator

ignored if type is "raw". When NULL (which is default), the sample mean aggregator is used. User can write specialized aggregators (see definition of class Aggregator) as necessary, for example, when the evaluated metric is not scalar-valued. Set spread_aggregator to NA to hide error bars.

use_ggplot2

whether to use ggplot2 (requires installation of ggplot2)

main

title of plot.

xlab

the x-axis label (default is varying)

ylab

the y-axis label (default is metric_label)

xlim

the x-axis limits to use

ylim

the y-axis limits to use

include_zero

whether ylim should include 0. Ignored if ylim is passed explicitly

legend_location

location of legend. Set to NULL to remove legend.

method_col

color to use for each method

method_lty

line style to use for each method

method_lwd

line thickness to use for each method

method_pch

point style to use for each method (default is that no points, only lines are drawn)

...

additional arguments to pass to plot (only when use_ggplot2 = FALSE).

Details

When type is "raw", the individual evals are shown (one point per model-draw-method triplet) along with a loess smooth. When type is "aggregated", then center_aggregator and spread_aggregator are used. center_aggregator is used to draw a single line per method in which the individual evals computed for each draw has been been aggregated in some way. By default, the mean_aggregator is used, which simply averages the evals computed across all draws. When spread_aggregator is non-NULL, "error bars" are drawn with (half)widths computed using spread_aggregator. By default, the se_aggregator is used, which gives an estimate of the standard error of the sample mean.

The arguments method_col, method_lty, method_lwd, method_pch only apply when use_ggplot2 is FALSE.

Examples

## Not run: 
 # suppose previously we had run the following:
 sim <- new_simulation(name = "normal-example",
                       label = "Normal Mean Estimation",
                       dir = tempdir()) %>%
   generate_model(make_my_example_model,
                  n = list(10, 20, 30),
                  vary_along = "n") %>%
   simulate_from_model(nsim = 50, index = 1:3) %>%
   run_method(my_example_method) %>%
   evaluate(my_example_loss)
   # then we could plot this
   plot_eval_by(sim, "myloss", varying = "n", include_zero = TRUE)
 
## End(Not run)