check_forecasts {scoringutils}

Check forecasts

Description

Function to check the input data before running score().

The data should come in one of three different formats:

• A format for binary predictions (see example_binary)

• A sample-based format for discrete or continuous predictions (see example_continuous and example_integer)

• A quantile-based format (see example_quantile)

Usage

check_forecasts(data)

Arguments

data

A data.frame or data.table with the predictions and observations. For scoring using score(), the following columns need to be present:

• true_value - the true observed values

• prediction - predictions or predictive samples for one true value. (You only don't need to provide a prediction column if you want to score quantile forecasts in a wide range format.)

For scoring integer and continuous forecasts a sample column is needed:

• sample - an index to identify the predictive samples in the prediction column generated by one model for one true value. Only necessary for continuous and integer forecasts, not for binary predictions.

For scoring predictions in a quantile-format forecast you should provide a column called quantile:

• quantile: quantile to which the prediction corresponds

In addition a model column is suggested and if not present this will be flagged and added to the input data with all forecasts assigned as an "unspecified model").

You can check the format of your data using check_forecasts() and there are examples for each format (example_quantile, example_continuous, example_integer, and example_binary).

Value

A list with elements that give information about what scoringutils thinks you are trying to do and potential issues.

• target_type the type of the prediction target as inferred from the input: 'binary', if all values in true_value are either 0 or 1 and values in prediction are between 0 and 1, 'discrete' if all true values are integers. and 'continuous' if not.

• prediction_type inferred type of the prediction. 'quantile', if there is a column called 'quantile', else 'discrete' if all values in prediction are integer, else 'continuous.

• forecast_unit unit of a single forecast, i.e. the grouping that uniquely defines a single forecast. This is assumed to be all present columns apart from the following protected columns: c("prediction", "true_value", "sample", "quantile","range", "boundary"). It is important that you remove all unnecessary columns before scoring.

• rows_per_forecast a data.frame that shows how many rows (usually quantiles or samples there are available per forecast. If a forecast model has several entries, then there a forecasts with differing numbers of quantiles / samples.

• unique_values A data.frame that shows how many unique values there are present per model and column in the data. This doesn't directly show missing values, but rather the maximum number of unique values across the whole data.

• warnings A vector with warnings. These can be ignored if you know what you are doing.

• errors A vector with issues that will cause an error when running score().

• messages A verbal explanation of the information provided above.

Author(s)

Nikos Bosse [e-mail]

See Also

Function to move from sample-based to quantile format: sample_to_quantile()

Examples

check <- check_forecasts(example_quantile)
print(check)
check_forecasts(example_binary)