compute_metrics {rchallenge}

Compute metrics of the submissions in the history.

Description

Compute metrics of the submissions in the history.

Usage

compute_metrics(hist_dir = "history", metrics, y_test, ind_quiz, read_fun)

Arguments

hist_dir

string. directory where the history of the submissions are stored. contains one subdirectory per team.

metrics

named list of functions. Each function in the list computes a performance criterion and is defined as: function(y_pred, y_test)

y_test

character or numeric vector. the test set output.

ind_quiz

indices of y_test in the quiz subset.

read_fun

function that reads a submission file and returns a vector of predictions.

Value

compute_metrics returns a named list with one named member per team. Each member is a data.frame where the rows are the submission files sorted by date and the columns are:

date

the date of the submission

file

the file name of the submission

<metric name>.quiz

the score obtained on the quiz subset

<metric name>.test

the score obtained on the test set