Ranking-based metrics {mldr}

Multi-label ranking-based evaluation metrics

Description

Functions that compute ranking-based metrics, given a matrix of true labels and a matrix of predicted probabilities.

Usage

average_precision(true_labels, predictions, ...)

one_error(true_labels, predictions)

coverage(true_labels, predictions, ...)

ranking_loss(true_labels, predictions)

macro_auc(true_labels, predictions, undefined_value = 0.5,
  na.rm = FALSE)

micro_auc(true_labels, predictions)

example_auc(true_labels, predictions, undefined_value = 0.5,
  na.rm = FALSE)

Arguments

true_labels

Matrix of true labels, columns corresponding to labels and rows to instances.

predictions

Matrix of probabilities predicted by a classifier.

...

Additional parameters to be passed to the ranking function.

undefined_value

A default value for the cases when macro-averaged and example-averaged AUC encounter undefined (not computable) values, e.g. 0, 0.5, or NA.

na.rm

Logical specifying whether to ignore undefined values when undefined_value is set to NA.

Details

Available metrics in this category

• average_precision: Example and ranking based average precision (how many steps have to be made in the ranking to reach a certain relevant label, averaged by instance)

• coverage: Example and ranking based coverage (how many steps have to be made in the ranking to cover all the relevant labels, averaged by instance)

• example_auc: Example based Area Under the Curve ROC (averaged by instance)

• macro_auc: Label and ranking based Area Under the Curve ROC (macro-averaged by label)

• micro_auc: Label and ranking based Area Under the Curve ROC (micro-averaged)

• one_error: Example and ranking based one-error (how many times the top-ranked label is not a relevant label, averaged by instance)

• ranking_loss: Example and ranking based ranking-loss (how many times a non-relevant label is ranked above a relevant one, evaluated for all label pairs and averaged by instance)

Breaking ties in rankings

The additional ties_method parameter for the ranking function is passed to R's own rank. It accepts the following values:

• "average"

• "first"

• "last"

• "random"

• "max"

• "min"

See rank for information on the effect of each parameter. The default behavior in mldr corresponds to value "last", since this is the behavior of the ranking method in MULAN, in order to facilitate fair comparisons among classifiers over both platforms.

Value

Atomical numeric vector specifying the resulting performance metric value.

See Also

mldr_evaluate, mldr_to_labels

Other evaluation metrics: Averaged metrics, Basic metrics

Examples

true_labels <- matrix(c(
1,1,1,
0,0,0,
1,0,0,
1,1,1,
0,0,0,
1,0,0
), ncol = 3, byrow = TRUE)
predicted_probs <- matrix(c(
.6,.5,.9,
.0,.1,.2,
.8,.3,.2,
.7,.9,.1,
.7,.3,.2,
.1,.8,.3
), ncol = 3, byrow = TRUE)

# by default, labels with same ranking are assigned ascending rankings
# in the order they are encountered
coverage(true_labels, predicted_probs)
# in the following, labels with same ranking will receive the same,
# averaged ranking
average_precision(true_labels, predicted_probs, ties_method = "average")

# the following will treat all undefined values as 0 (counting them
# for the average)
example_auc(true_labels, predicted_probs, undefined_value = 0)
# the following will ignore undefined values (not counting them for
# the average)
example_auc(true_labels, predicted_probs, undefined_value = NA, na.rm = TRUE)