comp_accu_freq {riskyr}

Compute accuracy metrics of current classification results.

Description

comp_accu_freq computes a list of current accuracy metrics from the 4 essential frequencies (hi, mi, fa, cr) that constitute the current confusion matrix and are contained in freq.

Usage

comp_accu_freq(hi = freq$hi, mi = freq$mi, fa = freq$fa, cr = freq$cr, w = 0.5)

Arguments

hi

The number of hits hi (or true positives).

mi

The number of misses mi (or false negatives).

fa

The number of false alarms fa (or false positives).

cr

The number of correct rejections cr (or true negatives).

w

The weighting parameter w (from 0 to 1) for computing weighted accuracy wacc. Default: w = .50 (i.e., yielding balanced accuracy bacc).

Details

Currently computed accuracy metrics include:

1. acc: Overall accuracy as the proportion (or probability) of correctly classifying cases or of dec_cor cases:

   acc = dec_cor/N = (hi + cr)/(hi + mi + fa + cr)

   Values range from 0 (no correct prediction) to 1 (perfect prediction).

2. wacc: Weighted accuracy, as a weighted average of the sensitivity sens (aka. hit rate HR, TPR, power or recall) and the the specificity spec (aka. TNR) in which sens is multiplied by a weighting parameter w (ranging from 0 to 1) and spec is multiplied by w's complement (1 - w):

   wacc = (w * sens) + ((1 - w) * spec)

   If w = .50, wacc becomes balanced accuracy bacc.

3. mcc: The Matthews correlation coefficient (with values ranging from -1 to +1):

   mcc = ((hi * cr) - (fa * mi)) / sqrt((hi + fa) * (hi + mi) * (cr + fa) * (cr + mi))

   A value of mcc = 0 implies random performance; mcc = 1 implies perfect performance.

   See Wikipedia: Matthews correlation coefficient for additional information.

4. f1s: The harmonic mean of the positive predictive value PPV (aka. precision) and the sensitivity sens (aka. hit rate HR, TPR, power or recall):

   f1s = 2 * (PPV * sens) / (PPV + sens)

   See Wikipedia: F1 score for additional information.

Notes:

• Accuracy metrics describe the correspondence of decisions (or predictions) to actual conditions (or truth).

  There are several possible interpretations of accuracy:

  1. as probabilities (i.e., acc being the proportion of correct classifications, or the ratio dec_cor/N),

  2. as frequencies (e.g., as classifying a population of N individuals into cases of dec_cor vs. dec_err),

  3. as correlations (e.g., see mcc in accu).

• Computing exact accuracy values based on probabilities (by comp_accu_prob) may differ from accuracy values computed from (possibly rounded) frequencies (by comp_accu_freq).

  When frequencies are rounded to integers (see the default of round = TRUE in comp_freq and comp_freq_prob) the accuracy metrics computed by comp_accu_freq correspond to these rounded values. Use comp_accu_prob to obtain exact accuracy metrics from probabilities.

Value

A list accu containing current accuracy metrics.

References

Consult Wikipedia: Confusion matrix for additional information.

See Also

accu for all accuracy metrics; comp_accu_prob computes exact accuracy metrics from probabilities; num for basic numeric parameters; freq for current frequency information; txt for current text settings; pal for current color settings; popu for a table of the current population.

Other metrics: accu, acc, comp_accu_prob(), comp_acc(), comp_err(), err

Other functions computing probabilities: comp_FDR(), comp_FOR(), comp_NPV(), comp_PPV(), comp_accu_prob(), comp_acc(), comp_comp_pair(), comp_complement(), comp_complete_prob_set(), comp_err(), comp_fart(), comp_mirt(), comp_ppod(), comp_prob_freq(), comp_prob(), comp_sens(), comp_spec()

Examples

comp_accu_freq()  # => accuracy metrics for freq of current scenario
comp_accu_freq(hi = 1, mi = 2, fa = 3, cr = 4)  # medium accuracy, but cr > hi

# Extreme cases:
comp_accu_freq(hi = 1, mi = 1, fa = 1, cr = 1)  # random performance
comp_accu_freq(hi = 0, mi = 0, fa = 1, cr = 1)  # random performance: wacc and f1s are NaN
comp_accu_freq(hi = 1, mi = 0, fa = 0, cr = 1)  # perfect accuracy/optimal performance
comp_accu_freq(hi = 0, mi = 1, fa = 1, cr = 0)  # zero accuracy/worst performance, but see f1s
comp_accu_freq(hi = 1, mi = 0, fa = 0, cr = 0)  # perfect accuracy, but see wacc and mcc

# Effects of w:
comp_accu_freq(hi = 3, mi = 2, fa = 1, cr = 4, w = 1/2)  # equal weights to sens and spec
comp_accu_freq(hi = 3, mi = 2, fa = 1, cr = 4, w = 2/3)  # more weight to sens
comp_accu_freq(hi = 3, mi = 2, fa = 1, cr = 4, w = 1/3)  # more weight to spec

## Contrasting comp_accu_freq and comp_accu_prob:
# (a) comp_accu_freq (based on rounded frequencies):
freq1 <- comp_freq(N = 10, prev = 1/3, sens = 2/3, spec = 3/4)   # => hi = 2, mi = 1, fa = 2, cr = 5
accu1 <- comp_accu_freq(freq1$hi, freq1$mi, freq1$fa, freq1$cr)  # => accu1 (based on rounded freq).
# accu1
#
# (b) comp_accu_prob (based on probabilities):
accu2 <- comp_accu_prob(prev = 1/3, sens = 2/3, spec = 3/4)      # => exact accu (based on prob).
# accu2
all.equal(accu1, accu2)  # => 4 differences!
#
# (c) comp_accu_freq (exact values, i.e., without rounding):
freq3 <- comp_freq(N = 10, prev = 1/3, sens = 2/3, spec = 3/4, round = FALSE)
accu3 <- comp_accu_freq(freq3$hi, freq3$mi, freq3$fa, freq3$cr)  # => accu3 (based on EXACT freq).
# accu3
all.equal(accu2, accu3)  # => TRUE (qed).