vetiver_pin_metrics {vetiver} | R Documentation |
Update model metrics over time for monitoring
Description
These three functions can be used for model monitoring (such as in a monitoring dashboard):
-
vetiver_compute_metrics()
computes metrics (such as accuracy for a classification model or RMSE for a regression model) at a chosen time aggregationperiod
-
vetiver_pin_metrics()
updates an existing pin storing model metrics over time -
vetiver_plot_metrics()
creates a plot of metrics over time
Usage
vetiver_pin_metrics(
board,
df_metrics,
metrics_pin_name,
.index = .index,
overwrite = FALSE,
type = NULL,
...
)
Arguments
board |
A pin board, created by |
df_metrics |
A tidy dataframe of metrics over time, such as created by
|
metrics_pin_name |
Pin name for where the metrics are stored (as
opposed to where the model object is stored with |
.index |
The variable in |
overwrite |
If |
type |
File type used to save metrics to disk. With the default |
... |
Additional arguments passed on to methods for a specific board. |
Details
Sometimes when you monitor a model at a given time aggregation, you
may end up with dates in your new metrics (like new_metrics
in the example)
that are the same as dates in your existing aggregated metrics (like
original_metrics
in the example). This can happen if you need to re-run a
monitoring report because something failed. With overwrite = FALSE
(the
default), vetiver_pin_metrics()
will error when there are overlapping
dates. With overwrite = TRUE
, vetiver_pin_metrics()
will replace such
metrics with the new values. You probably want FALSE
for interactive use
and TRUE
for dashboards or reports that run on a schedule.
You can initially create your pin with type = "arrow"
or the default
(type = "rds"
). vetiver_pin_metrics()
will update the pin using the
same type
by default.
Value
A dataframe of metrics.
See Also
vetiver_compute_metrics()
, vetiver_plot_metrics()
Examples
library(dplyr)
library(parsnip)
data(Chicago, package = "modeldata")
Chicago <- Chicago %>% select(ridership, date, all_of(stations))
training_data <- Chicago %>% filter(date < "2009-01-01")
testing_data <- Chicago %>% filter(date >= "2009-01-01", date < "2011-01-01")
monitoring <- Chicago %>% filter(date >= "2011-01-01", date < "2012-12-31")
lm_fit <- linear_reg() %>% fit(ridership ~ ., data = training_data)
library(pins)
b <- board_temp()
## before starting monitoring, initiate the metrics and pin
## (for example, with the testing data):
original_metrics <-
augment(lm_fit, new_data = testing_data) %>%
vetiver_compute_metrics(date, "week", ridership, .pred, every = 4L)
pin_write(b, original_metrics, "lm_fit_metrics", type = "arrow")
## to continue monitoring with new data, compute metrics and update pin:
new_metrics <-
augment(lm_fit, new_data = monitoring) %>%
vetiver_compute_metrics(date, "week", ridership, .pred, every = 4L)
vetiver_pin_metrics(b, new_metrics, "lm_fit_metrics")