schedule_decay_time {brulee} | R Documentation |
Change the learning rate over time
Description
Learning rate schedulers alter the learning rate to adjust as training
proceeds. In most cases, the learning rate decreases as epochs increase.
The schedule_*()
functions are individual schedulers and
set_learn_rate()
is a general interface.
Usage
schedule_decay_time(epoch, initial = 0.1, decay = 1)
schedule_decay_expo(epoch, initial = 0.1, decay = 1)
schedule_step(epoch, initial = 0.1, reduction = 1/2, steps = 5)
schedule_cyclic(epoch, initial = 0.001, largest = 0.1, step_size = 5)
set_learn_rate(epoch, learn_rate, type = "none", ...)
Arguments
epoch |
An integer for the number of training epochs (zero being the initial value), |
initial |
A positive numeric value for the starting learning rate. |
decay |
A positive numeric constant for decreasing the rate (see Details below). |
reduction |
A positive numeric constant stating the proportional decrease
in the learning rate occurring at every |
steps |
The number of epochs before the learning rate changes. |
largest |
The maximum learning rate in the cycle. |
step_size |
The half-length of a cycle. |
learn_rate |
A constant learning rate (when no scheduler is used), |
type |
A single character value for the type of scheduler. Possible values are: "decay_time", "decay_expo", "none", "cyclic", and "step". |
... |
Arguments to pass to the individual scheduler functions (e.g.
|
Details
The details for how the schedulers change the rates:
-
schedule_decay_time()
:rate(epoch) = initial/(1 + decay \times epoch)
-
schedule_decay_expo()
:rate(epoch) = initial\exp(-decay \times epoch)
-
schedule_step()
:rate(epoch) = initial \times reduction^{floor(epoch / steps)}
-
schedule_cyclic()
:cycle = floor( 1 + (epoch / 2 / step size) )
,x = abs( ( epoch / step size ) - ( 2 * cycle) + 1 )
, andrate(epoch) = initial + ( largest - initial ) * \max( 0, 1 - x)
Value
A numeric value for the updated learning rate.
See Also
Examples
library(ggplot2)
library(dplyr)
library(purrr)
iters <- 0:50
bind_rows(
tibble(epoch = iters, rate = map_dbl(iters, schedule_decay_time), type = "decay_time"),
tibble(epoch = iters, rate = map_dbl(iters, schedule_decay_expo), type = "decay_expo"),
tibble(epoch = iters, rate = map_dbl(iters, schedule_step), type = "step"),
tibble(epoch = iters, rate = map_dbl(iters, schedule_cyclic), type = "cyclic")
) %>%
ggplot(aes(epoch, rate)) +
geom_line() +
facet_wrap(~ type)
# ------------------------------------------------------------------------------
# Use with neural network