nnf_gelu {torch}

Gelu

Description

Gelu

Usage

nnf_gelu(input, approximate = "none")

Arguments

input

(N,*) tensor, where * means, any number of additional dimensions

approximate

By default it's none, and applies element-wise x*pnorm(x), if 'tanh', then GELU is estimated. See GELU for more info.

gelu(input) -> Tensor

Applies element-wise the function GELU(x) = x * \Phi(x)

where \Phi(x) is the Cumulative Distribution Function for Gaussian Distribution.

See Gaussian Error Linear Units (GELUs).