decoder_byte_level {tok}

Byte level decoder

Description

Byte level decoder

Byte level decoder

Details

This decoder is to be used with the pre_tokenizer_byte_level.

Super class

tok::tok_decoder -> tok_decoder_byte_level

Methods

Public methods

• decoder_byte_level$new()

• decoder_byte_level$clone()

Method new()

Initializes a byte level decoder

Usage

decoder_byte_level$new()

Method clone()

The objects of this class are cloneable with this method.

Usage

decoder_byte_level$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other decoders: tok_decoder