processor_byte_level {tok}R Documentation

Byte Level post processor

Description

Byte Level post processor

Byte Level post processor

Details

This post-processor takes care of trimming the offsets. By default, the ByteLevel BPE might include whitespaces in the produced tokens. If you don’t want the offsets to include these whitespaces, then this PostProcessor must be used.

Super class

tok::tok_processor -> tok_processor_byte_level

Methods

Public methods


Method new()

Initializes the byte level post processor

Usage
processor_byte_level$new(trim_offsets = TRUE)
Arguments
trim_offsets

Whether to trim the whitespaces from the produced offsets.


Method clone()

The objects of this class are cloneable with this method.

Usage
processor_byte_level$clone(deep = FALSE)
Arguments
deep

Whether to make a deep clone.

See Also

Other processors: tok_processor


[Package tok version 0.1.3 Index]