processor_byte_level {tok} | R Documentation |
Byte Level post processor
Description
Byte Level post processor
Byte Level post processor
Details
This post-processor takes care of trimming the offsets. By default, the ByteLevel BPE might include whitespaces in the produced tokens. If you don’t want the offsets to include these whitespaces, then this PostProcessor must be used.
Super class
tok::tok_processor
-> tok_processor_byte_level
Methods
Public methods
Method new()
Initializes the byte level post processor
Usage
processor_byte_level$new(trim_offsets = TRUE)
Arguments
trim_offsets
Whether to trim the whitespaces from the produced offsets.
Method clone()
The objects of this class are cloneable with this method.
Usage
processor_byte_level$clone(deep = FALSE)
Arguments
deep
Whether to make a deep clone.
See Also
Other processors:
tok_processor
[Package tok version 0.1.3 Index]