pre_tokenizer {tok}

Generic class for tokenizers

Description

Generic class for tokenizers

Generic class for tokenizers

Public fields

.pre_tokenizer

Internal pointer to tokenizer object

Methods

Public methods

• pre_tokenizer$new()

• pre_tokenizer$clone()

Method new()

Initializes a tokenizer

Usage

pre_tokenizer$new(pre_tokenizer)

Arguments

pre_tokenizer

a raw pointer to a tokenizer

Method clone()

The objects of this class are cloneable with this method.

Usage

pre_tokenizer$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other pre_tokenizer: pre_tokenizer_byte_level, pre_tokenizer_whitespace