tok_model {tok}

Generic class for tokenization models

Description

Generic class for tokenization models

Generic class for tokenization models

Public fields

.model

stores the pointer to the model. internal

Methods

Public methods

• tok_model$new()

• tok_model$clone()

Method new()

Initializes a genric abstract tokenizer model

Usage

tok_model$new(model)

Arguments

model

Pointer to a tokenization model

Method clone()

The objects of this class are cloneable with this method.

Usage

tok_model$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other model: model_bpe, model_unigram, model_wordpiece