tok-package {tok}

tok: Fast Text Tokenization

Description

Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm https://huggingface.co/docs/tokenizers/index. It's extremely fast for both training new vocabularies and tokenizing texts.

Author(s)

Maintainer: Daniel Falbel [e-mail]

Other contributors:

• Posit [copyright holder]

See Also

Useful links:

• https://github.com/mlverse/tok

• Report bugs at https://github.com/mlverse/tok/issues