| magicTokenizer {MacBehaviour} | R Documentation |
magicTokenizer
Description
This function provides the number of tokens in a specified text, acting as a wrapper for an internal tokenizer function.
Usage
magicTokenizer(text)
Arguments
text |
A character string: the text for which the number of tokens is required. |
Value
Returns the number of tokens in the provided text.
[Package MacBehaviour version 1.2.4 Index]