as_tokens {mclm}

Coerce object to class tokens

Description

This function coerces a character object or another object that can be coerced to a character into an object of class tokens.

Usage

as_tokens(x, ...)

Arguments

x

Object to coerce.

...

Additional arguments (not implemented).

Value

An object of class tokens.

Examples

toy_corpus <- "Once upon a time there was a tiny toy corpus.
It consisted of three sentences. And it lived happily ever after."

tks <- tokenize(toy_corpus)
print(tks, n = 1000)

tks[3:12]
print(as_tokens(tks[3:12]), n = 1000)
as_tokens(tail(tks))