merge_tokens {mclm}

Merge tokens objects

Description

tokens_merge() merges two tokens objects x and y into a larger tokens object. tokens_merge_all() merge all the arguments into one tokens object. The result is a concatenation of the tokens, in which the order of the items in the input is preserved.

Usage

tokens_merge(x, y)

tokens_merge_all(...)

Arguments

x, y

An object of class tokens

...

Objects of class tokens or a list with objects of class tokens.

Value

An object of class tokens.

Examples

(tks1 <- tokenize(c("This is a first sentence.")))
(tks2 <- tokenize(c("It is followed by a second one.")))
(tks3 <- tokenize(c("Then a third one follows.")))

tokens_merge(tks1, tks2)
tokens_merge_all(tks1, tks2, tks3)
tokens_merge_all(list(tks1, tks2, tks3))