split_token {textshape}

Split Tokens

Description

Split tokens.

Usage

split_token(x, ...)

## Default S3 method:
split_token(x, lower = TRUE, ...)

## S3 method for class 'data.frame'
split_token(x, text.var = TRUE, lower = TRUE, ...)

Arguments

x

A data.frame or character vector with tokens.

lower

logical. If TRUE the words are converted to lower case.

text.var

The name of the text variable. If TRUE split_token tries to detect the text column with tokens.

...

Ignored.

Value

Returns a list of vectors of tokens or an expanded data.table with tokens split apart.

Examples

(x <- c(
    "Mr. Brown comes! He says hello. i give him coffee.",
    "I'll go at 5 p. m. eastern time.  Or somewhere in between!",
    "go there"
))
split_token(x)
split_token(x, lower=FALSE)

data(DATA)
split_token(DATA)
split_token(DATA, lower=FALSE)

## Larger data set
split_token(hamlet)