get_tokens {discoverableresearch}

Remove stopwords from text

Description

Removes stopwords from text in whichever language is specified.

Removes stop words from a text string (adapted from 'litsearchr' https://github.com/elizagrames/litsearchr/) and returns the remaining words as a vector of strings

Usage

get_tokens(text, language = "English")

get_tokens(text, language = "English")

Arguments

text

An input string

language

The language used to look up stop words (default is "English")

Value

Returns the input text with stopwords removed.

A vector of strings consisting of the non-stop words from the 'text' input

Examples

get_tokens("On the Origin of Species", language="English")
text <- "A methodology for systematic mapping in environmental sciences"
tokens <- get_tokens(text)
tokens;