tokenize_sentences {sbo}

Sentence tokenizer

Description

Get sentence tokens from text

Usage

tokenize_sentences(input, EOS = ".?!:;")

Arguments

input

a character vector.

EOS

a length one character vector listing all (single character) end-of-sentence tokens.

Value

a character vector, each entry of which corresponds to a single sentence.

Author(s)

Valerio Gherardi

Examples

tokenize_sentences("Hi there! I'm using `sbo`.")