tokenizer {sudachir}

Sudachi tokenizer

Description

Sudachi tokenizer

Usage

tokenizer(x, mode, instance = NULL)

Arguments

x

Input text vectors

mode

Select split mode (A, B, C)

instance

This is optional if you already have an instance of ⁠<sudachipy.tokenizer.Tokenizer>⁠ Giving them a predefined instance will speed up their execution.

Examples

## Not run: 
tokenizer("Tokyo, Japan", mode = "A")

## End(Not run)