morphemepiece-package |
morphemepiece: Morpheme Tokenization |
load_lookup |
Load a morphemepiece lookup file |
load_or_retrieve_lookup |
Load a lookup file, or retrieve from cache |
load_or_retrieve_vocab |
Load a vocabulary file, or retrieve from cache |
load_vocab |
Load a vocabulary file |
morphemepiece_cache_dir |
Retrieve Directory for Morphemepiece Cache |
morphemepiece_tokenize |
Tokenize Sequence with Morpheme Pieces |
prepare_vocab |
Format a Token List as a Vocabulary |
set_morphemepiece_cache_dir |
Set a Cache Directory for Morphemepiece |