split_text {TheOpenAIR}

Split Text into Chunks

Description

This function splits a text string into a vector of strings with a specified number of tokens each.

Usage

split_text(text, N)

Arguments

text

A character vector containing the text to be split.

N

An integer specifying the number of tokens per chunk.

Value

A character vector containing the chunks of text with N tokens each.

Examples

large_text <- "This is an example of a large text string 
that will be split into chunks of N tokens each by our custom R function."
num_tokens_per_chunk <- 5
split_text(large_text, num_tokens_per_chunk)