Token Splitter node

The Token Splitter node splits a raw text string by first converting the text into BPE tokens, then splits these tokens into chunks and converts the tokens within a single chunk back into text.

On this page, you'll find the node parameters for the Token Splitter node, and links to more resources.

Node parameters

• Chunk Size: Enter the number of characters in each chunk.
• Chunk Overlap: Enter how much overlap to have between chunks.

Related resources

Refer to LangChain's token documentation (opens in a new tab) and LangChain's text splitter documentation (opens in a new tab) for more information about the service.