Tokenizer
Package:
Summary:
Extract list of tokens from a text string
Groups:
Author:
Description:
This package can be used to extract list of tokens from a text string.
It can take a text string and split it in tokens separated by text strings or one or more characters.
It supports markers strings that define the beginning and end of a token that may have separator strings inside.
It also supports operators that may be used to combine multiple tokens by performing defined operations and returning the result value.
It can take a text string and split it in tokens separated by text strings or one or more characters.
It supports markers strings that define the beginning and end of a token that may have separator strings inside.
It also supports operators that may be used to combine multiple tokens by performing defined operations and returning the result value.
Comments are closed.