uk.co.cogitolearning.cogpar.Tokenizer Class Reference

A class for reading an input string and separating it into tokens that can be fed into Parser. More...

Collaboration diagram for uk.co.cogitolearning.cogpar.Tokenizer:

Classes
class	TokenInfo
	Internal class holding the information about a token type. More...

Public Member Functions
	Tokenizer ()
	Default constructor.
void	add (String regex, int token)
	Add a regular expression and a token id to the internal list of recognized tokens.
void	tokenize (String str)
	Tokenize an input string.
LinkedList< Token >	getTokens ()
	Get the tokens generated in the last call to tokenize.

Static Public Member Functions
static Tokenizer	getExpressionTokenizer ()
	A static method that returns a tokenizer for mathematical expressions.

Static Private Member Functions
static Tokenizer	createExpressionTokenizer ()
	A static method that actually creates a tokenizer for mathematical expressions.

Private Attributes
LinkedList< TokenInfo >	tokenInfos
	a list of TokenInfo objects
LinkedList< Token >	tokens
	the list of tokens produced when tokenizing the input

Static Private Attributes
static Tokenizer	expressionTokenizer = null
	a tokenizer that can handle mathematical expressions

Detailed Description

A class for reading an input string and separating it into tokens that can be fed into Parser.

The user can add regular expressions that will be matched against the front of the string. Regular expressions should not contain beginning-of-string or end-of-string anchors or any capturing groups as these will be added by the tokenizer itslef.

Definition at line 40 of file Tokenizer.java.

Member Function Documentation

void uk.co.cogitolearning.cogpar.Tokenizer.add	(	String	regex,
		int	token
	)

Add a regular expression and a token id to the internal list of recognized tokens.

Parameters

regex	the regular expression to match against
token	the token id that the regular expression is linked to

Definition at line 125 of file Tokenizer.java.

static Tokenizer uk.co.cogitolearning.cogpar.Tokenizer.createExpressionTokenizer ( )

staticprivate

A static method that actually creates a tokenizer for mathematical expressions.

Returns

a tokenizer that can handle mathematical expressions

Definition at line 101 of file Tokenizer.java.

static Tokenizer uk.co.cogitolearning.cogpar.Tokenizer.getExpressionTokenizer ( )

static

A static method that returns a tokenizer for mathematical expressions.

Returns

a tokenizer that can handle mathematical expressions

Definition at line 90 of file Tokenizer.java.

LinkedList<Token> uk.co.cogitolearning.cogpar.Tokenizer.getTokens

(

)

Get the tokens generated in the last call to tokenize.

Returns

a list of tokens to be fed to Parser

Definition at line 169 of file Tokenizer.java.

void uk.co.cogitolearning.cogpar.Tokenizer.tokenize

(

String

str

)

Tokenize an input string.

The reult of tokenizing can be accessed via getTokens

Parameters

str	the string to tokenize

Definition at line 137 of file Tokenizer.java.

Member Data Documentation

LinkedList<TokenInfo> uk.co.cogitolearning.cogpar.Tokenizer.tokenInfos

private

a list of TokenInfo objects

Each token type corresponds to one entry in the list

Definition at line 68 of file Tokenizer.java.

---

The documentation for this class was generated from the following file:

• Tokenizer.java