Package edu.udo.cs.wvtool.generic.tokenizer

Provides classes and interfaces for tokenization.

See:
Description

Interface Summary

WVTTokenizer

Interface, which represents a mechanism to convert a stream of characters to a stream of tokens, deleting all seperators.

Class Summary

NGramTokenizer	Creates tokens by creating ngrams of the tokens received from an inner tokenizer.
SimpleTokenizer	This class implements a simple tokenizer.

Package edu.udo.cs.wvtool.generic.tokenizer Description

Provides classes and interfaces for tokenization.

Package Specification