Uses of Class
org.apache.lucene.analysis.Tokenizer

Packages that use Tokenizer
Package
Description
Text analysis.
Fast, general-purpose grammar-based tokenizers.
Analyzer for Simplified Chinese, which indexes words.
Basic, general-purpose analysis components.
Fast, general-purpose URLs and email addresses tokenizers.
Tokenizer that breaks text into words with the Unicode Text Segmentation algorithm.
Analyzer for Japanese.
Analyzer for Korean.
Character n-gram tokenizers and filters.
Analysis components for path-like strings such as filenames.
Set of components for pattern-based (regex) analysis.
Fast, general-purpose grammar-based tokenizer StandardTokenizer implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.
Analyzer for Thai.
Utility functions for text analysis.
Tokenizer that is aware of Wikipedia syntax.