Package org.apache.nutch.scoring.similarity.util

## Class LuceneTokenizer

- java.lang.Object
- - org.apache.nutch.scoring.similarity.util.LuceneTokenizer

-     public class LuceneTokenizer
      extends Object

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class | Description
    --- | --- | ---
    `static class` | `LuceneTokenizer.TokenizerType` |

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `LuceneTokenizer​(String content, LuceneTokenizer.TokenizerType tokenizer, boolean useStopFilter, LuceneAnalyzerUtil.StemFilterType stemFilterType)` | Creates a tokenizer based on param values
    `LuceneTokenizer​(String content, LuceneTokenizer.TokenizerType tokenizer, List<String> stopWords, boolean addToDefault, LuceneAnalyzerUtil.StemFilterType stemFilterType)` | Creates a tokenizer based on param values
    `LuceneTokenizer​(String content, LuceneTokenizer.TokenizerType tokenizer, LuceneAnalyzerUtil.StemFilterType stemFilterType, int mingram, int maxgram)` | Creates a tokenizer for the ngram model based on param values

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `org.apache.lucene.analysis.TokenStream` | `getTokenStream()` | get the tokenStream created by `Tokenizer`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### LuceneTokenizer

          public LuceneTokenizer​(String content,
                                 LuceneTokenizer.TokenizerType tokenizer,
                                 boolean useStopFilter,
                                 LuceneAnalyzerUtil.StemFilterType stemFilterType)

      Creates a tokenizer based on param values
      Parameters:
        `content` - - The text to tokenize
        `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
        `useStopFilter` - - if set to true the token stream will be filtered using default Lucene stopset
        `stemFilterType` - a preferred `LuceneAnalyzerUtil.StemFilterType` to use. Can be one of `LuceneAnalyzerUtil.StemFilterType.PORTERSTEM_FILTER`, `LuceneAnalyzerUtil.StemFilterType.ENGLISHMINIMALSTEM_FILTER`, or `LuceneAnalyzerUtil.StemFilterType.NONE`

    - #### LuceneTokenizer

          public LuceneTokenizer​(String content,
                                 LuceneTokenizer.TokenizerType tokenizer,
                                 List<String> stopWords,
                                 boolean addToDefault,
                                 LuceneAnalyzerUtil.StemFilterType stemFilterType)

      Creates a tokenizer based on param values
      Parameters:
        `content` - - The text to tokenize
        `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
        `stopWords` - - Provide a set of user defined stop words
        `addToDefault` - - If set to true, the stopSet words will be added to the Lucene default stop set. If false, then only the user provided words will be used as the stop set
        `stemFilterType` - a preferred `LuceneAnalyzerUtil.StemFilterType` to use. Can be one of `LuceneAnalyzerUtil.StemFilterType.PORTERSTEM_FILTER`, `LuceneAnalyzerUtil.StemFilterType.ENGLISHMINIMALSTEM_FILTER`, or `LuceneAnalyzerUtil.StemFilterType.NONE`

    - #### LuceneTokenizer

          public LuceneTokenizer​(String content,
                                 LuceneTokenizer.TokenizerType tokenizer,
                                 LuceneAnalyzerUtil.StemFilterType stemFilterType,
                                 int mingram,
                                 int maxgram)

      Creates a tokenizer for the ngram model based on param values
      Parameters:
        `content` - - The text to tokenize
        `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
        `stemFilterType` - - Type of stemming to perform
        `mingram` - - Value of mingram for tokenizing
        `maxgram` - - Value of maxgram for tokenizing

  - ### Method Detail

    - #### getTokenStream

          public org.apache.lucene.analysis.TokenStream getTokenStream()

      get the tokenStream created by `Tokenizer`
      Returns:
        The `TokenStream`