Class TokenizerPreNgram
- java.lang.Object
- 
- org.apache.sysds.runtime.transform.tokenize.TokenizerPreNgram
 
- 
- All Implemented Interfaces:
- Serializable,- TokenizerPre
 
 public class TokenizerPreNgram extends Object implements TokenizerPre - See Also:
- Serialized Form
 
- 
- 
Constructor SummaryConstructors Constructor Description TokenizerPreNgram(List<Integer> idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)
 - 
Method SummaryAll Methods Instance Methods Concrete Methods Modifier and Type Method Description List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens>tokenizePre(FrameBlock in)List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>wordTokenListToNgrams(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokens)List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>wordTokenToNgrams(org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token wordTokens)
 
- 
- 
- 
Method Detail- 
wordTokenToNgramspublic List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokenToNgrams(org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token wordTokens) 
 - 
wordTokenListToNgramspublic List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokenListToNgrams(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokens) 
 - 
tokenizePrepublic List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tokenizePre(FrameBlock in) - Specified by:
- tokenizePrein interface- TokenizerPre
 
 
- 
 
-