matchzoo.preprocessors.units.tokenize¶
Module Contents¶
Classes¶
Process unit for text tokenization. |
-
class
matchzoo.preprocessors.units.tokenize.Tokenize¶ Bases:
matchzoo.preprocessors.units.unit.UnitProcess unit for text tokenization.
-
transform(self, input_: str) → list¶ Process input data from raw terms to list of tokens.
- Parameters
input – raw textual input.
- Return tokens
tokenized tokens as a list.
-