matchzoo.preprocessors.units.tokenize

Module Contents

class matchzoo.preprocessors.units.tokenize.Tokenize

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for text tokenization.

transform(self, input_:str)

Process input data from raw terms to list of tokens.

Parameters:input – raw textual input. Return tokens:tokenized tokens as a list.