matchzoo.preprocessors.units

Package Contents

class matchzoo.preprocessors.units.Unit

Process unit do not persive state (i.e. do not need fit).

transform(self, input_:typing.Any)

Abstract base method, need to be implemented in subclass.

class matchzoo.preprocessors.units.DigitRemoval

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit to remove digits.

transform(self, input_:list)

Remove digits from list of tokens.

Parameters:input – list of tokens to be filtered.
Return tokens:tokens of tokens without digits.
class matchzoo.preprocessors.units.FrequencyFilter(low:float=0, high:float=float('inf'), mode:str='df')

Bases: matchzoo.preprocessors.units.stateful_unit.StatefulUnit

Frequency filter unit.

Parameters:
  • low – Lower bound, inclusive.
  • high – Upper bound, exclusive.
  • mode – One of tf (term frequency), df (document frequency), and idf (inverse document frequency).
Examples::
>>> import matchzoo as mz
To filter based on term frequency (tf):
>>> tf_filter = mz.preprocessors.units.FrequencyFilter(
...     low=2, mode='tf')
>>> tf_filter.fit([['A', 'B', 'B'], ['C', 'C', 'C']])
>>> tf_filter.transform(['A', 'B', 'C'])
['B', 'C']
To filter based on document frequency (df):
>>> tf_filter = mz.preprocessors.units.FrequencyFilter(
...     low=2, mode='df')
>>> tf_filter.fit([['A', 'B'], ['B', 'C']])
>>> tf_filter.transform(['A', 'B', 'C'])
['B']
To filter based on inverse document frequency (idf):
>>> idf_filter = mz.preprocessors.units.FrequencyFilter(
...     low=1.2, mode='idf')
>>> idf_filter.fit([['A', 'B'], ['B', 'C', 'D']])
>>> idf_filter.transform(['A', 'B', 'C'])
['A', 'C']
fit(self, list_of_tokens:typing.List[typing.List[str]])

Fit list_of_tokens by calculating mode states.

transform(self, input_:list)

Transform a list of tokens by filtering out unwanted words.

classmethod _tf(cls, list_of_tokens:list)
classmethod _df(cls, list_of_tokens:list)
classmethod _idf(cls, list_of_tokens:list)
class matchzoo.preprocessors.units.Lemmatization

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for token lemmatization.

transform(self, input_:list)

Lemmatization a sequence of tokens.

Parameters:input – list of tokens to be lemmatized.
Return tokens:list of lemmatizd tokens.
class matchzoo.preprocessors.units.Lowercase

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for text lower case.

transform(self, input_:list)

Convert list of tokens to lower case.

Parameters:input – list of tokens.
Return tokens:lower-cased list of tokens.
class matchzoo.preprocessors.units.MatchingHistogram(bin_size:int=30, embedding_matrix=None, normalize=True, mode:str='LCH')

Bases: matchzoo.preprocessors.units.unit.Unit

MatchingHistogramUnit Class.

Parameters:
  • bin_size – The number of bins of the matching histogram.
  • embedding_matrix – The word embedding matrix applied to calculate the matching histogram.
  • normalize – Boolean, normalize the embedding or not.
  • mode – The type of the historgram, it should be one of ‘CH’, ‘NG’, or ‘LCH’.

Examples

>>> embedding_matrix = np.array([[1.0, -1.0], [1.0, 2.0], [1.0, 3.0]])
>>> text_left = [0, 1]
>>> text_right = [1, 2]
>>> histogram = MatchingHistogram(3, embedding_matrix, True, 'CH')
>>> histogram.transform([text_left, text_right])
[[3.0, 1.0, 1.0], [1.0, 2.0, 2.0]]
_normalize_embedding(self)

Normalize the embedding matrix.

transform(self, input_:list)

Transform the input text.

class matchzoo.preprocessors.units.NgramLetter(ngram:int=3, reduce_dim:bool=True)

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for n-letter generation.

Triletter is used in DSSMModel. This processor is expected to execute before Vocab has been created.

Examples

>>> triletter = NgramLetter()
>>> rv = triletter.transform(['hello', 'word'])
>>> len(rv)
9
>>> rv
['#he', 'hel', 'ell', 'llo', 'lo#', '#wo', 'wor', 'ord', 'rd#']
>>> triletter = NgramLetter(reduce_dim=False)
>>> rv = triletter.transform(['hello', 'word'])
>>> len(rv)
2
>>> rv
[['#he', 'hel', 'ell', 'llo', 'lo#'], ['#wo', 'wor', 'ord', 'rd#']]
transform(self, input_:list)

Transform token into tri-letter.

For example, word should be represented as #wo, wor, ord and rd#.

Parameters:input – list of tokens to be transformed.
Return n_letters:
 generated n_letters.
class matchzoo.preprocessors.units.PuncRemoval

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for remove punctuations.

_MATCH_PUNC
transform(self, input_:list)

Remove punctuations from list of tokens.

Parameters:input – list of toekns.
Return rv:tokens without punctuation.
class matchzoo.preprocessors.units.StatefulUnit

Bases: matchzoo.preprocessors.units.unit.Unit

Unit with inner state.

Usually need to be fit before transforming. All information gathered in the fit phrase will be stored into its context.

state

Get current context. Same as unit.context.

Deprecated since v2.2.0, and will be removed in the future. Used unit.context instead.

context

Get current context. Same as unit.state.

fit(self, input_:typing.Any)

Abstract base method, need to be implemented in subclass.

class matchzoo.preprocessors.units.Stemming(stemmer='porter')

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for token stemming.

Parameters:stemmer – stemmer to use, porter or lancaster.
transform(self, input_:list)

Reducing inflected words to their word stem, base or root form.

Parameters:input – list of string to be stemmed.
class matchzoo.preprocessors.units.StopRemoval(lang:str='english')

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit to remove stop words.

Example

>>> unit = StopRemoval()
>>> unit.transform(['a', 'the', 'test'])
['test']
>>> type(unit.stopwords)
<class 'list'>
stopwords

Get stopwords based on language.

Params lang:language code.
Returns:list of stop words.
transform(self, input_:list)

Remove stopwords from list of tokenized tokens.

Parameters:
  • input – list of tokenized tokens.
  • lang – language code for stopwords.
Return tokens:

list of tokenized tokens without stopwords.
class matchzoo.preprocessors.units.Tokenize

Bases: matchzoo.preprocessors.units.unit.Unit

Process unit for text tokenization.

transform(self, input_:str)

Process input data from raw terms to list of tokens.

Parameters:input – raw textual input.
Return tokens:tokenized tokens as a list.
class matchzoo.preprocessors.units.Vocabulary(pad_value:str='<PAD>', oov_value:str='<OOV>')

Bases: matchzoo.preprocessors.units.stateful_unit.StatefulUnit

Vocabulary class.

Parameters:
  • pad_value – The string value for the padding position.
  • oov_value – The string value for the out-of-vocabulary terms.

Examples

>>> vocab = Vocabulary(pad_value='[PAD]', oov_value='[OOV]')
>>> vocab.fit(['A', 'B', 'C', 'D', 'E'])
>>> term_index = vocab.state['term_index']
>>> term_index  # doctest: +SKIP
{'[PAD]': 0, '[OOV]': 1, 'D': 2, 'A': 3, 'B': 4, 'C': 5, 'E': 6}
>>> index_term = vocab.state['index_term']
>>> index_term  # doctest: +SKIP
{0: '[PAD]', 1: '[OOV]', 2: 'D', 3: 'A', 4: 'B', 5: 'C', 6: 'E'}

>>> term_index['out-of-vocabulary-term']
1
>>> index_term[0]
'[PAD]'
>>> index_term[42]
Traceback (most recent call last):
    ...
KeyError: 42
>>> a_index = term_index['A']
>>> c_index = term_index['C']
>>> vocab.transform(['C', 'A', 'C']) == [c_index, a_index, c_index]
True
>>> vocab.transform(['C', 'A', '[OOV]']) == [c_index, a_index, 1]
True
>>> indices = vocab.transform(list('ABCDDZZZ'))
>>> ' '.join(vocab.state['index_term'][i] for i in indices)
'A B C D D [OOV] [OOV] [OOV]'
class TermIndex

Bases: dict

Map term to index.

__missing__(self, key)

Map out-of-vocabulary terms to index 1.

fit(self, tokens:list)

Build a TermIndex and a IndexTerm.

transform(self, input_:list)

Transform a list of tokens to corresponding indices.

class matchzoo.preprocessors.units.WordHashing(term_index:dict)

Bases: matchzoo.preprocessors.units.unit.Unit

Word-hashing layer for DSSM-based models.

The input of WordHashingUnit should be a list of word sub-letter list extracted from one document. The output of is the word-hashing representation of this document.

NgramLetterUnit and VocabularyUnit are two essential prerequisite of WordHashingUnit.

Examples

>>> letters = [['#te', 'tes','est', 'st#'], ['oov']]
>>> word_hashing = WordHashing(
...     term_index={
...      '_PAD': 0, 'OOV': 1, 'st#': 2, '#te': 3, 'est': 4, 'tes': 5
...      })
>>> hashing = word_hashing.transform(letters)
>>> hashing[0]
[0.0, 0.0, 1.0, 1.0, 1.0, 1.0]
>>> hashing[1]
[0.0, 1.0, 0.0, 0.0, 0.0, 0.0]
transform(self, input_:list)

Transform list of letters into word hashing layer.

Parameters:input – list of tri_letters generated by NgramLetterUnit.
Returns:Word hashing representation of tri-letters.
class matchzoo.preprocessors.units.CharacterIndex(char_index:dict)

Bases: matchzoo.preprocessors.units.unit.Unit

CharacterIndexUnit for DIIN model.

The input of :class:’CharacterIndexUnit’ should be a list of word character list extracted from a text. The output is the character index representation of this text.

NgramLetterUnit and VocabularyUnit are two essential prerequisite of CharacterIndexUnit.

Examples

>>> input_ = [['#', 'a', '#'],['#', 'o', 'n', 'e', '#']]
>>> character_index = CharacterIndex(
...     char_index={
...      '<PAD>': 0, '<OOV>': 1, 'a': 2, 'n': 3, 'e':4, '#':5})
>>> index = character_index.transform(input_)
>>> index
[[5, 2, 5], [5, 1, 3, 4, 5]]
transform(self, input_:list)

Transform list of characters to corresponding indices.

Parameters:input – list of characters generated by :class:’NgramLetterUnit’.
Returns:character index representation of a text.
class matchzoo.preprocessors.units.WordExactMatch(match:str, to_match:str)

Bases: matchzoo.preprocessors.units.unit.Unit

WordExactUnit Class.

Process unit to get a binary match list of two word index lists. The word index list is the word representation of a text.

Examples

>>> import pandas
>>> input_ = pandas.DataFrame({
...  'text_left':[[1, 2, 3],[4, 5, 7, 9]],
...  'text_right':[[5, 3, 2, 7],[2, 3, 5]]}
... )
>>> left_word_exact_match = WordExactMatch(
...     match='text_left', to_match='text_right'
... )
>>> left_out = input_.apply(left_word_exact_match.transform, axis=1)
>>> left_out[0]
[0, 1, 1]
>>> left_out[1]
[0, 1, 0, 0]
>>> right_word_exact_match = WordExactMatch(
...     match='text_right', to_match='text_left'
... )
>>> right_out = input_.apply(right_word_exact_match.transform, axis=1)
>>> right_out[0]
[0, 1, 1, 0]
>>> right_out[1]
[0, 0, 1]
transform(self, input_)

Transform two word index lists into a binary match list.

Parameters:input – a dataframe include ‘match’ column and ‘to_match’ column.
Returns:a binary match result list of two word index lists.
class matchzoo.preprocessors.units.TruncatedLength(text_length:int, truncate_mode:str='pre')

Bases: matchzoo.preprocessors.units.unit.Unit

TruncatedLengthUnit Class.

Process unit to truncate the text that exceeds the set length.

Examples

>>> from matchzoo.preprocessors.units import TruncatedLength
>>> truncatedlen = TruncatedLength(3)
>>> truncatedlen.transform(list(range(1, 6))) == [3, 4, 5]
True
>>> truncatedlen.transform(list(range(2))) == [0, 1]
True
transform(self, input_:list)

Truncate the text that exceeds the specified maximum length.

Parameters:input – list of tokenized tokens.
Return tokens:list of tokenized tokens in fixed length if its origin length larger than text_length.
matchzoo.preprocessors.units.list_available() → list