matchzoo.preprocessors.units.vocabulary
¶
Module Contents¶
Classes¶
Vocabulary class. |
-
class
matchzoo.preprocessors.units.vocabulary.
Vocabulary
(pad_value: str = '<PAD>', oov_value: str = '<OOV>')¶ Bases:
matchzoo.preprocessors.units.stateful_unit.StatefulUnit
Vocabulary class.
- Parameters
pad_value – The string value for the padding position.
oov_value – The string value for the out-of-vocabulary terms.
Examples
>>> vocab = Vocabulary(pad_value='[PAD]', oov_value='[OOV]') >>> vocab.fit(['A', 'B', 'C', 'D', 'E']) >>> term_index = vocab.state['term_index'] >>> term_index {'[PAD]': 0, '[OOV]': 1, 'D': 2, 'A': 3, 'B': 4, 'C': 5, 'E': 6} >>> index_term = vocab.state['index_term'] >>> index_term {0: '[PAD]', 1: '[OOV]', 2: 'D', 3: 'A', 4: 'B', 5: 'C', 6: 'E'}
>>> term_index['out-of-vocabulary-term'] 1 >>> index_term[0] '[PAD]' >>> index_term[42] Traceback (most recent call last): ... KeyError: 42 >>> a_index = term_index['A'] >>> c_index = term_index['C'] >>> vocab.transform(['C', 'A', 'C']) == [c_index, a_index, c_index] True >>> vocab.transform(['C', 'A', '[OOV]']) == [c_index, a_index, 1] True >>> indices = vocab.transform(list('ABCDDZZZ')) >>> ' '.join(vocab.state['index_term'][i] for i in indices) 'A B C D D [OOV] [OOV] [OOV]'
-
class
TermIndex
¶ Bases:
dict
Map term to index.
-
__missing__
(self, key)¶ Map out-of-vocabulary terms to index 1.
-
-
transform
(self, input_: list) → list¶ Transform a list of tokens to corresponding indices.