matchzoo.dataloader.callbacks
¶
Submodules¶
Package Contents¶
-
class
matchzoo.dataloader.callbacks.
LambdaCallback
(on_batch_data_pack=None, on_batch_unpacked=None)¶ Bases:
matchzoo.engine.base_callback.BaseCallback
LambdaCallback. Just a shorthand for creating a callback class.
See
matchzoo.engine.base_callback.BaseCallback
for more details.Example
>>> import matchzoo as mz >>> from matchzoo.dataloader.callbacks import LambdaCallback >>> data = mz.datasets.toy.load_data() >>> batch_func = lambda x: print(type(x)) >>> unpack_func = lambda x, y: print(type(x), type(y)) >>> callback = LambdaCallback(on_batch_data_pack=batch_func, ... on_batch_unpacked=unpack_func) >>> dataset = mz.dataloader.Dataset( ... data, callbacks=[callback]) >>> _ = dataset[0] <class 'matchzoo.data_pack.data_pack.DataPack'> <class 'dict'> <class 'numpy.ndarray'>
-
on_batch_data_pack
(self, data_pack)¶ on_batch_data_pack.
-
on_batch_unpacked
(self, x, y)¶ on_batch_unpacked.
-
-
class
matchzoo.dataloader.callbacks.
Histogram
(embedding_matrix: np.ndarray, bin_size: int = 30, hist_mode: str = 'CH')¶ Bases:
matchzoo.engine.base_callback.BaseCallback
Generate data with matching histogram.
Parameters: - embedding_matrix – The embedding matrix used to generator match histogram.
- bin_size – The number of bin size of the histogram.
- hist_mode – The mode of the
MatchingHistogramUnit
, one of CH, NH, and LCH.
-
on_batch_unpacked
(self, x, y)¶ Insert match_histogram to x.
-
class
matchzoo.dataloader.callbacks.
Ngram
(preprocessor: mz.preprocessors.BasicPreprocessor, mode: str = 'index')¶ Bases:
matchzoo.engine.base_callback.BaseCallback
Generate the character n-gram for data.
Parameters: - preprocessor – The fitted
BasePreprocessor
object, which contains the n-gram units information. - mode – It can be one of ‘index’, ‘onehot’, ‘sum’ or ‘aggregate’.
Example
>>> import matchzoo as mz >>> from matchzoo.dataloader.callbacks import Ngram >>> data = mz.datasets.toy.load_data() >>> preprocessor = mz.preprocessors.BasicPreprocessor(ngram_size=3) >>> data = preprocessor.fit_transform(data) >>> callback = Ngram(preprocessor=preprocessor, mode='index') >>> dataset = mz.dataloader.Dataset( ... data, callbacks=[callback]) >>> _ = dataset[0]
-
on_batch_unpacked
(self, x, y)¶ Insert ngram_left and ngram_right to x.
- preprocessor – The fitted
-
class
matchzoo.dataloader.callbacks.
BasicPadding
(fixed_length_left: int = None, fixed_length_right: int = None, pad_word_value: typing.Union[int, str] = 0, pad_word_mode: str = 'pre', with_ngram: bool = False, fixed_ngram_length: int = None, pad_ngram_value: typing.Union[int, str] = 0, pad_ngram_mode: str = 'pre')¶ Bases:
matchzoo.engine.base_callback.BaseCallback
Pad data for basic preprocessor.
Parameters: - fixed_length_left – Integer. If set, text_left will be padded to this length.
- fixed_length_right – Integer. If set, text_right will be padded to this length.
- pad_word_value – the value to fill text.
- pad_word_mode – String, pre or post: pad either before or after each sequence.
- with_ngram – Boolean. Whether to pad the n-grams.
- fixed_ngram_length – Integer. If set, each word will be padded to this length, or it will be set as the maximum length of words in current batch.
- pad_ngram_value – the value to fill empty n-grams.
- pad_ngram_mode – String, pre or post: pad either before of after each sequence.
-
on_batch_unpacked
(self, x: dict, y: np.ndarray)¶ Pad x[‘text_left’] and x[‘text_right].
-
class
matchzoo.dataloader.callbacks.
DRMMPadding
(fixed_length_left: int = None, fixed_length_right: int = None, pad_value: typing.Union[int, str] = 0, pad_mode: str = 'pre')¶ Bases:
matchzoo.engine.base_callback.BaseCallback
Pad data for DRMM Model.
Parameters: - fixed_length_left – Integer. If set, text_left and match_histogram will be padded to this length.
- fixed_length_right – Integer. If set, text_right will be padded to this length.
- pad_value – the value to fill text.
- pad_mode – String, pre or post: pad either before or after each sequence.
-
on_batch_unpacked
(self, x: dict, y: np.ndarray)¶ Padding.
Pad x[‘text_left’], x[‘text_right] and x[‘match_histogram’].
-
class
matchzoo.dataloader.callbacks.
BertPadding
(fixed_length_left: int = None, fixed_length_right: int = None, pad_value: typing.Union[int, str] = 0, pad_mode: str = 'pre')¶ Bases:
matchzoo.engine.base_callback.BaseCallback
Pad data for bert preprocessor.
Parameters: - fixed_length_left – Integer. If set, text_left will be padded to this length.
- fixed_length_right – Integer. If set, text_right will be padded to this length.
- pad_value – the value to fill text.
- pad_mode – String, pre or post: pad either before or after each sequence.
-
on_batch_unpacked
(self, x: dict, y: np.ndarray)¶ Pad x[‘text_left’] and x[‘text_right].