import numpy as np
import tensorflow as tf
from m3tl.input_fn import train_eval_input_fn
from m3tl.test_base import TestBase, test_top_layer

test_base = TestBase()
params = test_base.params

hidden_dim = params.bert_config.hidden_size

train_dataset = train_eval_input_fn(params=params)
one_batch = next(train_dataset.as_numpy_iterator())

2021-06-12 21:42:24.148 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_fake_ner, problem type: seq_tag
2021-06-12 21:42:24.149 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_fake_multi_cls, problem type: multi_cls
2021-06-12 21:42:24.149 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_fake_cls, problem type: cls
2021-06-12 21:42:24.150 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_masklm, problem type: masklm
2021-06-12 21:42:24.150 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_fake_regression, problem type: regression
2021-06-12 21:42:24.151 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_fake_vector_fit, problem type: vector_fit
2021-06-12 21:42:24.151 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem weibo_premask_mlm, problem type: premask_mlm
2021-06-12 21:42:24.152 | INFO     | m3tl.base_params:register_multiple_problems:526 - Adding new problem fake_contrastive_learning, problem type: contrastive_learning
2021-06-12 21:42:24.152 | WARNING  | m3tl.base_params:assign_problem:620 - base_dir and dir_name arguments will be deprecated in the future. Please use model_dir instead.
2021-06-12 21:42:24.153 | WARNING  | m3tl.base_params:prepare_dir:364 - bert_config not exists. will load model from huggingface checkpoint.
2021-06-12 21:42:30.466 | WARNING  | m3tl.read_write_tfrecord:chain_processed_data:248 - Chaining problems with & may consume a lot of memory if data is not pyspark RDD.
2021-06-12 21:42:30.481 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_fake_cls_weibo_fake_ner_weibo_fake_regression_weibo_fake_vector_fit/train_00000.tfrecord
2021-06-12 21:42:30.547 | WARNING  | m3tl.read_write_tfrecord:chain_processed_data:248 - Chaining problems with & may consume a lot of memory if data is not pyspark RDD.
2021-06-12 21:42:30.561 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_fake_cls_weibo_fake_ner_weibo_fake_regression_weibo_fake_vector_fit/eval_00000.tfrecord
2021-06-12 21:42:30.588 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_fake_multi_cls/train_00000.tfrecord
2021-06-12 21:42:30.612 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_fake_multi_cls/eval_00000.tfrecord
2021-06-12 21:42:30.688 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_masklm/train_00000.tfrecord
2021-06-12 21:42:30.736 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_masklm/eval_00000.tfrecord
2021-06-12 21:42:30.801 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_premask_mlm/train_00000.tfrecord
2021-06-12 21:42:30.864 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/weibo_premask_mlm/eval_00000.tfrecord
2021-06-12 21:42:30.883 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/fake_contrastive_learning/train_00000.tfrecord
2021-06-12 21:42:30.900 | DEBUG    | m3tl.read_write_tfrecord:_write_fn:134 - Writing /tmp/tmpyjjr_dm7/fake_contrastive_learning/eval_00000.tfrecord
2021-06-12 21:42:31.923 | INFO     | m3tl.input_fn:train_eval_input_fn:56 - sampling weights: 
2021-06-12 21:42:31.924 | INFO     | m3tl.input_fn:train_eval_input_fn:57 - {
    "weibo_fake_cls_weibo_fake_ner_weibo_fake_regression_weibo_fake_vector_fit": 0.2127659574468085,
    "weibo_fake_multi_cls": 0.2127659574468085,
    "weibo_masklm": 0.14893617021276595,
    "weibo_premask_mlm": 0.2127659574468085,
    "fake_contrastive_learning": 0.2127659574468085
}

one_batch['array_input_ids'].shape

(32, 1, 10)

Imports and utils

Top Layer

class SimCSE[source]

SimCSE(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

get_contrastive_learning_model[source]

get_contrastive_learning_model(params:BaseParams, problem_name:str, model_name:str)

class ContrastiveLearning[source]

ContrastiveLearning(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

test_top_layer(ContrastiveLearning, problem='fake_contrastive_learning',
               params=params, sample_features=one_batch, hidden_dim=hidden_dim, test_batch_size_list=[0,2])

2021-06-12 21:42:33.553 | DEBUG    | m3tl.test_base:test_top_layer:247 - Testing ContrastiveLearning
2021-06-12 21:42:33.569 | WARNING  | __main__:get_contrastive_learning_model:7 - None not match any contrastive learning model, using SimCSE
2021-06-12 21:42:33.585 | DEBUG    | m3tl.test_base:test_top_layer:253 - testing batch size 0
2021-06-12 21:42:33.593 | DEBUG    | m3tl.test_base:test_top_layer:253 - testing batch size 2

Get or make label encoder function

contrastive_learning_get_or_make_label_encoder_fn[source]

contrastive_learning_get_or_make_label_encoder_fn(params:BaseParams, problem:str, mode:str, label_list:List[str], *args, **kwargs)

Label handing function

contrastive_learning_label_handling_fn[source]

contrastive_learning_label_handling_fn(target:str, label_encoder=None, tokenizer=None, decoding_length=None, *args, **kwargs)