/data/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

import tensorflow as tf
import numpy as np
from m3tl.test_base import TestBase
from m3tl.input_fn import train_eval_input_fn
from m3tl.params import Params
test_base = TestBase()
params = test_base.params

params.assign_problem('weibo_fake_ner&weibo_fake_cls|weibo_fake_multi_cls|weibo_masklm|weibo_pretrain',base_dir=test_base.tmpckptdir)

hidden_dim = params.bert_config.hidden_size

train_dataset = train_eval_input_fn(params=params)
one_batch = next(train_dataset.as_numpy_iterator())

# prepare top dummy inputs
# non-empty inputs
def create_dummy_features_hidden_features(batch_size=1, hidden_dim=768, sample_features: dict=None):
    seq_len = sample_features['input_ids'].shape[1]
    dummy_features = {}
    for feature_name, feature in sample_features.items():
        if len(feature.shape) > 1:
            dummy_features[feature_name] = np.zeros(shape=(batch_size, *feature.shape[1:]), dtype=feature.dtype)
        else:
            dummy_features[feature_name] = np.zeros(shape=(batch_size), dtype=feature.dtype)
    dummy_features['model_input_mask'] = np.ones_like(dummy_features['input_ids'], dtype=np.int32)
    dummy_hidden = {
        'seq': np.zeros(shape=(batch_size, seq_len, hidden_dim), dtype=np.float32),
        'pooled': np.zeros(shape=(batch_size, hidden_dim), dtype=np.float32),
    }
    return dummy_features, dummy_hidden

def test_top_layer(top_class, problem: str, params: Params, sample_features: dict, **kwargs):
    print('Testing {}'.format(top_class.__name__))
    top_layer = top_class(params, problem, **kwargs)
    # non empty test
    d_f, d_h = create_dummy_features_hidden_features(batch_size=1, hidden_dim=hidden_dim, sample_features=sample_features)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.TRAIN)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.EVAL)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.PREDICT)

    d_f, d_h = create_dummy_features_hidden_features(batch_size=2, hidden_dim=hidden_dim, sample_features=sample_features)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.TRAIN)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.EVAL)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.PREDICT)

    # empty test
    d_f, d_h = create_dummy_features_hidden_features(batch_size=0, hidden_dim=hidden_dim, sample_features=sample_features)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.TRAIN)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.EVAL)
    _ = top_layer((d_f, d_h), mode=tf.estimator.ModeKeys.PREDICT)

/data/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
WARNING:root:bert_config not exists. will load model from huggingface checkpoint.

Adding new problem weibo_fake_ner, problem type: seq_tag
Adding new problem weibo_cws, problem type: seq_tag
Adding new problem weibo_fake_multi_cls, problem type: multi_cls
Adding new problem weibo_fake_cls, problem type: cls
Adding new problem weibo_masklm, problem type: masklm
Adding new problem weibo_pretrain, problem type: pretrain

WARNING:root:bert_config not exists. will load model from huggingface checkpoint.

INFO:tensorflow:['科', '技', '全', '方', '位', '资', '讯', '智', '能', '，', '快', '捷', '的', '汽', '车', '生', '活', '需', '要', '有', '三', '屏', '一', '云', '爱', '你']
INFO:tensorflow:input_ids: [101, 4906, 2825, 1059, 3175, 855, 6598, 6380, 3255, 5543, 8024, 2571, 2949, 4638, 3749, 6756, 4495, 3833, 7444, 6206, 3300, 676, 2242, 671, 756, 4263, 872, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:weibo_fake_ner_label_ids: [9, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9]
INFO:tensorflow:['科', '技', '全', '方', '位', '资', '讯', '智', '能', '，', '快', '捷', '的', '汽', '车', '生', '活', '需', '要', '有', '三', '屏', '一', '云', '爱', '你']
INFO:tensorflow:input_ids: [101, 4906, 2825, 1059, 3175, 855, 6598, 6380, 3255, 5543, 8024, 2571, 2949, 4638, 3749, 6756, 4495, 3833, 7444, 6206, 3300, 676, 2242, 671, 756, 4263, 872, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:weibo_fake_cls_label_ids: 0
INFO:tensorflow:['一', '节', '课', '的', '时', '间', '真', '心', '感', '动', '了', '李', '开', '复', '感', '动']
INFO:tensorflow:input_ids: [101, 671, 5688, 6440, 4638, 3198, 7313, 4696, 2552, 2697, 1220, 749, 3330, 2458, 1908, 2697, 1220, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:weibo_fake_ner_label_ids: [9, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 3, 7, 7, 8, 8, 9]
INFO:tensorflow:['一', '节', '课', '的', '时', '间', '真', '心', '感', '动', '了', '李', '开', '复', '感', '动']
INFO:tensorflow:input_ids: [101, 671, 5688, 6440, 4638, 3198, 7313, 4696, 2552, 2697, 1220, 749, 3330, 2458, 1908, 2697, 1220, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:weibo_fake_cls_label_ids: 1
INFO:tensorflow:text: ['科', '技', '全', '方', '位', '资', '讯', '智', '能', '，', '快', '捷', '的', '汽', '车', '生', '活', '需', '要', '有', '三', '屏', '一', '云', '爱', '你']
INFO:tensorflow:image: [[0.37756362 0.427908   0.23033233 0.00389367 0.3717356  0.58322454
  0.5209911  0.12825859 0.4990815  0.16428422]
 [0.59135963 0.83579896 0.56453828 0.46776627 0.53173419 0.66503375
  0.93466986 0.35
INFO:tensorflow:input_ids: [101, 4906, 2825, 1059, 3175, 855, 6598, 6380, 3255, 5543, 8024, 2571, 2949, 4638, 3749, 6756, 4495, 3833, 7444, 6206, 3300, 676, 2242, 671, 756, 4263, 872, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:weibo_fake_multi_cls_label_ids: [1 1 1]
INFO:tensorflow:image_input: [[0.37756362 0.427908   0.23033233 0.00389367 0.3717356  0.58322454
  0.5209911  0.12825859 0.4990815  0.16428422]
 [0.59135963 0.83579896 0.56453828 0.46776627 0.53173419 0.66503375
  0.93466986 0.35
INFO:tensorflow:image_mask: [1, 1, 1, 1, 1]
INFO:tensorflow:image_segment_ids: [0 0 0 0 0]
INFO:tensorflow:text: ['一', '节', '课', '的', '时', '间', '真', '心', '感', '动', '了', '李', '开', '复', '感', '动']
INFO:tensorflow:image: [[0.1606243  0.53437483 0.55145979 0.58781826 0.12051976 0.33179046
  0.26405855 0.2948657  0.01190474 0.31461692]
 [0.80500281 0.41252208 0.95845158 0.36994308 0.6825559  0.33154005
  0.96662706 0.64
INFO:tensorflow:input_ids: [101, 671, 5688, 6440, 4638, 3198, 7313, 4696, 2552, 2697, 1220, 749, 3330, 2458, 1908, 2697, 1220, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:weibo_fake_multi_cls_label_ids: [1 1 1]
INFO:tensorflow:image_input: [[0.1606243  0.53437483 0.55145979 0.58781826 0.12051976 0.33179046
  0.26405855 0.2948657  0.01190474 0.31461692]
 [0.80500281 0.41252208 0.95845158 0.36994308 0.6825559  0.33154005
  0.96662706 0.64
INFO:tensorflow:image_mask: [1, 1, 1, 1, 1]
INFO:tensorflow:image_segment_ids: [0 0 0 0 0]
INFO:tensorflow:['对', '，', '输', '给', '一', '个', '女', '人', '，', '的', '成', '绩', '。', '失', '望']
INFO:tensorflow:input_ids: [101, 103, 103, 6783, 5314, 671, 702, 1957, 782, 8024, 4638, 2768, 5327, 511, 1927, 3307, 102]
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:masked_lm_positions: [1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:masked_lm_ids: [2190 8024    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0]
INFO:tensorflow:masked_lm_weights: [1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
INFO:tensorflow:['回', '复', '支', '持', '，', '赞', '成', '，', '哈', '哈', '米', '八', '吴', '够', '历', '史', '要', '的', '陈', '小', '奥', '丁', '丁', '我', '爱', '小', '肥', '肥', '一', '族', '大', '头', '仔', '大', '家', '团', '结', '一', '致', '，',
INFO:tensorflow:input_ids: [101, 103, 103, 3118, 2898, 8024, 6614, 103, 8024, 1506, 1506, 5101, 1061, 103, 1916, 1325, 1380, 103, 4638, 103, 2207, 1952, 672, 103, 2769, 4263, 103, 5503, 5503, 671, 3184, 1920, 1928, 798, 103, 21
INFO:tensorflow:input_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
INFO:tensorflow:segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:masked_lm_positions: [1, 2, 7, 13, 17, 19, 23, 26, 34, 48, 53, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO:tensorflow:masked_lm_ids: [1726 1908 2768 1426 6206 7357  672 2207 1920 6983 8024    0    0    0
    0    0    0    0    0    0]
INFO:tensorflow:masked_lm_weights: [1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0]
INFO:tensorflow:sampling weights: 
INFO:tensorflow:weibo_fake_cls_weibo_fake_ner: 0.4
INFO:tensorflow:weibo_fake_multi_cls: 0.2
INFO:tensorflow:weibo_masklm: 0.2
INFO:tensorflow:weibo_pretrain: 0.2

Imports and utils

empty_tensor_handling_loss[source]

empty_tensor_handling_loss(labels, logits, loss_fn)

nan_loss_handling[source]

nan_loss_handling(loss)

create_dummy_if_empty[source]

create_dummy_if_empty(inp_tensor:Tensor)

class BaseTop[source]

BaseTop(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

Sequence Labeling

class SequenceLabel[source]

SequenceLabel(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

test_top_layer(SequenceLabel, problem='weibo_fake_ner', params=params, sample_features=one_batch)
params.crf = False
test_top_layer(SequenceLabel, problem='weibo_fake_ner', params=params, sample_features=one_batch)

Testing SequenceLabel
Testing SequenceLabel

Classification

class Classification[source]

Classification(*args, **kwargs) :: Layer

This is the class from which all layers inherit.

A layer is a callable object that takes as input one or more tensors and that outputs one or more tensors. It involves computation, defined in the call() method, and a state (weight variables), defined either in the constructor __init__() or in the build() method.

Users will just instantiate a layer and then treat it as a callable.

Arguments: trainable: Boolean, whether the layer's variables should be trainable. name: String name of the layer. dtype: The dtype of the layer's computations and weights. Can also be a tf.keras.mixed_precision.Policy, which allows the computation and weight dtype to differ. Default of None means to use tf.keras.mixed_precision.global_policy(), which is a float32 policy unless set to different value. dynamic: Set this to True if your layer should only be run eagerly, and should not be used to generate a static computation graph. This would be the case for a Tree-RNN or a recursive network, for example, or generally for any layer that manipulates tensors using Python control flow. If False, we assume that the layer can safely be used to generate a static computation graph.

Attributes: name: The name of the layer (string). dtype: The dtype of the layer's weights. variable_dtype: Alias of dtype. compute_dtype: The dtype of the layer's computations. Layers automatically cast inputs to this dtype which causes the computations and output to also be in this dtype. When mixed precision is used with a tf.keras.mixed_precision.Policy, this will be different than variable_dtype. dtype_policy: The layer's dtype policy. See the tf.keras.mixed_precision.Policy documentation for details. trainable_weights: List of variables to be included in backprop. non_trainable_weights: List of variables that should not be included in backprop. weights: The concatenation of the lists trainable_weights and non_trainable_weights (in this order). trainable: Whether the layer should be trained (boolean), i.e. whether its potentially-trainable weights should be returned as part of layer.trainable_weights. input_spec: Optional (list of) InputSpec object(s) specifying the constraints on inputs that can be accepted by the layer.

We recommend that descendants of Layer implement the following methods:

  • __init__(): Defines custom layer attributes, and creates layer state variables that do not depend on input shapes, using add_weight().
  • build(self, input_shape): This method can be used to create weights that depend on the shape(s) of the input(s), using add_weight(). __call__() will automatically build the layer (if it has not been built yet) by calling build().
  • call(self, *args, **kwargs): Called in __call__ after making sure build() has been called. call() performs the logic of applying the layer to the input tensors (which should be passed in as argument). Two reserved keyword arguments you can optionally use in call() are:
    • training (boolean, whether the call is in inference mode or training mode)
    • mask (boolean tensor encoding masked timesteps in the input, used in RNN layers)
  • get_config(self): Returns a dictionary containing the configuration used to initialize this layer. If the keys differ from the arguments in __init__, then override from_config(self) as well. This method is used when saving the layer or a model that contains this layer.

Examples:

Here's a basic example: a layer with two variables, w and b, that returns y = w . x + b. It shows how to implement build() and call(). Variables set as attributes of a layer are tracked as weights of the layers (in layer.weights).

class SimpleDense(Layer):

  def __init__(self, units=32):
      super(SimpleDense, self).__init__()
      self.units = units

  def build(self, input_shape):  # Create the state of the layer (weights)
    w_init = tf.random_normal_initializer()
    self.w = tf.Variable(
        initial_value=w_init(shape=(input_shape[-1], self.units),
                             dtype='float32'),
        trainable=True)
    b_init = tf.zeros_initializer()
    self.b = tf.Variable(
        initial_value=b_init(shape=(self.units,), dtype='float32'),
        trainable=True)

  def call(self, inputs):  # Defines the computation from inputs to outputs
      return tf.matmul(inputs, self.w) + self.b

# Instantiates the layer.
linear_layer = SimpleDense(4)

# This will also call `build(input_shape)` and create the weights.
y = linear_layer(tf.ones((2, 2)))
assert len(linear_layer.weights) == 2

# These weights are trainable, so they're listed in `trainable_weights`:
assert len(linear_layer.trainable_weights) == 2

Note that the method add_weight() offers a shortcut to create weights:

class SimpleDense(Layer):

  def __init__(self, units=32):
      super(SimpleDense, self).__init__()
      self.units = units

  def build(self, input_shape):
      self.w = self.add_weight(shape=(input_shape[-1], self.units),
                               initializer='random_normal',
                               trainable=True)
      self.b = self.add_weight(shape=(self.units,),
                               initializer='random_normal',
                               trainable=True)

  def call(self, inputs):
      return tf.matmul(inputs, self.w) + self.b

Besides trainable weights, updated via backpropagation during training, layers can also have non-trainable weights. These weights are meant to be updated manually during call(). Here's a example layer that computes the running sum of its inputs:

class ComputeSum(Layer):

  def __init__(self, input_dim):
      super(ComputeSum, self).__init__()
      # Create a non-trainable weight.
      self.total = tf.Variable(initial_value=tf.zeros((input_dim,)),
                               trainable=False)

  def call(self, inputs):
      self.total.assign_add(tf.reduce_sum(inputs, axis=0))
      return self.total

my_sum = ComputeSum(2)
x = tf.ones((2, 2))

y = my_sum(x)
print(y.numpy())  # [2. 2.]

y = my_sum(x)
print(y.numpy())  # [4. 4.]

assert my_sum.weights == [my_sum.total]
assert my_sum.non_trainable_weights == [my_sum.total]
assert my_sum.trainable_weights == []

For more information about creating layers, see the guide Writing custom layers and models with Keras

test_top_layer(Classification, problem='weibo_fake_cls', params=params, sample_features=one_batch)

Testing Classification

Pretrain

class PreTrain[source]

PreTrain(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

test_top_layer(PreTrain, problem='weibo_pretrain', params=params, sample_features=one_batch, share_embedding=False)

Testing PreTrain

Seq2Seq

class Seq2Seq[source]

Seq2Seq(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

Multi-Label Classification

class MultiLabelClassification[source]

MultiLabelClassification(*args, **kwargs) :: Model

Model groups layers into an object with training and inference features.

Arguments: inputs: The input(s) of the model: a keras.Input object or list of keras.Input objects. outputs: The output(s) of the model. See Functional API example below. name: String, the name of the model.

There are two ways to instantiate a Model:

1 - With the "Functional API", where you start from Input, you chain layer calls to specify the model's forward pass, and finally you create your model from inputs and outputs:

import tensorflow as tf

inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

2 - By subclassing the Model class: in that case, you should define your layers in __init__ and you should implement the model's forward pass in call.

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)

  def call(self, inputs):
    x = self.dense1(inputs)
    return self.dense2(x)

model = MyModel()

If you subclass Model, you can optionally have a training argument (boolean) in call, which you can use to specify a different behavior in training and inference:

import tensorflow as tf

class MyModel(tf.keras.Model):

  def __init__(self):
    super(MyModel, self).__init__()
    self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
    self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
    self.dropout = tf.keras.layers.Dropout(0.5)

  def call(self, inputs, training=False):
    x = self.dense1(inputs)
    if training:
      x = self.dropout(x, training=training)
    return self.dense2(x)

model = MyModel()

Once the model is created, you can config the model with losses and metrics with model.compile(), train the model with model.fit(), or use the model to do prediction with model.predict().

Mask LM

Mask LM top layer

class MaskLM[source]

MaskLM(*args, **kwargs) :: Model

Multimodal MLM top layer.

test_top_layer(MaskLM, problem='weibo_masklm', params=params, sample_features=one_batch, share_embedding=False)

Testing MaskLM