携手创作，共同成长！这是我参与「掘金日新计划 · 8 月更文挑战」的第15天，点击查看活动详情
在上一篇文章中，我们进行Transformer模型的实现，载入数据，构建subword的方法，创建数据集，生成工具函数，构建mask,实现缩放点积注意力机制。
今天我们进行多头注意力机制的实现，feedforward 层次实现，EncoderLayer实现，DecoderLayer实现,EecoderModel实现,DecoderModel实现,Transformer实现,训练模型，自定义学习率，损失函数的实现， mask的创建与实现，模型预测，attention可视化，示例展示。
-
2.6 多头注意力机制的实现
理论上:
- x -> Wq0 -> q0
- x -> Wk0 -> k0
- x -> Wv0 -> v0
实战中:
- q -> Wq0 -> q0
- k -> Wk0 -> k0
- v -> Wv0 -> v0
实战中技巧：
- q -> Wq -> Q -> split -> q0, q1, q2...
做两次维度切换的原因：第一次是在做split_heads的时候，把第一和第二维度切换了一下；在计算完attention之后，在把这两个维度切换回来，这是因为我的scaled_dot_product_attention计算attentioin的时候是计算后两维的，因而，我们需要将num_heads先换过去，在换回来
class MultiHeadAttention(keras.layers.Layer):
def __init__(self, d_model, num_heads):
super(MultiHeadAttention, self).__init__()
self.num_heads = num_heads
self.d_model = d_model
assert self.d_model % self.num_heads == 0
self.depth = self.d_model // self.num_heads
self.WQ = keras.layers.Dense(self.d_model)
self.WK = keras.layers.Dense(self.d_model)
self.WV = keras.layers.Dense(self.d_model)
self.dense = keras.layers.Dense(self.d_model)
def split_heads(self, x, batch_size):
# x.shape: (batch_size, seq_len, d_model)
# d_model = num_heads * depth
# x -> (batch_size, num_heads, seq_len, depth)
x = tf.reshape(x,
(batch_size, -1, self.num_heads, self.depth))
return tf.transpose(x, perm=[0, 2, 1, 3])
def call(self, q, k, v, mask):
batch_size = tf.shape(q)[0]
q = self.WQ(q) # q.shape: (batch_size, seq_len_q, d_model)
k = self.WK(k) # k.shape: (batch_size, seq_len_k, d_model)
v = self.WV(v) # v.shape: (batch_size, seq_len_v, d_model)
# q.shape: (batch_size, num_heads, seq_len_q, depth)
q = self.split_heads(q, batch_size)
# k.shape: (batch_size, num_heads, seq_len_k, depth)
k = self.split_heads(k, batch_size)
# v.shape: (batch_size, num_heads, seq_len_v, depth)
v = self.split_heads(v, batch_size)
# scaled_attention_outputs.shape: (batch_size, num_heads, seq_len_q, depth)
# attention_weights.shape: (batch_size, num_heads, seq_len_q, seq_len_k)
scaled_attention_outputs, attention_weights = \
scaled_dot_product_attention(q, k, v, mask)
# scaled_attention_outputs.shape: (batch_size, seq_len_q, num_heads, depth)
scaled_attention_outputs = tf.transpose(
scaled_attention_outputs, perm = [0, 2, 1, 3])
# concat_attention.shape: (batch_size, seq_len_q, d_model)
concat_attention = tf.reshape(scaled_attention_outputs,
(batch_size, -1, self.d_model))
# output.shape : (batch_size, seq_len_q, d_model)
output = self.dense(concat_attention)
return output, attention_weights
temp_mha = MultiHeadAttention(d_model=512, num_heads=8)
y = tf.random.uniform((1, 60, 256)) # (batch_size, seq_len_q, dim)
output, attn = temp_mha(y, y, y, mask = None)
print(output.shape)
print(attn.shape)
运行结果：
(1, 60, 512)
(1, 8, 60, 60)
-
2.7 feedforward 层次实现
这个在encoder和decoder中都会用到，所以先抽象出来实现。
def feed_forward_network(d_model, dff):
# dff: dim of feed forward network.
return keras.Sequential([
keras.layers.Dense(dff, activation='relu'),
keras.layers.Dense(d_model)
])
sample_ffn = feed_forward_network(512, 2048)
sample_ffn(tf.random.uniform((64, 50, 512))).shape
运行结果：
TensorShape([64, 50, 512])
-
2.8 EncoderLayer实现
使用子类API来实现：把call函数构造出来把encoderlayer构造出来当函数来使用
EncoderLayer的网络结构：
- x -> self attention -> add & normalize & dropout-> feed_forward -> add & normalize & dropout
class EncoderLayer(keras.layers.Layer):
"""
x -> self attention -> add & normalize & dropout
-> feed_forward -> add & normalize & dropout
"""
def __init__(self, d_model, num_heads, dff, rate=0.1):
super(EncoderLayer, self).__init__()
self.mha = MultiHeadAttention(d_model, num_heads)
self.ffn = feed_forward_network(d_model, dff)
self.layer_norm1 = keras.layers.LayerNormalization(
epsilon = 1e-6)
self.layer_norm2 = keras.layers.LayerNormalization(
epsilon = 1e-6)
self.dropout1 = keras.layers.Dropout(rate)
self.dropout2 = keras.layers.Dropout(rate)
def call(self, x, training, encoder_padding_mask):
# x.shape : (batch_size, seq_len, dim=d_model)
# attn_output.shape: (batch_size, seq_len, d_model)
# out1.shape : (batch_size, seq_len, d_model)
attn_output, _ = self.mha(x, x, x, encoder_padding_mask)
attn_output = self.dropout1(attn_output, training=training)
out1 = self.layer_norm1(x + attn_output)
# ffn_output.shape: (batch_size, seq_len, d_model)
# out2.shape : (batch_size, seq_len, d_model)
ffn_output = self.ffn(out1)
ffn_output = self.dropout2(ffn_output, training=training)
out2 = self.layer_norm2(out1 + ffn_output)
return out2
sample_encoder_layer = EncoderLayer(512, 8, 2048)
sample_input = tf.random.uniform((64, 50, 512))
sample_output = sample_encoder_layer(sample_input, False, None)
print(sample_output.shape)
运行结果：
TensorShape([64, 50, 512])
可以看到，我们的输出也是64x50x512的矩阵，说明我们经过了encoderlayer之后的size没有变，这也是我们所期望的，因为我们做了两次残差连接。
-
2.8 DecoderLayer实现
DecoderLayer的网络结构:
- x -> self attention -> add & normalize & dropout -> out1
- out1 , encoding_outputs -> attention -> add & normalize & dropout -> out2
- out2 -> ffn -> add & normalize & dropout -> out3
这里的DecoderLayer和EncoderLayer是类似的，但是DecoderLayer会有一个多余的层次，那就是Encoder和Decoder之间的attention
这里依然使用子类API来实现
class DecoderLayer(keras.layers.Layer):
"""
x -> self attention -> add & normalize & dropout -> out1
out1 , encoding_outputs -> attention -> add & normalize & dropout -> out2
out2 -> ffn -> add & normalize & dropout -> out3
"""
def __init__(self, d_model, num_heads, dff, rate = 0.1):
super(DecoderLayer, self).__init__()
self.mha1 = MultiHeadAttention(d_model, num_heads)
self.mha2 = MultiHeadAttention(d_model, num_heads)
self.ffn = feed_forward_network(d_model, dff)
self.layer_norm1 = keras.layers.LayerNormalization(
epsilon = 1e-6)
self.layer_norm2 = keras.layers.LayerNormalization(
epsilon = 1e-6)
self.layer_norm3 = keras.layers.LayerNormalization(
epsilon = 1e-6)
self.dropout1 = keras.layers.Dropout(rate)
self.dropout2 = keras.layers.Dropout(rate)
self.dropout3 = keras.layers.Dropout(rate)
def call(self, x, encoding_outputs, training,
decoder_mask, encoder_decoder_padding_mask):
# decoder_mask: 由look_ahead_mask和decoder_padding_mask合并而来
# x.shape: (batch_size, target_seq_len, d_model)
# encoding_outputs.shape: (batch_size, input_seq_len, d_model)
# attn1, out1.shape : (batch_size, target_seq_len, d_model)
attn1, attn_weights1 = self.mha1(x, x, x, decoder_mask)
attn1 = self.dropout1(attn1, training = training)
out1 = self.layer_norm1(attn1 + x)
# attn2, out2.shape : (batch_size, target_seq_len, d_model)
attn2, attn_weights2 = self.mha2(
out1, encoding_outputs, encoding_outputs,
encoder_decoder_padding_mask)
attn2 = self.dropout2(attn2, training = training)
out2 = self.layer_norm2(attn2 + out1)
# ffn_output, out3.shape: (batch_size, target_seq_len, d_model)
ffn_output = self.ffn(out2)
ffn_output = self.dropout3(ffn_output, training=training)
out3 = self.layer_norm3(ffn_output + out2)
return out3, attn_weights1, attn_weights2
sample_decoder_layer = DecoderLayer(512, 8, 2048)
sample_decoder_input = tf.random.uniform((64, 60, 512))
sample_decoder_output, sample_decoder_attn_weights1, sample_decoder_attn_weights2 = sample_decoder_layer(
sample_decoder_input, sample_output, False, None, None)
print(sample_decoder_output.shape)
print(sample_decoder_attn_weights1.shape)
print(sample_decoder_attn_weights2.shape)
运行结果：
(64, 60, 512)
(64, 8, 60, 60)
(64, 8, 60, 50)
-
在这里sample_decoder_output的shape是64x60x512，这和我们期望的是一致的，因为我们的target_seq_len是60，d_model是512
-
我们的第一个attention_weights是64x8x60x60的四维矩阵，其中60x60就是self_attention
-
第二个attention_weights是64x8x60x50的四维矩阵，这是一个encoder到decoder的attention_weights。
-
2.9 EecoderModel实现
EecoderModel和DecoderModel分别是由EncoderLayer和DecoderLayer组成的，只要多堆叠几个EncoderLayer和DecoderLayer就可以搭建EncoderModel和DecoderModel
依然使用子类API来构建
class EncoderModel(keras.layers.Layer):
def __init__(self, num_layers, input_vocab_size, max_length,
d_model, num_heads, dff, rate=0.1):
super(EncoderModel, self).__init__()
self.d_model = d_model
self.num_layers = num_layers
self.max_length = max_length
self.embedding = keras.layers.Embedding(input_vocab_size,
self.d_model)
# position_embedding.shape: (1, max_length, d_model)
self.position_embedding = get_position_embedding(max_length,
self.d_model)
self.dropout = keras.layers.Dropout(rate)
self.encoder_layers = [
EncoderLayer(d_model, num_heads, dff, rate)
for _ in range(self.num_layers)]
def call(self, x, training, encoder_padding_mask):
# x.shape: (batch_size, input_seq_len)
input_seq_len = tf.shape(x)[1]
tf.debugging.assert_less_equal(
input_seq_len, self.max_length,
"input_seq_len should be less or equal to self.max_length")
# x.shape: (batch_size, input_seq_len, d_model)
x = self.embedding(x)
x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
x += self.position_embedding[:, :input_seq_len, :]
x = self.dropout(x, training = training)
for i in range(self.num_layers):
x = self.encoder_layers[i](x, training,
encoder_padding_mask)
# x.shape: (batch_size, input_seq_len, d_model)
return x
sample_encoder_model = EncoderModel(2, 8500, max_length,
512, 8, 2048)
sample_encoder_model_input = tf.random.uniform((64, 37))
sample_encoder_model_output = sample_encoder_model(
sample_encoder_model_input, False, encoder_padding_mask = None)
print(sample_encoder_model_output.shape)
运行结果：
(64, 37, 512)
最后的shape是64x37x512，64是batch_size,37是input_seq_len,512是d_model,从结果来看我们的Model是正确的。
-
2.10 DecoderModel实现
和EncoderModel非常的类似，也是使用子类API来实现
class DecoderModel(keras.layers.Layer):
def __init__(self, num_layers, target_vocab_size, max_length,
d_model, num_heads, dff, rate=0.1):
super(DecoderModel, self).__init__()
self.num_layers = num_layers
self.max_length = max_length
self.d_model = d_model
self.embedding = keras.layers.Embedding(target_vocab_size,
d_model)
self.position_embedding = get_position_embedding(max_length,
d_model)
self.dropout = keras.layers.Dropout(rate)
self.decoder_layers = [
DecoderLayer(d_model, num_heads, dff, rate)
for _ in range(self.num_layers)]
def call(self, x, encoding_outputs, training,
decoder_mask, encoder_decoder_padding_mask):
# x.shape: (batch_size, output_seq_len)
output_seq_len = tf.shape(x)[1]
tf.debugging.assert_less_equal(
output_seq_len, self.max_length,
"output_seq_len should be less or equal to self.max_length")
attention_weights = {}
# x.shape: (batch_size, output_seq_len, d_model)
x = self.embedding(x)
x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
x += self.position_embedding[:, :output_seq_len, :]
x = self.dropout(x, training = training)
for i in range(self.num_layers):
x, attn1, attn2 = self.decoder_layers[i](
x, encoding_outputs, training,
decoder_mask, encoder_decoder_padding_mask)
attention_weights[
'decoder_layer{}_att1'.format(i+1)] = attn1
attention_weights[
'decoder_layer{}_att2'.format(i+1)] = attn2
# x.shape: (batch_size, output_seq_len, d_model)
return x, attention_weights
sample_decoder_model = DecoderModel(2, 8000, max_length,
512, 8, 2048)
sample_decoder_model_input = tf.random.uniform((64, 35))
sample_decoder_model_output, sample_decoder_model_att \
= sample_decoder_model(
sample_decoder_model_input,
sample_encoder_model_output,
training = False, decoder_mask = None,
encoder_decoder_padding_mask = None)
print(sample_decoder_model_output.shape)
for key in sample_decoder_model_att:
print(sample_decoder_model_att[key].shape)
运行结果;
(64, 35, 512)
(64, 8, 35, 35)
(64, 8, 35, 37)
(64, 8, 35, 35)
(64, 8, 35, 37)
-
可以看到对于sample_decoder_model_output的shape是64x35x512，和我们预期的输出的size是一样的，
-
而sample_decoder_model_attention里面有四个值，
- 这是因为我们的DecoderModel里面有两个DecoderLayer，
- 其中在这个对中的每一个值都是self_attention,size是35x35的，
- 第二个就是encoder，decoder之间的attention，都是35x37，这个37就是上一步出Encoder出的length，
- 第二个DecoderLayer的size和第一个的两个attention_weights是一样的,从shape中可以看到我们的Model是没有问题的。
-
2.11 Transformer实现
依旧使用子类API
class Transformer(keras.Model):
def __init__(self, num_layers, input_vocab_size, target_vocab_size,
max_length, d_model, num_heads, dff, rate=0.1):
super(Transformer, self).__init__()
self.encoder_model = EncoderModel(
num_layers, input_vocab_size, max_length,
d_model, num_heads, dff, rate)
self.decoder_model = DecoderModel(
num_layers, target_vocab_size, max_length,
d_model, num_heads, dff, rate)
self.final_layer = keras.layers.Dense(target_vocab_size)
def call(self, inp, tar, training, encoder_padding_mask,
decoder_mask, encoder_decoder_padding_mask):
# encoding_outputs.shape: (batch_size, input_seq_len, d_model)
encoding_outputs = self.encoder_model(
inp, training, encoder_padding_mask)
# decoding_outputs.shape: (batch_size, output_seq_len, d_model)
decoding_outputs, attention_weights = self.decoder_model(
tar, encoding_outputs, training,
decoder_mask, encoder_decoder_padding_mask)
# predictions.shape: (batch_size, output_seq_len, target_vocab_size)
predictions = self.final_layer(decoding_outputs)
return predictions, attention_weights
sample_transformer = Transformer(2, 8500, 8000, max_length,
512, 8, 2048, rate = 0.1)
temp_input = tf.random.uniform((64, 26))
temp_target = tf.random.uniform((64, 31))
predictions, attention_weights = sample_transformer(
temp_input, temp_target, training = False,
encoder_padding_mask = None,
decoder_mask = None,
encoder_decoder_padding_mask = None)
print(predictions.shape)
for key in attention_weights:
print(key, attention_weights[key].shape)
运行结果：
(64, 31, 8000)
decoder_layer1_att1 (64, 8, 31, 31)
decoder_layer1_att2 (64, 8, 31, 26)
decoder_layer2_att1 (64, 8, 31, 31)
decoder_layer2_att2 (64, 8, 31, 26)
在这里呢，我们有了定义模型的实现之后呢，就可以开始模型的训练了，模型训练有这么几个步骤：
-
- initializes model.
-
- define loss, optimizer, learning_rate schedule
-
- train_step
-
- train process
-
2.12 训练模型
- 2.12.1 定义一些超参数：
num_layers = 4
d_model = 128
dff = 512
num_heads = 8
input_vocab_size = pt_tokenizer.vocab_size + 2
target_vocab_size = en_tokenizer.vocab_size + 2
dropout_rate = 0.1
transformer = Transformer(num_layers,
input_vocab_size,
target_vocab_size,
max_length,
d_model, num_heads, dff, dropout_rate)
- 2.12.2 自定义学习率：
lrate = (d_model ** -0.5) * min(step_num ** (-0.5), step_num * warm_up_steps **(-1.5))
class CustomizedSchedule(
keras.optimizers.schedules.LearningRateSchedule):
def __init__(self, d_model, warmup_steps = 4000):
super(CustomizedSchedule, self).__init__()
self.d_model = tf.cast(d_model, tf.float32)
self.warmup_steps = warmup_steps
def __call__(self, step):
arg1 = tf.math.rsqrt(step)
arg2 = step * (self.warmup_steps ** (-1.5))
arg3 = tf.math.rsqrt(self.d_model)
return arg3 * tf.math.minimum(arg1, arg2)
learning_rate = CustomizedSchedule(d_model)
optimizer = keras.optimizers.Adam(learning_rate,
beta_1 = 0.9,
beta_2 = 0.98,
epsilon = 1e-9)
把CustomizedSchedule给画出来：
temp_learning_rate_schedule = CustomizedSchedule(d_model)
plt.plot(
temp_learning_rate_schedule(
tf.range(40000, dtype=tf.float32)))
plt.ylabel("Leraning rate")
plt.xlabel("Train step")
运行结果：
- 2.12.3 损失函数的实现
loss_object = keras.losses.SparseCategoricalCrossentropy(
from_logits = True, reduction = 'none')
def loss_function(real, pred):
mask = tf.math.logical_not(tf.math.equal(real, 0))
loss_ = loss_object(real, pred)
mask = tf.cast(mask, dtype=loss_.dtype)
loss_ *= mask
return tf.reduce_mean(loss_)
- 2.12.4 Mask的创建和使用
Encoder:
- encoder_padding_mask (self attention of EncoderLayer)
Decoder:
- look_ahead_mask (self attention of DecoderLayer) 当前位置的词不能看到之后位置上的词，因为之后位置上的词还没有被预测出来
- encoder_decoder_padding_mask (encoder-decoder attention of DecoderLayer)
- decoder_padding_mask (self attention of DecoderLayer)
def create_masks(inp, tar):
encoder_padding_mask = create_padding_mask(inp)
encoder_decoder_padding_mask = create_padding_mask(inp)
look_ahead_mask = create_look_ahead_mask(tf.shape(tar)[1])
decoder_padding_mask = create_padding_mask(tar)
decoder_mask = tf.maximum(decoder_padding_mask,
look_ahead_mask)
return encoder_padding_mask, decoder_mask, encoder_decoder_padding_mask
去一组数据对它进行测试：
temp_inp, temp_tar = iter(train_dataset.take(1)).next()
print(temp_inp.shape)
print(temp_tar.shape)
create_masks(temp_inp, temp_tar)
运行结果：
(64, 38)
(64, 39)
可以看到，temp_inp中有64个样本，每一个长度都是38，temp_tar有64个样本，长度都是39.
- 2.12.5 模型训练
train_loss = keras.metrics.Mean(name = 'train_loss')
train_accuracy = keras.metrics.SparseCategoricalAccuracy(
name = 'train_accuracy')
@tf.function
def train_step(inp, tar):
tar_inp = tar[:, :-1]
tar_real = tar[:, 1:]
encoder_padding_mask, decoder_mask, encoder_decoder_padding_mask \
= create_masks(inp, tar_inp)
with tf.GradientTape() as tape:
predictions, _ = transformer(inp, tar_inp, True,
encoder_padding_mask,
decoder_mask,
encoder_decoder_padding_mask)
loss = loss_function(tar_real, predictions)
gradients = tape.gradient(loss, transformer.trainable_variables)
optimizer.apply_gradients(
zip(gradients, transformer.trainable_variables))
train_loss(loss)
train_accuracy(tar_real, predictions)
epochs = 20
for epoch in range(epochs):
start = time.time()
train_loss.reset_states()
train_accuracy.reset_states()
for (batch, (inp, tar)) in enumerate(train_dataset):
train_step(inp, tar)
if batch % 100 == 0:
print('Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}'.format(
epoch + 1, batch, train_loss.result(),
train_accuracy.result()))
print('Epoch {} Loss {:.4f} Accuracy {:.4f}'.format(
epoch + 1, train_loss.result(), train_accuracy.result()))
print('Time take for 1 epoch: {} secs\n'.format(
time.time() - start))
经过一段时间的训练之后，loss从原本的4.2左右降到了0.56左右，相应的，我们的accuracy从0.0经过20次迭代之后到了0.34，这里需要注意，accuracy并不是机器翻译的一个正规指标，只是一个参考，只代表一个趋势
-
2.12.6 模型预测
-
eg: A B C D -> E F G H.
-
Train: A B C D, E F G -> F G H
-
Eval: A B C D -> E
A B C D, E -> F
A B C D, E F -> G
A B C D, E F G -> H
这个和seq2seq+attention中也是类似的，但是在Transformer中，在Train中处理encorder和decoder中可以并行化的去处理，因为self_attention是没有前后向依赖的，而在seq2seq+attention中它使用的是LSTM或者循环神经网络，是由计算依赖的，但是在预测中，是一样的。
def evaluate(inp_sentence):
input_id_sentence = [pt_tokenizer.vocab_size] \
+ pt_tokenizer.encode(inp_sentence) + [pt_tokenizer.vocab_size + 1]
# encoder_input.shape: (1, input_sentence_length)
encoder_input = tf.expand_dims(input_id_sentence, 0)
# decoder_input.shape: (1, 1)
decoder_input = tf.expand_dims([en_tokenizer.vocab_size], 0)
for i in range(max_length):
encoder_padding_mask, decoder_mask, encoder_decoder_padding_mask \
= create_masks(encoder_input, decoder_input)
# predictions.shape: (batch_size, output_target_len, target_vocab_size)
predictions, attention_weights = transformer(
encoder_input,
decoder_input,
False,
encoder_padding_mask,
decoder_mask,
encoder_decoder_padding_mask)
# predictions.shape: (batch_size, target_vocab_size)
predictions = predictions[:, -1, :]
predicted_id = tf.cast(tf.argmax(predictions, axis = -1),
tf.int32)
if tf.equal(predicted_id, en_tokenizer.vocab_size + 1):
return tf.squeeze(decoder_input, axis = 0), attention_weights
decoder_input = tf.concat([decoder_input, [predicted_id]],
axis = -1)
return tf.squeeze(decoder_input, axis = 0), attention_weights
- 2.12.7 attention可视化
def plot_encoder_decoder_attention(attention, input_sentence,
result, layer_name):
fig = plt.figure(figsize = (16, 8))
input_id_sentence = pt_tokenizer.encode(input_sentence)
# attention.shape: (num_heads, tar_len, input_len)
attention = tf.squeeze(attention[layer_name], axis = 0)
for head in range(attention.shape[0]):
ax = fig.add_subplot(2, 4, head + 1)
ax.matshow(attention[head][:-1, :])
fontdict = {'fontsize': 10}
ax.set_xticks(range(len(input_id_sentence) + 2))
ax.set_yticks(range(len(result)))
ax.set_ylim(len(result) - 1.5, -0.5)
ax.set_xticklabels(
['<start>'] + [pt_tokenizer.decode([i]) for i in input_id_sentence] + ['<end>'],
fontdict = fontdict, rotation = 90)
ax.set_yticklabels(
[en_tokenizer.decode([i]) for i in result if i < en_tokenizer.vocab_size],
fontdict = fontdict)
ax.set_xlabel('Head {}'.format(head + 1))
plt.tight_layout()
plt.show()
在写一个函数，来调用刚才的两个函数，来实现翻译一个句子并画出attention图示的过程
def translate(input_sentence, layer_name = ''):
result, attention_weights = evaluate(input_sentence)
predicted_sentence = en_tokenizer.decode(
[i for i in result if i < en_tokenizer.vocab_size])
print("Input: {}".format(input_sentence))
print("Predicted translation: {}".format(predicted_sentence))
if layer_name:
plot_encoder_decoder_attention(attention_weights, input_sentence,
result, layer_name)
- 3.1 示例展示：
translate('está muito frio aqui.')
运行结果：
Input: está muito frio aqui.
Predicted translation: it 's very cold here .
translate('isto é minha vida')
运行结果：
Input: isto é minha vida
Predicted translation: this is my life .
translate('você ainda está em casa?')
运行结果：
Input: você ainda está em casa?
Predicted translation: are you still in home ?
translate('este é o primeiro livro que eu já li')
运行结果：
Input: este é o primeiro livro que eu já li
Predicted translation: this is the first book i ever stand in the book .
这个结果可以发现翻译的结果和原句很像但是有一点不同，这说明我们的model并没有那么强大，原因：
- 使用的训练数据的序列比较小
- model并没有训练充分
translate('este é o primeiro livro que eu já li',
layer_name = 'decoder_layer4_att2')
-
4.1 总结
-
首先使用tfds载入一个数据，
-
使用SubwordTextEncoder给数据进行预处理，处理成subword的形式，处理成subword id，
-
然后使用dataset对word id进行处理，处理成dataset，包括word到id的转换，过滤数据
-
生成工具，包括生成position_embedding，create_padding_mask，create_look_ahead_mask
-
实现scaled_dot_product_attention函数，其中mask的实现是给logits加一个非常小的值，从而使得它在做softmax之后的值都是0
-
使用layer实现MultiHeadAttention
-
使用子类的方式实现EncoderLayer和DecoderLayer，并且使用堆叠的方式实现EncoderModel和DecoderModel，EncoderModel和DecoderModel区别于EncoderLayer和DecoderLayer的地方就是EncoderModel和DecoderModel是多个EncoderLayer和DecoderLayer，并且有一个word embedding的过程，早做完embedding之后做了dropout
-
有了EncoderModel和DecoderModel之后，搭建了Transformer Model，这个就是把EncoderModel和DecoderModel串起来
-
自定义learning rate的变化
-
四种mask的方式
-
训练模型
-
预测模型
-
可视化attention
