『深度应用』NLP机器翻译深度学习实战课程·壹（RNN base）

小宋是呢
1,142 阅读6分钟

深度学习用的有一年多了，最近开始 NLP 自然处理方面的研发。刚好趁着这个机会写一系列 NLP 机器翻译深度学习实战课程。

本系列课程将从原理讲解与数据处理深入到如何动手实践与应用部署，将包括以下内容：（更新 ing）

本系列教程参考博客: me.csdn.net/chinateleco…

开源地址：github.com/xiaosongshi…

个人主页：www.yansongsong.cn/

0.  项目背景

在上个文章中，我们已经简单介绍了 NLP 机器翻译，这次我们将用实战的方式讲解基于 RNN 的翻译模型。

0.1  基于 RNN 的 seq2seq 架构翻译模型介绍

seq2seq 结构

基于 RNN 的 seq2seq 架构包含 encoder 和 decoder，decoder 部分又分 train 和 inference 两个过程，具体结构如下面两图所示：

可以看出结构很简单（相较于 CNN 与 Attention base），下面我们就通过代码的形式实现，来进一步探究理解模型内在原理。

1.  数据准备

1.1 下载数据

 

此网站 www.manythings.org/anki/ 上有许多翻译数据，包含多种语言，这里此教程选择的是中文到英语的数据集。

训练下载地址：www.manythings.org/anki/cmn-en…

解压 cmn-eng.zip，可以找到 cmn.txt 文件，内容如下：

with open('cmn.txt', 'r', encoding='utf-8') as f:

['Tom died.\t汤姆去世了。', 'Tom quit.\t汤姆不干了。', 'Tom swam.\t汤姆游泳了。', 'Trust me.\t相信我。', 'Try hard.\t努力。']

可以发现，每对翻译数据在同一行，左边是英文，右边是中文使用 \t 作为英语与中文的分界。

1.2  数据预处理

使用网络训练，需要我们把数据处理成网络可以接收的格式。

针对这个数据，具体来说就是需要把字符转换为数字（句子数字化），句子长度归一化处理。

句子数字化

可以参考我的这博客：『深度应用』NLP 命名实体识别 (NER) 开源实战教程，数据预处理的实现。

分别对英语与汉字做处理。

英文处理

因为英语每个单词都是用空格分开的（除了缩写词，这里缩写词当做一个词处理），还有就是标点符号和单词没有分开，也需要特殊处理一下

这里我用的是一个简单方法处理下，实现在标点前加空格：

def split_dot(strs,dots=", . ! ?"):

for d in dots.split(" "):

        strs = strs.replace(d," "+d)

使用这个方法来把词个字典化：

for token in sample.split(" "):

if token not in w_all_dict.keys():

    sort_w_list = sorted(w_all_dict.items(),  key=lambda d: d[1], reverse=True)

    w_keys = [x for x,_ in sort_w_list[:7000-2]]

    w_dict = { x:i for i,x in enumerate(w_keys) }

    i_dict = { i:x for i,x in enumerate(w_keys) }

中文处理

在处理中文时可以发现，有繁体也有简体，所以最好转换为统一形式：（参考地址）

pip install opencc-python-reimplemented

使用方法，把繁体转换为简体：

cc = opencc.OpenCC('t2s')

再使用 jieba 分词的方法来从句子中分出词来：

def get_chn_dicts(datas):

for token in jieba.cut(sample):

if token not in w_all_dict.keys():

    sort_w_list = sorted(w_all_dict.items(),  key=lambda d: d[1], reverse=True)

    w_keys = [x for x,_ in sort_w_list[:10000-4]]

    w_dict = { x:i for i,x in enumerate(w_keys) }

    i_dict = { i:x for i,x in enumerate(w_keys) }

下面进行 padding

def padding(lists,lens=LENS):

最后统一运行处理一下：

if __name__ == "__main__":

    df = read2df("cmn-eng/cmn.txt")

    eng_dict,id2eng = get_eng_dicts(df["eng"])

    chn_dict,id2chn = get_chn_dicts(df["chn"])

    print(list(eng_dict.keys())[:20])

    print(list(chn_dict.keys())[:20])

    enc_in = [[get_val(e,eng_dict) for e in eng.split(" ")] for eng in df["eng"]]

    dec_in = [[get_val("<GO>",chn_dict)]+[get_val(e,chn_dict) for e in jieba.cut(eng)]+[get_val("<EOS>",chn_dict)] for eng in df["chn"]]

    dec_out = [[get_val(e,chn_dict) for e in jieba.cut(eng)]+[get_val("<EOS>",chn_dict)] for eng in df["chn"]]

    enc_in_ar = np.array(padding(enc_in,32))

    dec_in_ar = np.array(padding(dec_in,30))

    dec_out_ar = np.array(padding(dec_out,30))

输出结果如下： 

(TF_GPU) D:\Files\Prjs\Pythons\Kerases\MNT_RNN>C:/Datas/Apps/RJ/Miniconda3/envs/TF_GPU/python.exe d:/Files/Prjs/Pythons/Kerases/MNT_RNN/mian.py        

Using TensorFlow backend.

Building prefix dict from the default dictionary ...

Loading model from cache C:\Users\xiaos\AppData\Local\Temp\jieba.cache

Loading model cost 0.788 seconds.

Prefix dict has been built succesfully.

['<UNK>', '<PAD>', '.', 'I', 'to', 'the', 'you', 'a', '?', 'is', 'Tom', 'He', 'in', 'of', 'me', ',', 'was', 'for', 'have', 'The']

['<UNK>', '<PAD>', '<GO>', '<EOS>', '。', '我', '的', '了', '你', '他', '？', '在', '汤姆', '是', '她', '吗', '我们', '，', '不', '很']
  1. 构建模型与训练

2.1 构建模型与超参数

用的是双层 LSTM 网络

from keras.models import Model

from keras.layers import Input, LSTM, Dense, Embedding,CuDNNLSTM

from keras.optimizers import Adam

    encoder_inputs = Input(shape=(None,))

    emb_inp = Embedding(output_dim=128, input_dim=EN_VOCAB_SIZE)(encoder_inputs)

    encoder_h1, encoder_state_h1, encoder_state_c1 = CuDNNLSTM(HIDDEN_SIZE, return_sequences=True, return_state=True)(emb_inp)

    encoder_h2, encoder_state_h2, encoder_state_c2 = CuDNNLSTM(HIDDEN_SIZE, return_state=True)(encoder_h1)

    decoder_inputs = Input(shape=(None,))

    emb_target = Embedding(output_dim=128, input_dim=CH_VOCAB_SIZE)(decoder_inputs)

    lstm1 = CuDNNLSTM(HIDDEN_SIZE, return_sequences=True, return_state=True)

    lstm2 = CuDNNLSTM(HIDDEN_SIZE, return_sequences=True, return_state=True)

    decoder_dense = Dense(CH_VOCAB_SIZE, activation='softmax')

    decoder_h1, _, _ = lstm1(emb_target, initial_state=[encoder_state_h1, encoder_state_c1])

    decoder_h2, _, _ = lstm2(decoder_h1, initial_state=[encoder_state_h2, encoder_state_c2])

    decoder_outputs = decoder_dense(decoder_h2)

    model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

    encoder_model = Model(encoder_inputs, [encoder_state_h1, encoder_state_c1, encoder_state_h2, encoder_state_c2])

    decoder_state_input_h1 = Input(shape=(HIDDEN_SIZE,))

    decoder_state_input_c1 = Input(shape=(HIDDEN_SIZE,))

    decoder_state_input_h2 = Input(shape=(HIDDEN_SIZE,))

    decoder_state_input_c2 = Input(shape=(HIDDEN_SIZE,))

    decoder_h1, state_h1, state_c1 = lstm1(emb_target, initial_state=[decoder_state_input_h1, decoder_state_input_c1])

    decoder_h2, state_h2, state_c2 = lstm2(decoder_h1, initial_state=[decoder_state_input_h2, decoder_state_input_c2])

    decoder_outputs = decoder_dense(decoder_h2)

    decoder_model = Model([decoder_inputs, decoder_state_input_h1, decoder_state_input_c1, decoder_state_input_h2, decoder_state_input_c2], 

                        [decoder_outputs, state_h1, state_c1, state_h2, state_c2])

return(model,encoder_model,decoder_model)

2.2 模型配置与训练

自定义了一个 acc，便于显示效果，keras 内置的 acc 无法使用

import keras.backend as K

from keras.models import load_model

def my_acc(y_true, y_pred):

    acc = K.cast(K.equal(K.max(y_true,axis=-1),K.cast(K.argmax(y_pred,axis=-1),K.floatx())),K.floatx())

if __name__ == "__main__":

    df = read2df("cmn-eng/cmn.txt")

    eng_dict,id2eng = get_eng_dicts(df["eng"])

    chn_dict,id2chn = get_chn_dicts(df["chn"])

    print(list(eng_dict.keys())[:20])

    print(list(chn_dict.keys())[:20])

    enc_in = [[get_val(e,eng_dict) for e in eng.split(" ")] for eng in df["eng"]]

    dec_in = [[get_val("<GO>",chn_dict)]+[get_val(e,chn_dict) for e in jieba.cut(eng)]+[get_val("<EOS>",chn_dict)] for eng in df["chn"]]

    dec_out = [[get_val(e,chn_dict) for e in jieba.cut(eng)]+[get_val("<EOS>",chn_dict)] for eng in df["chn"]]

    enc_in_ar = np.array(padding(enc_in,32))

    dec_in_ar = np.array(padding(dec_in,30))

    dec_out_ar = np.array(padding(dec_out,30))

        model,encoder_model,decoder_model = get_model()

        model.load_weights('e2c1.h5')

        opt = Adam(lr=LEARNING_RATE, beta_1=0.9, beta_2=0.99, epsilon=1e-08)

        model.compile(optimizer=opt, loss='sparse_categorical_crossentropy',metrics=[my_acc])

        model.fit([enc_in_ar, dec_in_ar], np.expand_dims(dec_out_ar,-1),

        encoder_model.save("enc1.h5")

        decoder_model.save("dec1.h5")

64Epoch 训练结果如下：

__________________________________________________________________________________________________

Layer (type)                    Output Shape         Param #     Connected to

==================================================================================================

input_1 (InputLayer)            (None, None)         0

__________________________________________________________________________________________________

input_2 (InputLayer)            (None, None)         0

__________________________________________________________________________________________________

embedding_1 (Embedding)         (None, None, 128)    896000      input_1[0][0]

__________________________________________________________________________________________________

embedding_2 (Embedding)         (None, None, 128)    1280000     input_2[0][0]

__________________________________________________________________________________________________

cu_dnnlstm_1 (CuDNNLSTM)        [(None, None, 256),  395264      embedding_1[0][0]

__________________________________________________________________________________________________

cu_dnnlstm_3 (CuDNNLSTM)        [(None, None, 256),  395264      embedding_2[0][0]

__________________________________________________________________________________________________

cu_dnnlstm_2 (CuDNNLSTM)        [(None, 256), (None, 526336      cu_dnnlstm_1[0][0]

__________________________________________________________________________________________________

cu_dnnlstm_4 (CuDNNLSTM)        [(None, None, 256),  526336      cu_dnnlstm_3[0][0]

__________________________________________________________________________________________________

dense_1 (Dense)                 (None, None, 10000)  2570000     cu_dnnlstm_4[0][0]

==================================================================================================

__________________________________________________________________________________________________

19004/19004 [==============================] - 98s 5ms/step - loss: 0.1371 - my_acc: 0.9832 - val_loss: 2.7299 - val_my_acc: 0.7412

19004/19004 [==============================] - 96s 5ms/step - loss: 0.1234 - my_acc: 0.9851 - val_loss: 2.7378 - val_my_acc: 0.7410

19004/19004 [==============================] - 96s 5ms/step - loss: 0.1132 - my_acc: 0.9867 - val_loss: 2.7477 - val_my_acc: 0.7419

19004/19004 [==============================] - 96s 5ms/step - loss: 0.1050 - my_acc: 0.9879 - val_loss: 2.7660 - val_my_acc: 0.7426

19004/19004 [==============================] - 96s 5ms/step - loss: 0.0983 - my_acc: 0.9893 - val_loss: 2.7569 - val_my_acc: 0.7408

19004/19004 [==============================] - 96s 5ms/step - loss: 0.0933 - my_acc: 0.9903 - val_loss: 2.7775 - val_my_acc: 0.7414

19004/19004 [==============================] - 96s 5ms/step - loss: 0.0885 - my_acc: 0.9911 - val_loss: 2.7885 - val_my_acc: 0.7420

19004/19004 [==============================] - 96s 5ms/step - loss: 0.0845 - my_acc: 0.9920 - val_loss: 2.7914 - val_my_acc: 0.7423
  1. 模型应用与预测

从训练集选取部分数据进行测试

if __name__ == "__main__":

df = read2df("cmn-eng/cmn.txt")

eng_dict,id2eng = get_eng_dicts(df["eng"])

chn_dict,id2chn = get_chn_dicts(df["chn"])

print(list(eng_dict.keys())[:20])

print(list(chn_dict.keys())[:20])

enc_in = [[get_val(e,eng_dict) for e in eng.split(" ")] for eng in df["eng"]]

dec_in = [[get_val("<GO>",chn_dict)]+[get_val(e,chn_dict) for e in jieba.cut(eng)]+[get_val("<EOS>",chn_dict)] for eng in df["chn"]]

dec_out = [[get_val(e,chn_dict) for e in jieba.cut(eng)]+[get_val("<EOS>",chn_dict)] for eng in df["chn"]]

enc_in_ar = np.array(padding(enc_in,32))

dec_in_ar = np.array(padding(dec_in,30))

dec_out_ar = np.array(padding(dec_out,30))

encoder_model,decoder_model = load_model("enc1.h5",custom_objects={"my_acc":my_acc}),load_model("dec1.h5",custom_objects={"my_acc":my_acc})

for k in range(16000-20,16000):

test_data = enc_in_ar[k:k+1]

h1, c1, h2, c2 = encoder_model.predict(test_data)

target_seq = np.zeros((1,1))

target_seq[0, len(outputs)] = chn_dict["<GO>"]

output_tokens, h1, c1, h2, c2 = decoder_model.predict([target_seq, h1, c1, h2, c2])

sampled_token_index = np.argmax(output_tokens[0, -1, :])

outputs.append(sampled_token_index)

target_seq[0, 0] = sampled_token_index

if sampled_token_index == chn_dict["<EOS>"] or len(outputs) > 28: break

print("< "+' '.join([id2chn[i] for i in outputs[:-1]]))

测试结果如下：基本上都翻译正确了。

> I can understand you to some extent .

> I can't recall the last time we met .

< 我 想不起来 我们 上次 见面 的 情况 了 。

> I can't remember which is my racket .

> I can't stand that noise any longer .

> I can't stand this noise any longer .

> I caught the man stealing the money .

> I could not afford to buy a bicycle .

> I couldn't answer all the questions .

> I couldn't think of anything to say .

> I cry every time I watch this movie .

> I did not participate in the dialog .

> I didn't really feel like going out .

> I don't care a bit about the future .
avatar
文章
阅读
粉丝