在实际情况下，许多问题的建模都与时间序列信息有关。特别是在自然语言处理的许多任务中，数据间的上下文非常依赖。考虑到这类问题的建模，Elman出循环神经网络(Recurrent Neural Network,RNN)，其结构如图所示。显然，循环神经网络的输入输出之间有一个循环过程。循环过程扩展后，可以发现循环神经网络的隐藏层节点关系基于两个输入，一个是当前时刻的标准输入，另一个是上一个节点的隐藏层信息，反映为： 其中，o代表循环神经单元的输出值，h代表循环神经单元的隐藏层值。显然，循环神经元有两个输出，不同于全连接网络。此外，循环神经网络中的神经网络中的参数，即同一层输入中使用的参数相同。这种参数共享不仅可以大大降低模型的参数，还可以提高模型的泛化能力，特别是在自然语言处理的相关问题上。当模型接收到超过训练样本长度的输入时，模型仍然可以 这种参数共享也给模型误差的传播带来了一些障碍。目前，对RNN训练采用时间方向传播的方法(Back-propagationThrough Time,BPTT)，从图2-2右侧的展开式可以看出，展开后RNN时序的深度取决于序列的长度BPTT该算法的求导链与此长度密切相关。因此，当序列变长时，BPTT梯度消失和梯度爆炸有两个问题。梯度消失是指BPTT反馈到一定长度后，梯度接近零，无法学习，而梯度爆炸恰恰相反，这意味着梯度呈现大值，导致长程神经元学习无用。无论BPTT任何问题都会使序列中的上下文关系无法反映和偏离RNN构建模序列关系的初衷。目前解决这两个问题的主流解决方案是RNN在神经元上增加门控机制来控制数据流，确保有用数据的传输，其中最著名的是长期和短期记忆单元(longshort-Term Memory,LSTM)和门控循环单元(Gated Recurrent Units,GRU)。 长短期记忆单元LSTM在普通的RNN在神经元中增加了三个门控制单元来控制数据流，可以生动地称为输入门、遗忘门和输出门，三个门对应三个平行的全连接层。最后，输出由三扇门的结果与前一个神经元的状态集成，具体公式如下： 其中，i,f,o输入门、遗忘门和输出门分别对应，c表示神经元的细胞状态，h对于当前神经元的隐藏状态值，也是当前神经元的输出，它融合了三门的结果和之前神经元的状态信息。W{i,f,o,c},U{i,f,o,c},b{i,f,o,c}对应不同门的可学习参数，同时，和普通的RNN所有神经元都共享这些参数。

因此，当序列被采用时LSTM作为基本单本单元RNN当时，门控单元将控制序列信息流的传输，以确保有用信息的传输，并在一定程度上减少模型学习的无用问题。但LSTM在神经元内部进行的一系列全连接计算也导致了模型效率的降低。Cho等人在基于LSTM在简化门控单元的基础上， 变体结构：门控循环单元GRU。在GRU门控单元减少到两个：重置门和更新门(z)，其结构由图纸给出，计算如下： 不难发现，GRU直接使用重置门刷新前一刻神经元的隐藏状态，然后整合状态和当前输入值获得新的细胞状态，更新门根据前一刻的细胞状态和隐藏状态控制数据的传输。由于GRU显式减少了门控单元，因此GRU所需参数较多LSTM序列任务的计算效率较低LSTM性能损失较高。

from __future__ import absolute_import from recurrentshop import LSTMCell, RecurrentSequential from .cells import LSTMDecoderCell, AttentionDecoderCell from keras.models import Sequential, Model from keras.layers import Dense, Dropout, TimeDistributed, Bidirectional, Input def SimpleSeq2Seq(output_dim, output_length, hidden_dim=None, input_shape=None,                   batch_size=None, batch_input_shape=None, input_dim=None,
                  input_length=None, depth=1, dropout=0.0, unroll=False,
                  stateful=False):

    ''' Simple model for sequence to sequence learning. The encoder encodes the input sequence to vector (called context vector) The decoder decodes the context vector in to a sequence of vectors. There is no one on one relation between the input and output sequence elements. The input sequence and output sequence may differ in length. Arguments: output_dim : Required output dimension. hidden_dim : The dimension of the internal representations of the model. output_length : Length of the required output sequence. depth : Used to create a deep Seq2seq model. For example, if depth = 3, there will be 3 LSTMs on the enoding side and 3 LSTMs on the decoding side. You can also specify depth as a tuple. For example, if depth = (4, 5), 4 LSTMs will be added to the encoding side and 5 LSTMs will be added to the decoding side. dropout : Dropout probability in between layers. '''

    if isinstance(depth, int):
        depth = (depth, depth)
    if batch_input_shape:
        shape = batch_input_shape
    elif input_shape:
        shape = (batch_size,) + input_shape
    elif input_dim:
        if input_length:
            shape = (batch_size,) + (input_length,) + (input_dim,)
        else:
            shape = (batch_size,) + (None,) + (input_dim,)
    else:
        # TODO Proper error message
        raise TypeError
    if hidden_dim is None:
        hidden_dim = output_dim
    encoder = RecurrentSequential(unroll=unroll, stateful=stateful)
    encoder.add(LSTMCell(hidden_dim, batch_input_shape=(shape[0], shape[-1])))

    for _ in range(1, depth[0]):
        encoder.add(Dropout(dropout))
        encoder.add(LSTMCell(hidden_dim))

    decoder = RecurrentSequential(unroll=unroll, stateful=stateful,
                                  decode=True, output_length=output_length)
    decoder.add(Dropout(dropout, batch_input_shape=(shape[0], hidden_dim)))

    if depth[1] == 1:
        decoder.add(LSTMCell(output_dim))
    else:
        decoder.add(LSTMCell(hidden_dim))
        for _ in range(depth[1] - 2):
            decoder.add(Dropout(dropout))
            decoder.add(LSTMCell(hidden_dim))
    decoder.add(Dropout(dropout))
    decoder.add(LSTMCell(output_dim))

    _input = Input(batch_shape=shape)
    x = encoder(_input)
    output = decoder(x)
    return Model(_input, output)


def Seq2Seq(output_dim, output_length, batch_input_shape=None,
            input_shape=None, batch_size=None, input_dim=None, input_length=None,
            hidden_dim=None, depth=1, broadcast_state=True, unroll=False,
            stateful=False, inner_broadcast_state=True, teacher_force=False,
            peek=False, dropout=0.):

    ''' Seq2seq model based on [1] and [2]. This model has the ability to transfer the encoder hidden state to the decoder's hidden state(specified by the broadcast_state argument). Also, in deep models (depth > 1), the hidden state is propogated throughout the LSTM stack(specified by the inner_broadcast_state argument. You can switch between [1] based model and [2] based model using the peek argument.(peek = True for [2], peek = False for [1]). When peek = True, the decoder gets a 'peek' at the context vector at every timestep. [1] based model: Encoder: X = Input sequence C = LSTM(X); The context vector Decoder: y(t) = LSTM(s(t-1), y(t-1)); Where s is the hidden state of the LSTM (h and c) y(0) = LSTM(s0, C); C is the context vector from the encoder. [2] based model: Encoder: X = Input sequence C = LSTM(X); The context vector Decoder: y(t) = LSTM(s(t-1), y(t-1), C) y(0) = LSTM(s0, C, C) Where s is the hidden state of the LSTM (h and c), and C is the context vector from the encoder. Arguments: output_dim : Required output dimension. hidden_dim : The dimension of the internal representations of the model. output_length : Length of the required output sequence. depth : Used to create a deep Seq2seq model. For example, if depth = 3, there will be 3 LSTMs on the enoding side and 3 LSTMs on the decoding side. You can also specify depth as a tuple. For example, if depth = (4, 5), 4 LSTMs will be added to the encoding side and 5 LSTMs will be added to the decoding side. broadcast_state : Specifies whether the hidden state from encoder should be transfered to the deocder. inner_broadcast_state : Specifies whether hidden states should be propogated throughout the LSTM stack in deep models. peek : Specifies if the decoder should be able to peek at the context vector at every timestep. dropout : Dropout probability in between layers. '''

    if isinstance(depth, int):
        depth = (depth, depth)
    if batch_input_shape:
        shape = batch_input_shape
    elif input_shape:
        shape = (batch_size,) + input_shape
    elif input_dim:
        if input_length:
            shape = (batch_size,) + (input_length,) + (input_dim,)
        else:
            shape = (batch_size,) + (None,) + (input_dim,)
    else:
        # TODO Proper error message
        raise TypeError
    if hidden_dim is None:
        hidden_dim = output_dim

    encoder = RecurrentSequential(readout=True, state_sync=inner_broadcast_state,
                                  unroll=unroll, stateful=stateful,
                                  return_states=broadcast_state)
    for _ in range(depth[0]):
        encoder.add(LSTMCell(hidden_dim, batch_input_shape=(shape[0], hidden_dim)))
        encoder.add(Dropout(dropout))

    dense1 = TimeDistributed(Dense(hidden_dim))
    dense1.supports_masking = True
    dense2 = Dense(output_dim)

    decoder = RecurrentSequential(readout='add' if peek else 'readout_only',
                                  state_sync=inner_broadcast_state, decode=True,
                                  output_length=output_length, unroll=unroll,
                                  stateful=stateful, teacher_force=teacher_force)

    for _ in range(depth[1]):
        decoder.add(Dropout(dropout, batch_input_shape=(shape[0], output_dim)))
        decoder.add(LSTMDecoderCell(output_dim=output_dim, hidden_dim=hidden_dim,
                                    batch_input_shape=(shape[0], output_dim)))

    _input = Input(batch_shape=shape)
    _input._keras_history[0].supports_masking = True
    encoded_seq = dense1(_input)
    encoded_seq = encoder(encoded_seq)
    if broadcast_state:
        assert type(encoded_seq) is list
        states = encoded_seq[-2:]
        encoded_seq = encoded_seq[0]
    else:
        states = None
    encoded_seq = dense2(encoded_seq)
    inputs = [_input]
    if teacher_force:
        truth_tensor = Input(batch_shape=(shape[0], output_length, output_dim))
        truth_tensor._keras_history[0].supports_masking = True
        inputs += [truth_tensor]


    decoded_seq = decoder(encoded_seq,
                          ground_truth=inputs[1] if teacher_force else None,
                          initial_readout=encoded_seq, initial_state=states)
    
    model = Model(inputs, decoded_seq)
    model.encoder = encoder
    model.decoder = decoder
    return model


def AttentionSeq2Seq(output_dim, output_length, batch_input_shape=None,
                     batch_size=None, input_shape=None, input_length=None,
                     input_dim=None, hidden_dim=None, depth=1,
                     bidirectional=True, unroll=False, stateful=False, dropout=0.0,):
    ''' This is an attention Seq2seq model based on [3]. Here, there is a soft allignment between the input and output sequence elements. A bidirection encoder is used by default. There is no hidden state transfer in this model. The math: Encoder: X = Input Sequence of length m. H = Bidirection_LSTM(X); Note that here the LSTM has return_sequences = True, so H is a sequence of vectors of length m. Decoder: y(i) = LSTM(s(i-1), y(i-1), v(i)); Where s is the hidden state of the LSTM (h and c) and v (called the context vector) is a weighted sum over H: v(i) = sigma(j = 0 to m-1) alpha(i, j) * H(j) The weight alpha[i, j] for each hj is computed as follows: energy = a(s(i-1), H(j)) alpha = softmax(energy) Where a is a feed forward network. '''

    if isinstance(depth, int):
        depth = (depth, depth)
    if batch_input_shape:
        shape = batch_input_shape
    elif input_shape:
        shape = (batch_size,) + input_shape
    elif input_dim:
        if input_length:
            shape = (batch_size,) + (input_length,) + (input_dim,)
        else:
            shape = (batch_size,) + (None,) + (input_dim,)
    else:
        # TODO Proper error message
        raise TypeError
    if hidden_dim is None:
        hidden_dim = output_dim

    _input = Input(batch_shape=shape)
    _input._keras_history[0].supports_masking = True

    encoder = RecurrentSequential(unroll=unroll, stateful=stateful,
                                  return_sequences=True)
    encoder.add(LSTMCell(hidden_dim, batch_input_shape=(shape[0], shape[2])))

    for _ in range(1, depth[0]):
        encoder.add(Dropout(dropout))
        encoder.add(LSTMCell(hidden_dim))

    if bidirectional:
        encoder = Bidirectional(encoder, merge_mode='sum')
        encoder.forward_layer.build(shape)
        encoder.backward_layer.build(shape)
        # patch
        encoder.layer = encoder.forward_layer

    encoded = encoder(_input)
    decoder = RecurrentSequential(decode=True, output_length=output_length,
                                  unroll=unroll, stateful=stateful)
    decoder.add(Dropout(dropout, batch_input_shape=(shape[0], shape[1], hidden_dim)))
    if depth[1] == 1:
        decoder.add(AttentionDecoderCell(output_dim=output_dim, hidden_dim=hidden_dim))
    else:
        decoder.add(AttentionDecoderCell(output_dim=output_dim, hidden_dim=hidden_dim))
        for _ in range(depth[1] - 2):
            decoder.add(Dropout(dropout))
            decoder.add(LSTMDecoderCell(output_dim=hidden_dim, hidden_dim=hidden_dim))
        decoder.add(Dropout(dropout))
        decoder.add(LSTMDecoderCell(output_dim=output_dim, hidden_dim=hidden_dim))
    
    inputs = [_input]
    decoded = decoder(encoded)
    model = Model(inputs, decoded)
    return model

和循环神经网络 出的原因一样，卷积神经网络(Convolutional Neural Network,CNN) 出也是为了针对性的处理问题。与循环神经网络的 出是针对序列关系建模不同的是，卷积神经网络最初的 出是为了高效处理非序列数据问题，例如取图像中的特征问题。具体地，图像由于其具有空间三维性且包含的数据量过大，使用全连接网络会导致网络庞大且效率低下，为了 高神经网络对图像等类似数据的高效处理，基于局部特征取的卷积神经网络被出。它通过模拟生物捕捉图像特征的方法，对输入进行局部特征的 取，再依靠深度网络对高层特征的捕捉能力实现对图像的特征取。后来卷积神经网络被证明也可应用于自然语言处理任务，并且效率更高。一个完整的卷积神经网络包含三个部分，卷积层、池化层、全连接层。 卷积层是卷积神经网络的核心计算层，用于取输入的特征。一个卷积层由多个卷积核(Kernel)组成，每个卷积核负责取输入的一部分局部特征，其深度和输入相同，但高度和宽度都远小于输入。一次卷积计算，卷积核仅能和在其视野内，也就是在其高度和宽度范围内的输入数据交互，这个视野也被称之为感受野，是由人为设定的。通过滑动，卷积核的感受野不断变换，从而完成和输入的所有数据交互。感受野滑动的幅度称之为步长，是一个人为设定的经验值，一般设为1。卷积的计算过程用公式表示如下： 其中，h[m,n]代表尺寸为m×n的输入经过卷积后得到的隐藏层输出值，卷积核集合为K×L,一次卷积从中取出一个k×l的卷积核进行操作，I为输入数据，可以是输入源数据，也可以是前一隐藏层的输出数据。σ代表激活函数。图演示了一个卷积核尺寸为3×3的一次卷积计算示例。

显然，若想获得输入的更多特征，则需要设定多种不同的卷积核去捕捉输入，这样仍然会使得模型庞大。为了减少网络的参数量，卷积神经网络引入下采样操作周期性的在卷积层之后进行操作，被称为池化层。池化层的计算方法一般有两种，最大池化(Max Pooling)和平均池化(Average Pooling)，顾名思义，最大池化即是在池化区域中取最大值作为输出，而平均池化是取池化区域中的平均值作为输出。同卷积层操作一样，池化层也需要人工设定工作区域和移动步长，图2演示了一个池化区域为2×2，在步长为2时的最大池化/平均池化示例。 卷积神经网络通过堆叠卷积层和池化层获取到输入的特征，再将该特征拉伸为向量后通过一个全连接层后输出整个网络的结果。