Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Motivation

The paper "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation" by Kyunghyun Cho et al. proposes a novel approach to machine translation using a sequence-to-sequence model with an RNN (Recurrent Neural Network) encoder-decoder architecture. The authors introduce the Gated Recurrent Unit (GRU) as a key component, demonstrating its effectiveness in capturing and representing phrases in a source language for translation into a target language.

The main idea revolves around the use of a neural network to encode an input sequence (source language) into a fixed-size vector representation, which is then decoded into the target sequence. This end-to-end learning approach allows the model to automatically learn complex mappings between source and target languages without the need for manual feature engineering or alignment.

One intriguing aspect of this paper lies in the broader context of the evolution of neural machine translation (NMT). The big question that arises is: How can the lessons learned from this paper, which introduces a pioneering approach to sequence-to-sequence modeling, inform and inspire further advancements in the field?

This question encompasses various aspects, including the exploration of more advanced architectures beyond RNNs, the investigation of attention mechanisms, and the development of models that can handle nuances, idioms, and cultural subtleties in translation. Additionally, understanding the transferability of the proposed methods to other natural language processing tasks and the integration of external knowledge sources for improved translation quality are areas of interest.

In essence, the big question revolves around how the fundamental principles and innovations introduced in this paper can guide and shape the future of machine translation and broader applications of sequence-to-sequence models in natural language processing.

Literature Review

The development of Recurrent Neural Networks (RNNs) has been marked by several key milestones, with various technical papers contributing to their evolution. Here's a brief history of RNNs in terms of significant papers:

1. Jordan (1986): "Attractor Dynamics and Parallelism in a Connectionist Sequential Machine"
   Michael Jordan's work laid the foundation for the idea of using recurrent connections in neural networks. He introduced the concept of attractor dynamics and explored the potential of recurrent connections for sequence processing.
2. Elman (1990): "Finding Structure in Time"
   Jeffrey L. Elman proposed the Elman network, a type of simple recurrent neural network. Elman networks introduced the idea of a context layer, allowing them to capture dependencies over short time lags.
3. Hochreiter and Schmidhuber (1997): "Long Short-Term Memory"
   Sepp Hochreiter and Jurgen Schmidhuber introduced Long Short-Term Memory (LSTM) networks to address the vanishing gradient problem in traditional RNNs. LSTMs incorporate memory cells and gating mechanisms to better capture long-term dependencies.
4. Schmidhuber (1992): "Learning Composed Representations with Neural Networks"
   Jurgen Schmidhuber's work on learning composed representations helped establish the importance of neural networks with memory, paving the way for more sophisticated recurrent architectures.
5. Graves et al. (2009): "Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks"
   Graves and Schmidhuber introduced a multidimensional recurrent neural network, showcasing the potential of RNNs for sequential data processing. This work demonstrated improved performance in offline handwriting recognition tasks.
6. Graves et al. (2013): "Speech Recognition with Deep Recurrent Neural Networks"
   This paper by Graves et al. demonstrated the effectiveness of deep recurrent neural networks, specifically LSTMs, for speech recognition. The deep RNNs showed improved performance over traditional models.

Kyunghyun Cho

Professor at CILVR Group, NYU (2015-Present)
Co-Founder & Senior Director of Frontier Research at Genentech Research and Early Development (2021-Present)
Research scientist at Facebook AI Research (2017-2020)
Postdoctoral fellow at Universite de Montreal under the supervision of Prof. Yoshua Bengio (2014-2015)

Bart van Merrienboer

Research Scientist at Google's DeepMind (2019-Present)
Postdoctoral fellow at University of Montreal (2014-2019)

Caglar Gulcehre

Assistant Professor at EPFL (Ecole polytechnique federale de Lausanne) (2023-Present)
Research Scientist at Google's DeepMind (2017-2023)
Postdoctoral under the supervision of Prof. Yoshua Bengio at Mila (2012-2017)
Work related to reinforcement learning, deep learning and natural language understanding

Dzmitry Bahdanau

Adjunct Professor at McGill University (Present)
Research Scientist at ServiceNow Element AI (2021-Present)
PhD at Mila and Universite de Montreal working with Yoshua Bengio (2014-2020)
Master's from Jacobs University Bremen (2013-2015)

Fethi Bougares

Head of Research at ELYADATA (2021-Present)
Associate Professor, University of Le Mans (2013-2021)
Postdoctoral from Universite du Maine, France (2009-2013)
Work related to Deep Learning, NLP, Machine Translation, Speech Recognition, Machine Learning

Holger Schwenk

Research scientist at Facebook AI Research, Paris (2015-Present)
Spent one year at the University of Montreal working with Y. Bengio (2014)
Awarded senior member of the Institut Universitaire de France (2013)
Professor of computer science at the University of Le Mans
PhD in computer science from the University of Paris (1996)

Yoshua Bengio

Founder and Scientific Director of Mila - Quebec AI Institute
Won the Turing Award (2018)
Professor at Universite de Montreal (2013-Present)
Co-directs the CIFAR Learning in Machines & Brains
Postdoctoral from MIT (1991-1992)
Machine Learning Researcher, Brain and cognitive sciences department, AI lab at MIT ((1991-1992))
PhD in Computer Science from McGill University (1988-1991)

The Gated Recurrent Unit (GRU) is a type of recurrent neural network (RNN) architecture designed to address the vanishing gradient problem in traditional RNNs. It consists of a set of recurrent units that maintain a hidden state and selectively update this state using gating mechanisms. The key components are the reset gate and update gate. The reset gate determines how much of the past information to forget, while the update gate controls how much of the new information to incorporate into the current hidden state. GRUs thus enable the model to capture long-range dependencies in sequential data by selectively updating and forgetting information, allowing for more effective learning and representation of temporal patterns. This architecture strikes a balance between the simplicity of traditional RNNs and the more complex Long Short-Term Memory (LSTM) networks, offering a computationally efficient solution for sequential data processing tasks.

The Recurrent Neural Network (RNN) encoder-decoder architecture is a framework commonly used for sequence-to-sequence tasks, such as machine translation or text summarization. The encoder processes the input sequence step by step, producing a fixed-size context vector that captures the input sequence's information. Each step of the encoder involves updating its hidden state based on the input at that time step. Once the entire input sequence is encoded, the decoder generates the output sequence step by step, utilizing the context vector produced by the encoder. Similar to the encoder, the decoder maintains a hidden state that evolves with each generated output. At each time step, the decoder combines its current hidden state with the context vector to produce the output. This architecture enables the model to handle input and output sequences of varying lengths, making it versatile for tasks involving sequential data. The Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are often used as building blocks for the encoder-decoder architecture to address vanishing gradient problems and capture long-range dependencies in the data.

We start with calculating the update gate z_t for time step t using the formula:

Update gate equation

When x_t is plugged into the network unit, it is multiplied by its own weight W(z). The same goes for h_(t-1) which holds the information for the previous t-1 units and is multiplied by its own weight U(z). Both results are added together and a sigmoid activation function is applied to squash the result between 0 and 1.

The update gate helps the model to determine how much of the past information (from previous time steps) needs to be passed along to the future. That is really powerful because the model can decide to copy all the information from the past and eliminate the risk of vanishing gradient problem. We will see the usage of the update gate later on. For now remember the formula for z_t.

Essentially, this gate is used from the model to decide how much of the past information to forget. To calculate it, we use:

Reset gate equation

This formula is the same as the one for the update gate. The difference comes in the weights and the usage of gate.

Let us see how exactly the gates will affect the final output. First, we start with the usage of the reset gate. We introduce a new memory content which will use the reset gate to store the relevant information from the past. It is calculated as follows:

New hidden state equation

Multiply the input x_t with a weight W and h_(t-1) with a weight U.

Calculate the Hadamard (element-wise) product between the reset gate r_t and Uh_(t-1). That will determine what to remove from the previous time steps. Let us say we have a sentiment analysis problem for determining opinion of one about a book from a review he wrote. The text starts with This is a fantasy book which illustrates and after a couple paragraphs ends with I didn’t quite enjoy the book because I think it captures too many details. To determine the overall level of satisfaction from the book we only need the last part of the review. In that case as the neural network approaches to the end of the text it will learn to assign r_t vector close to 0, washing out the past and focusing only on the last sentences.

Sum up the results of step 1 and 2.

Apply the nonlinear activation function tanh.

As the last step, the network needs to calculate h_t vector which holds information for the current unit and passes it down to the network. In order to do that the update gate is needed. It determines what to collect from the current memory content h_t and what from the previous steps h_(t-1). That is done as follows:

Final hidden state equation

Apply element-wise multiplication to the update gate z_t and h_(t-1).

Apply element-wise multiplication to (1-z_t) and h_t.

Sum the results from step 1 and 2.

Let us bring up the example about the book review. This time, the most relevant information is positioned in the beginning of the text. The model can learn to set the vector z_t close to 1 and keep a majority of the previous information. Since z_t will be close to 1 at this time step, 1-z_t will be close to 0 which will ignore big portion of the current content (in this case the last part of the review which explains the book plot) which is irrelevant for our prediction.

Now, you can see how GRUs are able to store and filter the information using their update and reset gates. That eliminates the vanishing gradient problem since the model is not washing out the new input every single time but keeps the relevant information and passes it down to the next time steps of the network. If carefully trained, they can perform extremely well even in complex scenarios.

WMT 2014 is a workshop of statistical machine translation; it has a collection of datasets. The authors use english-french dataset for translation tasks. The baseline phrase-based SMT system was built using Moses software with its default settings and the BLEU scores were calculated. The Bilingual Evaluation Understudy Score, or BLEU for short, is a metric for evaluating a generated sentence to a reference sentence. A perfect match results in a score of 1, whereas a perfect mismatch results in a score of 0. Scores greater than 1 usually mean a better match.

The proposed RNN Encoder-Decoder model has 1000 hidden units with the update and reset gates in the encoder and decoder. A rank-100 matrix, equivalent to learning an embedding of dimension 100 for each word was used. The activation function used for h in Eq. (8) is a hyperbolic tangent function. All the weight parameters were initialized by sampling from an isotropic zero-mean Gaussian distribution with a standard deviation of 0.01, except for the recurrent weight parameters. For the recurrent weight matrices, they sampled from a white Gaussian distribution and used its left singular vectors matrix.

During training this RNN, frequency of phrase pairs are ignored in order to promote learning of linguistic regularities. Due to this, the model is expected to

1. reduce the computational expense of randomly selecting phrase pairs from a large phrase table according to the normalized frequencies
2. ensure that the RNN Encoder-Decoder does not simply learn to rank the phrase pairs according to their numbers of occurrences

The table above shows the samples generated from the RNN Encoder–Decoder for each source phrase sorted by the scores.

To assess the performance of the proposed architecture, the authors also trained a more traditional CSLM (continuous space language model) on 7-grams. By doing this, they aimed to clarify whether the contributions from multiple neural networks in different parts of the SMT system add up or are redundant. CSLM captures the frequency of the words as well while the RNN encoder-decoder model mainly learns linguistic characteristics.
As shown in the table we see an improvement when the RNN model and CSLM model are both used; suggesting contributions from both models are not correlated.

Penalizing the unknown words using word penalty (WP) in the final model does not improve performance on test data as shown in the table above.

The RNN Encoder-Decoder projects a sequence of words to and from a continuous vector space. Word and phrase representations are plotted to understand if the model is learning semantically and syntactically.

The bottom plot shows the 2d word embeddings learnt by the RNN model; clear clusters of semantically similar words are observed.

Similarly for phrases, both semantic and syntactic structures are captured. Both red and pink colour-coded plots at the bottom show semantically similar phrases related to time and countries respectively while the plot on the top right blue colour-coded shows syntactically similar phrases.