PhD

sentence.tex (27410B)

---

 \section{Distributed Representation of Sentences}
 \label{sec:context:sentence}
 Most \textsc{nlp} tasks are tackled at the sentence level.
 In the previous section, we saw how to obtain representations of words.
 We now focus on how to aggregate these word representations in order to process whole sentences.
 Henceforth, given a sentence of length \(m\), we assume symbolic words \(\vctr{w}\in V^m\) are embedded as \(\mtrx{X}\in\symbb{R}^{m\times d}\) in a vector space of dimension \(d\).
 This can be achieved through the use of an embedding matrix \(\mtrx{U}\in\symbb{R}^{V\times d}\) such as the one provided by word2vec.
 
 An early approach to sentence representation was to use \emph{bag-of-words}, that is to simply ignore the ordering of the words.
 In this section, we focus on more modern, deep learning approaches.
 Section~\ref{sec:context:cnn} presents \textsc{cnn}s, which process fixed-length sequences of words to produce representations of sentences.
 We then focus on \textsc{rnn}s in Section~\ref{sec:context:rnn}, a method to get representations of sentences through a causal language model.
 \textsc{rnn}s can be improved by an attention mechanism as explained in Section~\ref{sec:context:attention}.
 Finally, we present transformers in Section~\ref{sec:context:transformers}, which build upon the concept of attention to extract state-of-the-art contextualized word representations.
 
 \subsection{Convolutional Neural Network}
 \label{sec:context:cnn}
 \begin{marginfigure}[-45mm]
 	\centering
 	\input{mainmatter/context/cnn.tex}
 	\scaption[Architecture of a single convolutional filter with a pooling layer.]{
 		Architecture of a single convolutional filter with a pooling layer.
 		The filter is of width 3, which means it works on trigrams.
 		A single filter (the \(i\)-th) is shown here, this is repeated \(d'\) times, meaning that \(\vctr{h}_t,\vctr{o}\in\symbb{R}^{d'}\).}
 	\label{fig:context:cnn}
 \end{marginfigure}
 
 Convolutional neural networks (\textsc{cnn}) can be used to build the representation of a sentence from the representation of its constituting words~\parencite{unified_nlp,cnn_classification}.
 These words embeddings can come from word2vec (Section~\ref{sec:context:word2vec}) or can be learned using a \textsc{cnn} with a language model objective (Section~\ref{sec:context:language model}), the latter being the original approach proposed by \textcitex{unified_nlp}.
 
 The basic idea behind \textsc{cnn} is to recognize patterns in a position-invariant fashion~\parencite{tdnn}.
 This is applicable to natural language following the principle of compositionality: the words composing an expression and the rules used to combine them determine its meaning, with little influence from the location of the expression in the text.
 So, given a sequence of \(d\)-dimensional embeddings \(\vctr{x}_1, \dotsc, \vctr{x}_m\in\symbb{R}^d\), a one dimensional \textsc{cnn} works on the \(n\)-grams of the sequence, that is the subwords%
 \sidenote{Here we use \emph{subwords} in its formal language theory meaning. In the simple setting where we deal with words in a sentence, this \emph{subword} actually designates a sequence of consecutive words.}
 \(\vctr{x}_{t:t+n-1} = (\vctr{x}_t, \dotsc, \vctr{x}_{t+n-1})\) of length \(n\).
 The basic design of a \textsc{cnn} is illustrated in Figure~\ref{fig:context:cnn}.
 A convolutional layer is parametrized by \(d'\) filters \(\mtrx{W}^{(i)}\in\symbb{R}^{n\times d}\) of width \(n\) and a bias \(b^{(i)}\in\symbb{R}\).
 The \(t\)-th output of the \(i\)-th filter layer is defined as:
 \begin{equation}
 	h^{(i)}_t = f(\mtrx{W}^{(i)} * \vctr{x}_{t:t+n-1} + b^{(i)})
 	\label{eq:context:convolution}
 \end{equation}
 where \(*\) is the convolution operator%
 \sidenote{
 	Usually, a cross-correlation operator is actually used, which is equivalent up to a mirroring of the filters when they are real-valued.
 }
 and \(f\) is a non-linear function.
 As is usual with neural networks, several layers of this kind can be stacked.
 To obtain a fixed-size representation---which does not depend on the length of the sequence \(m\)---a pooling layer can be used.
 Most commonly, max-over-time pooling~\parencite{maxpool}, which simply takes the maximum activation over time---that is sequence length---for each feature \(i=1, \dotsc, d'\).
 
 In the same way that word2vec produces a real vector space where words with similar meanings are close to each other, the sentence representations \(\vctr{o}\) extracted by a \textsc{cnn} tend to be close to each other when the sentences convey similar meanings.
 This is somewhat dependent on the task on which the \textsc{cnn} is trained.
 However, the purpose of \textsc{cnn} is usually to extract the semantics of a sentence, and the nature of most tasks makes it so that sentences with similar meanings should have similar representations.
 
 \subsection{Recurrent Neural Network}
 \label{sec:context:rnn}
 A limitation of \textsc{cnn}s is the difficulty they have modeling patterns of non-adjacent words.
 A second approach to process whole sentences is to use recurrent neural networks (\textsc{rnn}).
 \textsc{rnn}s purpose to sum up an entire sentence prefix into a fixed-size hidden state, updating this hidden state as the sentence is processed.
 This can be used to build a causal language model following the decomposition of Equation~\ref{eq:context:causal lm}.
 As showcased by Figure~\ref{fig:context:rnn}, the hidden state \(\vctr{h}_t\) can be used to predict the next word \(w_{t+1}\) with a simple linear layer followed by a softmax, formally:
 \begin{marginfigure}
 	\centering
 	\input{mainmatter/context/rnn lm.tex}
 	\scaption{\textsc{rnn} language model unrolled through time.}
 	\label{fig:context:rnn}
 \end{marginfigure}
 \begin{align}
 	\vctr{h}_t & = f(\mtrx{W}^{(x)} \vctr{x}_t + \mtrx{W}^{(h)} \vctr{h}_{t-1} + \vctr{b}^{(h)})
 		\label{eq:context:rnn} \\
 	\hat{w}_t & = \softmax(\mtrx{W}^{(o)} \vctr{h}_t + \vctr{b}^{(o)})
 		\nonumber
 \end{align}
 where \(\mtrx{W}^{(x)}\), \(\mtrx{W}^{(h)}\), \(\mtrx{W}^{(o)}\), \(\vctr{b}^{(h)}\) and \(\vctr{b}^{(o)}\) are model parameters and \(f\) is a non-linearity, usually a sigmoid \(f(x) = \sigmoid(x) = \frac{1}{1 + \symup{e}^{-x}}\).
 This model is usually trained by minimizing the negative log-likelihood:
 \marginnote{
 	We generally use \(\vctr{\theta}\) to refer to the set of model parameters.
 	In this case \(\vctr{\theta} = \{\mtrx{W}^{(x)}, \mtrx{W}^{(h)}, \mtrx{W}^{(o)}, \vctr{b}^{(h)}, \vctr{b}^{(o)}\}\).
 }[1cm]
 \begin{equation*}
 	\loss{rnn}(\vctr{\theta}) = \sum_{t=1}^m - \log P(w_t\mid \vctr{x}_1, \dotsc, \vctr{x}_{t-1}; \vctr{\theta})
 \end{equation*}
 using the backpropagation-through time algorithm.
 The gradient is run through all the steps of the \textsc{rnn} until reaching the beginning of the sequence.
 When the sequence is a sentence, this can easily be achieved.
 However, when longer spans of text are considered, the gradient only goes back a fixed number of tokens in order to limit memory usage.
 
 \subsubsection{Long Short-term Memory}
 \label{sec:context:lstm}
 Standard \textsc{rnn}s tend to have a hard time dealing with long sequences.
 This problem is linked to the vanishing and exploding gradient problems.
 When the gradient goes through several non-linearities, it tends to be less meaningful, and gradient descent does not lead to satisfying parameters anymore.
 In particular, when \(\mtrx{W}^{(h)}\) has a large spectral norm, the values \(\vctr{h}_t\) tend to get bigger and bigger with long sequences, on the other hand when its spectral norm is small, these values get smaller and smaller.
 When \(\vctr{h}_t\) has a large magnitude, the sigmoid activation saturates and \(\frac{\partial \loss{rnn}}{\partial \vctr{h}_t}\) gets close to zero, the gradient vanishes.
 \textsc{rnn} variants are used to alleviate this vanishing gradient problem, the most common being long short-term memory (\textsc{lstm}, \citex{lstm}).
 \begin{figure}[ht!]
 	\centering
 	\input{mainmatter/context/lstm.tex}
 	\scaption[Architecture of an \textsc{lstm} cell.]{
 		\label{fig:context:lstm cell}
 		Architecture of an \textsc{lstm} cell.
 		In its simplest form, this block replaces the linear layer at the bottom of Figure~\ref{fig:context:rnn}.
 		The link between \(\vctr{c}_t\) and \(\vctr{c}_{t-1}\) is illustrated by a self-loop but could be seen as an additional input and output.
 	}
 \end{figure}
 
 \textsc{lstm}s redefine the recurrence of \textsc{rnn}s (Equation~\ref{eq:context:rnn}) by adding multiplicative gates as illustrated by Figure~\ref{fig:context:lstm cell}.
 It is governed by the following set of equations:
 \begin{equation*}
 \def\arraystretch{1.25}
 \begin{array}{l l l l}
 		\vctr{x}'_t        & = & \begin{bmatrix}\vctr{x}_t\\ \vctr{h}_{t-1}\end{bmatrix}                               & \text{Recurrent input}\\[3.9mm]
 		\tilde{\vctr{c}}_t & = & \tanh(\mtrx{W}^{(c)} \vctr{x}'_t + \vctr{b}^{(c)})                                    & \text{Cell candidate}\\
 		\vctr{i}_t         & = & \sigmoid(\mtrx{W}^{(i)} \vctr{x}'_t + \mtrx{U}^{(i)} \vctr{c}_{t-1} + \vctr{b}^{(i)}  & \text{Input gate}\\
 		\vctr{f}_t         & = & \sigmoid(\mtrx{W}^{(f)} \vctr{x}'_t + \mtrx{U}^{(f)} \vctr{c}_{t-1} + \vctr{b}^{(f)}) & \text{Forget gate}\\
 		\vctr{c}_t         & = & \vctr{i}_t\odot \tilde{\vctr{c}}_t + \vctr{f}_t\odot \vctr{c}_{t-1}                   & \text{New cell}\\
 		\vctr{o}_t         & = & \sigmoid(\mtrx{W}^{(o)} \vctr{x}'_t + \mtrx{U}^{(o)} \vctr{c}_t + \vctr{b}^{(o)})     & \text{Output gate}\\
 		\vctr{h}_t         & = & \vctr{o}_t\odot \tanh(\vctr{c}_t)                                                     & \text{Hidden layer output}\\
 \end{array}
 \end{equation*}
 \marginnote{
 	\(\odot\) is the element-wise multiplication and \(\sigmoid\) the sigmoid function.
 }[-18mm]
 \marginnote{
 	As with \textsc{rnn}, \(\vctr{\theta} = \{ \mtrx{W}^{(c)}, \mtrx{W}^{(i)}, \mtrx{U}^{(i)},\\\mtrx{W}^{(f)}, \mtrx{U}^{(f)}, \mtrx{W}^{(o)}, \mtrx{U}^{(o)}, \vctr{b}^{(c)}, \vctr{b}^{(f)}, \vctr{b}^{(i)},\\\vctr{b}^{(o)} \}\) are model parameters.
 }[-8mm]
 
 The main peculiarity of \textsc{lstm} is the presence of multiple gates used as masks or mixing factors in the unit.
 \textsc{lstm} units are interpreted as having an internal cell memory \(\vctr{c}_t\) which is an additional (internal) state alongside \(\vctr{h}_t\) and is used as input of the cell alongside \(\vctr{x}_t\) and \(\vctr{h}_{t-1}\).
 When computing its activation, we first compute a cell candidate \(\tilde{\vctr{c}}_t\) which is the potential successor to \(\vctr{c}_t\).
 Then, the multiplicative gates come into play, the cell \(\vctr{c}_t\) is partially updated with a mix of \(\vctr{c}_{t-1}\) and \(\tilde{\vctr{c}}_t\) controlled by the input and forget gates \(\vctr{i}_t\) and \(\vctr{f}_t\).
 Finally, the output of the unit is masked by the output gate \(\vctr{o}_t\).%
 \sidenote{Note that the output gate \(\vctr{o}_t\) has its value computed from the new cell value \(\vctr{c}_t\) instead of \(\vctr{c}_{t-1}\) in contrast to the expression of \(\vctr{i}_t\) and \(\vctr{f}_t\).}
 
 It has been theorized~\parencite{lstm_vanishing} that the gates are what makes \textsc{lstm}s so powerful.
 The multiplications allow the model to learn to control the flow of information in the unit, thus counteracting the vanishing gradient problem.
 The basic building block of multiplicative gates has been reused for other \textsc{rnn} cell designs such as gated recurrent unit (\textsc{gru}, \cite{nmt_encdec}).
 Furthermore, random cell designs using multiplicative gates can be shown to perform as well as \textsc{lstm}~\parencite{lstm_odyssey}.
 However, standard practice is to always use \textsc{lstm} or \textsc{gru} for recurrent neural networks.
 
 \subsubsection{\textsc{elm}o}
 \label{sec:context:elmo}
 Recurrent neural networks with \textsc{lstm} cells were widely used for language modeling, both at the character-level~\parencite{charrnn} and at the word-level~\parencite{lm_limits}.
 The first language model to become widely used for extracting contextual word embeddings was \textsc{elm}o (Embeddings from Language Model, \citex{elmo}) which uses several \textsc{lstm} layers.
 
 The peculiarity of the word embeddings extracted by \textsc{elm}o is that they are contextualized (see Section~\ref{sec:context:language model}).
 Static word embeddings models like word2vec (Section~\ref{sec:context:word2vec}) map each word to a unique vector.
 However, this fares poorly with polysemic words and homographs whose meaning depends on the context in which they are used.
 \begin{marginparagraph}
 	Before \textsc{elm}o, \textcite{cove} already trained contextualized word representations using an \textsc{nmt} task.
 \end{marginparagraph}
 Contextualized word embeddings provide an answer to this problem.
 Given a sentence, \textsc{elm}o proposes to use the hidden states \(\vctr{h}_t\) as a representation of each constituting word \(w_t\).
 These representations are hence a function of the whole sentence.%
 \sidenote{
 	In order to encode both the left and right context of a word, \textsc{elm}o uses bidirectional \textsc{lstm}, meaning that each layer contains two \textsc{lstm}, one running from left-to-right and one right-to-left.}
 Thus words are mapped to different vectors in different contexts.
 
 \subsection{Attention Mechanism}
 \label{sec:context:attention}
 To obtain a vector representation of a sentence from an \textsc{rnn}, two straightforward methods are to use the last hidden state \(\vctr{h}_m\) or use a pooling layer similar to the one used in \textsc{cnn}, such as max-over-time pooling.
 However, both of these approaches present shortcomings: the last hidden state tends to encode little information about the beginning of the sentence, while pooling is too indiscriminate and influenced by unimportant words.
 Using an attention mechanism is a way to avoid these shortcomings.
 Furthermore, an attention mechanism is parametrized by a \emph{query} which allows us to select the piece of information we want to extract from the sentence.
 
 The concept of attention first appeared in neural machine translation (\textsc{nmt}) under the name ``alignment''~\parencitex{attention} before becoming ubiquitous in \textsc{nlp}.
 The same principle was also presented under the name \emph{memory network}~\parencite{memory_networks, memory_networks_end-to-end}.
 It is also the building block of transformers, which are presented \hyperref[sec:context:transformers]{next}.
 With this in mind, we use the vocabulary of memory networks to describe the attention mechanism.
 
 \begin{figure}[ht!]
 	\centering
 	\input{mainmatter/context/attention.tex}
 	\scaption[Schema of an attention mechanism.]{
 		Schema of an attention mechanism.
 		The attention scores are obtained by an inner product between the query and the memory.
 		The output is obtained as a sum of the memory weighted by the softmax of the attention scores.
 		\label{fig:context:attention}
 	}
 \end{figure}
 
 \subsubsection{Attention as a Mechanism for \textsc{rnn}}
 The principle of an attention layer on top of an \textsc{rnn} is illustrated by Figure~\ref{fig:context:attention}.
 The layer takes three inputs: a query \(\vctr{q}\in\symbb{R}^d\), memory keys \(\mtrx{K}\in\symbb{R}^{\ell\times d}\) and memory values \(\mtrx{V}\in\symbb{R}^{\ell\times d'}\).
 Originally, more often than not, \(\mtrx{K}=\mtrx{V}\).
 In the model of Figure~\ref{fig:context:attention}, the memory corresponds to the hidden states of the \textsc{rnn}, which was the most common architecture when attention was introduced in 2014.
 First, attention weights are computed from the query \(\vctr{q}\) and keys \(\mtrx{K}\), then these weights are used to compute the output \(\vctr{o}\in\symbb{R}^{d'}\) as a convex combination of the values \(\mtrx{V}\)\,:
 \begin{marginparagraph}[-11mm]
 	Where \(\softmax\) is a smooth version of the \(\argmax\) function.
 	It can also be seen as a multi-dimensional sigmoid, defined as:
 	\begin{equation*}
 		\softmax(\vctr{x})_i = \frac{\exp x_i}{\sum_j \exp x_j}
 	\end{equation*}
 \end{marginparagraph}
 \begin{equation}
 	\vctr{o} = \softmax(\mtrx{K}\vctr{q})\mtrx{V}.
 \end{equation}
 
 In \textsc{nmt}, the memory is built from the hidden states of an \textsc{rnn} running on the sentence to be translated (meaning \(\ell=m\)), while the query is the state of the translated sentence (``what was already translated''), the attention is then recomputed for each output position.
 In other words, a new representation of the source sentence is recomputed for each word in the target sentence.
 The attention weights---that is, the output of the softmax---can provide an interpretation of what the model is focusing on when making a prediction.
 In the case of \textsc{nmt}, the attention for producing a translated word usually focuses on the corresponding word or group of words in the source sentence.
 
 \subsubsection{Attention as a Standalone Model}
 \label{sec:context:attention lm}
 Since the attention mechanism produces a fixed-size representation (\(\vctr{o}\)) from a variable length sequence (\(\mtrx{K}\), \(\mtrx{V}\)\,), it can actually be used by itself without an \textsc{rnn}.
 This was already mentioned in \textcitex{memory_networks_end-to-end}[-10mm] and used for language modeling.
 We now succinctly present their approach.
 As shown Figure~\ref{fig:context:memory network lm}, this is a causal language model (Section~\ref{sec:context:language model}), at each step \(P(w_t\mid w_1,\dotsc,w_{t-1})\) is modeled.
 While the previous words constitute the memory of the attention mechanism, there is no natural value for the query.
 As such, for the first layer, it is simply taken to be a constant vector \(q^{(1)}_i = 0.1\) for all \(i=1,\dotsc, d\).
 When several attention layers are stacked, the output \(o^{(l)}\) of a layer \(l\) is used as the query \(q^{(l+1)}\) for the layer \(l+1\).
 Furthermore, residual connections with linear layers and modified ReLU non-linearities%
 \sidenote[][-35.5mm]{
 	While the standard ReLU activation~\parencite{relu} is defined as \(\ReLU(x)=\max(0, x)\).
 	The non-linearity used in this model is \(\ReLU_{\halfCircleScript}\), which applies the ReLU activation to half of the units in the layer.}
 are introduced between layers thus: \(\vctr{q}^{(l+1)} = \ReLU_{\halfCircleScript}(\mtrx{W}^{(l)} \vctr{q}^{(l)} + \vctr{o}^{(l)})\) where the matrices \(\mtrx{W}^{(l)}\in\symbb{R}^{d\times d}\) are parameters of the model.
 As usual, the next word prediction \(\hat{w}_i\) is made using a softmax layer.
 
 \begin{marginfigure}[-30mm]
 	\centering
 	\input{mainmatter/context/memory network lm.tex}
 	\scaption[Schema of a memory network language model with two layers.]{
 		Schema of a memory network language model with two layers.
 		Each red block corresponds to an attention mechanism as illustrated by Figure~\ref{fig:context:attention}.
 	}
 	\label{fig:context:memory network lm}
 \end{marginfigure}
 
 \paragraph{Temporal Encoding}
 The attention mechanism as described above is invariant to a permutation of the memory.
 This is not a problem when an \textsc{rnn} is run on the sentence, as it can encode the relative positions of each token.
 However, in the \textsc{rnn}-less approach of \textcite{memory_networks_end-to-end} this information is lost, which is quite damaging for language modeling.
 Indeed, this would mean that shuffling the words in a sentence---like inverting the subject and object of a verb---does not change its meaning.
 In order to solve this problem, temporal encoding is introduced.
 When predicting \(w_i\), each word embedding \(\vctr{x}_j\) in the memory is summed with a relative position embedding \(\vctr{e}_{i-j}\).
 These position embeddings are trained through back-propagation like any other parameters.
 
 \bigskip
 
 Attention mechanisms form the basis of current state-of-the-art approaches in \textsc{nlp}.
 One of the explanations behind their success is that, in a sense, they are more shallow than \textsc{rnn}.
 Indeed, when computing \(\frac{\partial \hat{w}_i}{\partial \vctr{x}_j}\) for the language model of \textcite{memory_networks_end-to-end}, one can see that part of the gradient goes through few non-linearities.
 In contrast, the information from \(\vctr{x}_j\) to \(\hat{w}_i\) must go through the composition of at least \(i-j\) non-linearities in an \textsc{rnn}, which may cause the gradient to vanish.
 However, an attention mechanism has linear complexity in the length of the sequence for a total of \(\Theta(m\times d^2)\) operations at each step.
 When \(m\) is large, this can be prohibitive compared to \textsc{rnn}, which have a \(\Theta(d^2)\) complexity at each step.
 On the other hand, an attention layer can easily be parallelized while an \textsc{rnn} always necessitates \(\Omega(m)\) sequential operations.
 
 \subsection{Transformers}
 \label{sec:context:transformers}
 Transformers~\parencitex{transformers} were originally introduced for \textsc{nmt}.
 Likewise to the memory network language model presented above, they introduce several slight modifications of its architecture which make them the current state of the art for most \textsc{nlp} tasks.
 For conciseness, we present the concept of transformers as used by \textsc{bert} (Bidirectional Encoder Representations from Transformers, \citex{bert}).
 \textsc{bert} is a language model used to extract contextualized embeddings similar to \textsc{elm}o but using attention layers in place of \textsc{lstm} layers.
 
 \subsubsection{Transformer Attention}
 \label{sec:context:transformer attention}
 The attention layers used by transformers are slightly modified.
 \marginpar{Note that in contrast to the classical attention mechanism presented in Section~\ref{sec:context:attention}, transformers have \(\mtrx{K}\neq\mtrx{V}\).}
 First, it is often advisable that in a neural network, all activations follow a standard normal distribution \(\normalDistribution(0, 1)\).
 In order to achieve this, transformers use scaled attention:
 \begin{equation}
 	\operatorname{Attention}(\vctr{q}, \mtrx{K}, \mtrx{V}) = \softmax\left(\frac{\mtrx{K}\vctr{q}}{\sqrt{d}}\right)\mtrx{V}.
 \end{equation}
 This ensures that if \(\mtrx{K}\) and \(\vctr{q}\) follow a standard normal distribution, so does the input of the softmax.
 
 Second, multi-head attention is used: each layer actually applies \(h=8\) attentions in parallel.
 To ensure each individual attention captures a different part of the semantic, its input is projected by different matrices, one for each attention head:
 \begin{equation*}
 	\operatorname{MultiHeadAttention}(\vctr{q}, \mtrx{K}, \mtrx{V}) =
 	\begin{bmatrix}
 		\operatorname{head}_1(\vctr{q}, \mtrx{K}, \mtrx{V}) \\
 		\operatorname{head}_2(\vctr{q}, \mtrx{K}, \mtrx{V}) \\
 		\vdots\\
 		\operatorname{head}_h(\vctr{q}, \mtrx{K}, \mtrx{V}) \\
 	\end{bmatrix} \mtrx{W}^{(o)}
 \end{equation*}
 \begin{equation*}
 	\operatorname{head}_i(\vctr{q}, \mtrx{K}, \mtrx{V}) = \operatorname{Attention}(\vctr{q}\mtrx{W}_i^{(q)}, \mtrx{K}\mtrx{W}_i^{(k)}, \mtrx{V}\mtrx{W}_i^{(v)}).
 \end{equation*}
 
 Lastly, on top of each attention layer is a linear layer with ReLU activation and a linear layer followed by layer normalization~\parencite{layernorm}.
 These linear layers are identical along the sequence length, akin to a convolution with kernel size 1.
 While the query of each layer is the output of the preceding layer, similarly to the model of \textcite{memory_networks_end-to-end}, the initial query is now the current word itself \(\vctr{x}_t\).
 This architecture is illustrated in Figure~\ref{fig:context:bert}.
 
 \Textcite{bert} introduce two \textsc{bert} architectures dubbed \bertArch{small} and \bertArch{large}.
 Like their names imply, \bertArch{small} has fewer parameters than \bertArch{large}, in particular, \bertArch{small} is composed of 12 layers while \bertArch{large} is composed of 24 layers.
 
 \begin{marginfigure}
 	\centering
 	\input{mainmatter/context/bert.tex}
 	\scaption[Schema of \textsc{bert}, a transformer masked language model.]{
 		Schema of \textsc{bert}, a transformer masked language model.
 		The schema is focused on the prediction for a single position \(t\), this is repeated for the whole sentence \(t=1, \dotsc, m\).
 		The model presented is the \bertArch{small} variant containing only 12 layers.
 		The input vectors \(\vctr{\tilde{x}}_t\) are obtained from the corrupted sentence \(\vctr{\tilde{w}}\) using an embedding layer.
 		To obtain \(\hat{w}_t\) from the last \textsc{bert} layer output, a linear layer with softmax over the vocabulary is used.
 		\label{fig:context:bert}
 	}
 \end{marginfigure}
 
 \subsubsection{Masked Language Model}
 \label{sec:context:mlm}
 While some transformer models such as \textsc{gpt} (Generative Pre-Training, \cite{gpt}) are causal language models, \textsc{bert} is a \emph{masked} language model (\textsc{mlm}).
 Instead of following Equation~\ref{eq:context:causal lm}, the following approximation is used:
 \begin{equation}
 	P(\vctr{w}) \propto \prod_{t\in C} P(w_t \mid \tilde{\vctr{w}})
 \end{equation}
 where \(C\) is a random set of indices, 15\% of tokens being uniformly selected to be part of \(C\), and \(\tilde{\vctr{w}}\) is a corrupted sequence defined as follow:
 \begin{equation*}
 	\tilde{w}_t = \left\{\begin{array}{@{}ll@{}}
 		w_t & \text{if } t\not\in C \\
 		\left.\begin{array}{@{}ll@{}}
 			\text{\blanktag{} token} & \text{with probability } 80\% \\
 			\text{random token} & \text{with probability } 10\% \\
 			w_t & \text{with probability } 10\% \\
 		\end{array}\right\} & \text{if } i\in C\\
 	\end{array}\right.
 \end{equation*}
 The masked tokens \blanktag{} make up the majority of the set \(C\) of tokens predicted by the model, thus the name ``masked language model''.
 The main advantage of this approach compared to causal language model is that the probability distribution at a given position is parametrized by the whole sentence, including both the left and right context of a token.
 
 \subsubsection{Transfer Learning}
 \label{sec:context:transfer learning}
 The main purpose of \textsc{bert} is to be used on a \emph{downstream task}, transferring the knowledge gained on masked language modeling to a different problem.
 As with \textsc{elm}o, the hidden state of the topmost layer, just before the linear and softmax, can be used as contextualized word representations.
 Furthermore, the first token, usually called ``beginning of sentence'' but dubbed \textsc{cls} in \textsc{bert}, can be used as a representation of the whole sentence.%
 \sidenote{
 	This is by virtue of an additional \emph{next sentence prediction} loss with which \textsc{bert} is trained.
 	We do not detail this task here as it is not essential to \textsc{bert}'s training.
 	Furthermore, the embedding of the \textsc{cls} token is considered a poor representation of the sentence and is rarely used~\parencite{xlm, xlnet}.}
 In contrast with \textsc{elm}o, \textsc{bert} is usually fully fine-tuned on the downstream task.
 In the original article~\parencite{bert}, this was shown to outperform previous models on a wide variety of tasks, from question answering to textual entailments.
 
 \bigskip
 
 In this section, we presented several \textsc{nlp} models which allow us to get a distributed representation for words, sentences and words contextualized in sentences.
 These representations can then be used on a downstream task, such as relation extraction, as we do from Chapter~\ref{chap:relation extraction} onward.
 We now focus on the other kind of data handled in this thesis: knowledge bases.