conclusion.tex (3402B)
1 \section{Conclusion} 2 \label{sec:context:conclusion} 3 As exposed in Section~\ref{sec:context:history}, we are in the middle of a transition away from symbolic representations towards distributed ones. 4 We inscribe this thesis within this transition. 5 We deal with two kinds of symbolic representations of meaning: unstructured language and structured knowledge bases. 6 In this chapter, we presented methods to extract distributed representations for both of these systems. 7 While in the following chapters, we will deal with the link between language and knowledge bases. 8 9 Following word2vec (Section~\ref{sec:context:word2vec}), feature extraction for textual inputs is now mostly done through word embeddings. 10 In order to obtain a representation of a sentence, the models on top of these word embeddings progressively evolved from \textsc{cnn} (Section~\ref{sec:context:cnn}) and \textsc{rnn} (Section~\ref{sec:context:rnn}) towards transformers and contextualized word embeddings (Section~\ref{sec:context:transformers}). 11 As we will see in the \hyperref[chap:relation extraction]{next} chapter, this trend was exactly followed by relation extraction models. 12 13 We then introduce the structured knowledge representation we handle throughout this thesis, knowledge bases. 14 In particular, Section~\ref{sec:context:relation algebra} gives a formal notation for handling relations which we use to write modeling hypotheses in subsequent chapters. 15 Finally, Section~\ref{sec:context:knowledge base completion} presents common models making use of distributed representations of knowledge bases for the task of knowledge base completion. 16 This task is not only the usual evaluation framework for distributed knowledge base representations but is also of special interest for Chapter~\ref{chap:fitb}, where we leverage the similarity between the knowledge base completion and the relation extraction tasks. 17 18 The progression of models presented in this chapter also reflects a progression of the scale of problems. 19 We started by exploring the representation of words, one of the smallest semantic units, then moved on to sentences, then to knowledge bases, which purpose to represent whole pans of human knowledge. 20 Another underlying thread to this chapter is the notion of relationship. 21 While the idea is quite pervasive in Section~\ref{sec:context:knowledge base}, it is also present in Section~\ref{sec:context:word} through the not-so-randomly chosen example of Figure~\ref{fig:context:word2vec pca}.% 22 \sidenote{ 23 This figure presented the word embeddings of some countries and their capitals. 24 The relationship between the words seems to bear the same regularity as the relationship between the underlying entities. 25 This regularity being representative of the \textsl{capital of} relationship. 26 } 27 Even in Section~\ref{sec:context:sentence}, representations of sentences are obtained by modeling the relationship of words with each other. 28 For example, in a transformer, the attention weights capture the relationship between two words: the query and one element of the memory. 29 30 In the next chapter, we make the link between the two symbolic representations of meaning we studied: language and knowledge bases. 31 More specifically, we present relation extraction models. 32 State-of-the-art models build heavily on the distributed representations methods introduced in this chapter and are the main focus of this thesis.