PhD

history.tex (10925B)

---

 \section{Historical Development}
 \label{sec:context:history}
 In this section, we expose the rationale for applying deep learning to relation extraction, how the related fields appeared and why the task is relevant.
 Since algorithms were first given to train generic deep neural networks~\parencite{deepbeeliefnets,relu}, most problems tackled by machine learning can now be approached with deep learning methods.
 Over the last few years, deep learning has been very successful in a variety of tasks such as image classification~\parencite{cnn_imagenet}, machine translation~\parencite{nmt_encdec}, audio synthesis~\parencite{wavenet}, etc.
 This is why it is not surprising that deep learning is now applied to more tasks traditionally tackled by other machine learning methods, such as in this thesis, where we apply it to relation extraction.
 
 From a historical point of view, machine learning---and hence deep learning---are deeply anchored in \emph{empiricism}.
 Empiricism is the epistemological paradigm in which knowledge is anchored in sensory experiences of the world, which are called empirical evidence.
 This is not to say that there are no theoretical arguments motivating the use of certain machine learning methods; the universal approximation theorems~\parencite{universal_approximator_sigmoid, universal_approximator_nonpolynomial} can be seen as a theoretical argument for deep learning.
 But in the end, a machine learning method draws its legitimacy from the observation that they perform strongly on a real dataset.
 This is in stark contrast to the rationalist paradigm, which posits that knowledge comes primarily from reason.
 
 This strong leaning on empiricism can also be seen in \textsc{nlp}.
 \textsc{nlp} comes from the \emph{externalist} approach to linguistic theorizing, focusing its analyses on actual utterances.
 A linguistic tool that externalists often avoid while being widely used by other schools is elicitation through prospective questioning: ``Is this sentence grammatical?''
 Externalists consider that language is acquired through distributional properties of words and other constituents;%
 \sidenote{In other words, language is acquired by observing empirical co-occurrences: where words go and where they don't in actual utterances tell us where they can go and where they can't.}
 and study these properties by collecting corpora of naturally occurring utterances.
 The associated school of structural linguistics inscribes itself into the broader view of \emph{structuralism}, the belief that phenomena are intelligible through a concept of structure that connects them together, the focus being more on these interrelations instead of each individual object.
 In the case of linguistics, this view was pioneered by Ferdinand de Saussure which stated in its course in general linguistics:
 \begin{quote}
 	\begin{epigraph}{Ferdinand de Saussure}{\citetitle{linguistique_generale}}{\cite*{linguistique_generale}}
 		La langue est un système dont toutes les parties peuvent et doivent être considérées dans leur solidarité synchronique.
 	\end{epigraph}
 	Language is a system whose parts can and must all be considered in their synchronic%
 	\sidenote{
 		Saussure makes a distinction between syn\-chron\-ic---at a certain point in time---and dia\-chron\-ic---changing over time---analyses.
 		This does not mean that the meaning of a word is not influenced by its history, but that this influence is entirely captured by the relations of the word with others at the present time and that conditioned on these relations, the current meaning of the word is independent of its past meaning.
 	}
 	solidarity.\\
 	\null\hfill---
 	\begin{minipage}[t]{5cm}
 		Ferdinand de Saussure, \citetitle{linguistique_generale}~(\cite*{linguistique_generale})
 	\end{minipage}
 \end{quote}
 This train of thought gave rise to \emph{distributionalism} whose ideas are best illustrated by the distributional hypothesis stated in \textcite{distributional_hypothesis}:
 \begin{spacedblock}
 	\strong{Distributional Hypothesis:}
 	\emph{Words that occur in similar contexts convey similar meanings.}
 \end{spacedblock}
 This can be pushed further by stating that a word is solely characterized by the context in which it appears.
 
 On the artificial intelligence side, deep learning is usually compared to symbolic approaches.
 The distinction originates in the way information is represented by the system.
 In the symbolic approach, information is carried by strongly structured representations in which a concept is usually associated with a single entity, such as a variable in a formula or in a probabilistic graphical model.
 On the other hand, deep learning uses distributed representations in which there is a many-to-many relationship between concepts and neurons; each concept is represented by many neurons, and each neuron represents many concepts.
 The idea that mental phenomena can be represented using this paradigm is known as \emph{connectionism}.
 One particular argument in favor of connectionism is the ability to degrade gracefully: deleting a unit in a symbolic representation equates to deleting a concept, while deleting a unit in a distributed representation merely lowers the precision with which concepts are defined.
 Note that connectionism is not necessarily incompatible with a symbolic theory of cognition.
 Distributed representations can be seen as a low-level explanation of cognition, while from this point of view, symbolic representation is a high-level interpretation encoded by distributed representations.%
 \sidenote[][-3cm]{
 	This view on the relation between distributed and symbolic representations can be seen in the early neural networks literature as can be seen in \textcite{concept_backprop}, which is often cited for its formalization of the backpropagation algorithm.
 	More recently, \textcite{binding_symbolic} investigate the  binding problem between symbols and distributed representations.
 }
 
 Furthermore, we can make a distinction on how structured is the kind of data used.
 In this thesis, we will especially focus on the relationship between unstructured text%
 \sidenote{
 	Of course, language does have a structure.
 	We do not deny the existence of grammar but merely state that text is less structured than other structures studied in this chapter (see Section~\ref{sec:context:knowledge base}).
 }
 and structured data (in the form of knowledge bases).
 To give a sense of this difference, compare the following text from the Paris Wikipedia page to facts from the Wikidata knowledge base:
 \begin{spacedblock}
 \null\hfill%
 \begin{minipage}{0.425\textwidth}
 	Paris is the capital and most populous city of France.
 	The City of Paris is the centre and seat of government of the region and province of Île-de-France.
 \end{minipage}%
 \hfill%
 \begin{minipage}{0.425\textwidth}
 	\begin{itemize}[label={},leftmargin=0mm]
 		\item Paris \textsl{capital of} France
 		\item Paris \textsl{located in the administrative territorial entity} Île-de-France
 	\end{itemize}
 \end{minipage}%
 \hfill\null%
 \end{spacedblock}
 \begin{marginparagraph}[-1cm]
 	We use \textsl{slanted text} to indicate a relational surface form such as ``\textsl{capital of}'' in the fact ``Paris \textsl{capital of} France.''
 \end{marginparagraph}%
 
 Through this example, we see that both natural languages and knowledge bases encode meaning.
 To talk about what they encode, we assume the existence of a semantic space containing all possible meanings.
 We do not assume any theory of meaning used to define this space; this allows us to stay neutral on whether language is ontologically prior to propositional attitudes and its link with reality or semantically evaluable mental states.
 In the same way that different natural languages are different methods to address this semantic space, knowledge bases seek to refer to the same semantic space%
 \sidenote{
 	Strictly speaking, practical knowledge bases only seek to index a subset of this space, see note~\ref{note:context:knowledge vs meaning} in the margin of page \pageref{note:context:knowledge vs meaning}.
 }
 with an extremely rigid grammar.
 
 Both natural language and knowledge bases are discrete systems.
 For both these systems, we can use the distributional hypothesis to obtain continuous distributed representations.
 These representations purpose to capture the semantic as a simple topological space such as a Euclidean vector space where distance encodes dissimilarity, as shown in Figure~\ref{fig:context:word2vec pca}.
 Moreover, using a differentiable manifold allows us to train these representations through backpropagation using neural architectures.
 
 The question of how to process texts algorithmically has evolved over the last fifty years.
 Language being conveyed through symbolic representations, it is quite natural for us to manipulate them.
 As such, early machine learning models strongly relied on them.
 For a long time, symbolic approaches had an empirical advantage: they worked better.
 However, in the last few years, distributed representations have shown unyielding results, and most tasks are now tackled with deep learning using distributed representations.
 \begin{marginparagraph}
 	This transition from rule-based models to statistical models to neural network models can also be seen in relation extraction with Hearst (\cite*{hearst_hyponyms}, symbolic rule-based, Section~\ref{sec:relation extraction:bootstrap}), \textsc{sift} (\cite*{sift}, symbolic statistical, Section~\ref{sec:relation extraction:hand-designed features}) and \textsc{pcnn} (\cite*{pcnn}, distributed neural, Section~\ref{sec:relation extraction:pcnn}).
 \end{marginparagraph}
 As an example, this can be seen in the machine translation task.
 Early models from the 1950s onward were rule-based.
 Starting in the 1990s, statistical approaches were used, first using statistics of words then of phrases.
 Looking at the Workshop on statistical machine translation (\textsc{wmt}): at the beginning of the last decade, no neural approaches were used and the report~\parencite{wmt2010} deplored the disappearance of rule-based systems, at the end of the decade, most systems were based on distributed representations~\parencite{wmt2020}.%
 \sidenote{To be more precise, most models use transformers which are a kind of neural network introduced in Section~\ref{sec:context:transformers}.}
 While this transition occurred in \textsc{nlp}, knowledge representation has been a stronghold of symbolic approaches until very recently.
 The research reported in this thesis aims to develop the distributed approach to knowledge representation for the task of relation extraction.
 In the remainder of this chapter, we first report the distributed approaches to \textsc{nlp}, which showcased state-of-the-art results for the last decade, before presenting a structured symbolic representation, knowledge bases, and some methods to obtain distributed representations from them.