Structure and geometry in sequence-processing neural networks

Abstract

Recent success of state-of-the-art neural models on various natural language processing (NLP) tasks has spurred interest in understanding their representation space. In the following chapters we will use various techniques of representational analysis to understand the nature of neural-network based language modelling. To introduce the concept of linguistic probing, we explore how various language features affect model representations and long-term behavior through the use of linear probing techniques. To tease out the geometrical properties of BERT's internal representations, we task the model with 5 linguistic abstractions (word, part-of-speech, combinatory categorical grammar, dependency parse tree depth, and semantic tag). By using a Mean Field theory backed manifold capacity (MFT) metric, we show that BERT entangles linguistic information when contextualizing a normal sentence but detangles the same information when it must form a token prediction. To mend our findings to those of previous works that used linear probing, we reproduce the prior results and show that linear separation between classes follows the trends we present. To show that linguistic structure of a sentence is being geometrically embedded in BERT representations, we swap words in sentences such that the underlying tree structure becomes perturbed. By using canonical correlation analysis (CCA) to compare sentence representations, we find that the distance between swapped words is directly proportional to the decrease in geometric similarity of model representations.