Uncovering brain-like computations for natural language processing

Abstract

You are using your brain to understand this sentence, but can you explain precisely how you do it? Although we constantly experience language processing first-hand, we're still not entirely sure how it's done. Linguists and computer scientists have been separately working on discovering mechanisms that are necessary for high-performing language processing, but they have yet to discover the holy-grail. In this work, we take the first steps toward bridging the gap between these two approaches by developing a method that discovers, with statistical significance, brain-like neural network sub-architectures in their simplest form. Instead of just evaluating established NLP models for brain-likeness, our objective is to find new architectures and computations that are especially brain-like. The method randomly generates a large and varied collection of neural network architectures in the pursuit of finding architectures that mimic fMRI data in tasks like language modeling, translation, and summarization. All hyper-parameters are fixed across models. Thus, the method is then able to identify the sub-architectures that are associated with the most brain-like models, and return them in their simplest form. The sub-architectures that the method returns enable two important analyses: because they are pruned to the most brain-like components, the computations that these smaller sub-architectures perform are easier to interpret than those of the architecture as a whole. And interpretability is crucial for understanding the mechanisms that are intrinsic to language processing. The second reason is that the sub-architectures may help improve future architecture samples. For instance, those brain-like computations may be defined as unit operations in order to bias more models to include them.