Augmenting Transformers for Open Domain Procedural Text Comprehesion

Abstract

Recent advances in deep learning model architectures have permitted state-of-the-art results in various fields such as NLP and CV. Although these systems have matched and, in some cases, surpassed human performance, many of them are still treated as black boxes, with sometimes unpredictable results. To try and shed some light on behaviors of natural language generation models, we examine the task of procedural text comprehension using neuro-symbolic techniques. We use this task as a testbed for exploring the limitations of state-of-the-art systems such as GPT on the task of predicting the resulting state changes from the text description of a procedure. We also experiment with whether and how symbolic augmentations may help these systems with understanding language. We see some promising results in concept-net knowledge injection, and note that other augmentations provide more natural generations.