On the Power of Decision Trees in Auto-Regressive Language Modeling
Name
CBMM-Memo-149.pdf
Size
2.11 MB
Format
Adobe PDF
Checksum (MD5)
621795d477f1a68246800940d3f3347a
Author(s)
Gan, Yulu
Galanti, Tomer
Poggio, Tomaso
Malach, Eran
Date Issued
September 27, 2024
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;149
Abstract
Originally proposed for handling time series data, Auto-regressive Decision Trees (ARDTs) have not yet been explored for language modeling. This paper delves into both the theoretical and practical applications of ARDTs in this new context. We theoretically demonstrate that ARDTs can compute complex functions, such as simulating automata, Turing machines, and sparse circuits, by leveraging "chain-of-thought" computations. Our analysis provides bounds on the size, depth, and computational efficiency of ARDTs, highlighting their surprising computational power. Empirically, we train ARDTs on simple language generation tasks, showing that they can learn to generate coherent and grammatically correct text on par with a smaller Transformer model. Additionally, we show that ARDTs can be used on top of transformer representations to solve complex reasoning tasks. This research reveals the unique computational abilities of ARDTs, aiming to broaden the architectural diversity in language model development.
Persistent DSpace Link
