Balancing Memory Efficiency and Accuracy in
Spectral-Based Graph Transformers
Name
ho-kellyho7-meng-eecs-2023-thesis.pdf
Description
Thesis PDF
Size
755.49 KB
Format
Adobe PDF
Checksum (MD5)
ace365a4ccda7443311730b9072bb724
Author(s)
Ho, Kelly
Advisor(s)
Arvind
Date Issued
September 2023
Publisher
Massachusetts Institute of Technology
Abstract
The transformer architecture has been a significant driving force behind advancements in deep learning, yet transformer-based models for graph representation learning have not caught up to mainstream Graph Neural Network (GNN) variants. A major limitation is the large O(𝑛2) memory consumption of graph transformers, where 𝑛 is the number of nodes. Therefore, we develop a memory-efficient graph transformer for node classification, capable of handling graphs with thousands of nodes while maintaining accuracy. Specifically, we reduce the memory use in the attention mechanism and add a random-walk positional encoding to improve upon the SAN graph transformer architecture. We evaluate our model on standard node classification benchmarks: Cora, Citeseer, and Chameleon. Unlike SAN, which runs out of memory, our memory-efficient graph transformer can be run on these benchmarks. Compared with landmark GNN models GCN and GAT, our graph transformer requires 27.92% less memory while being competitive in accuracy.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
