Graph-aware isomorphic attention for adaptive dynamics in transformers
Name
026108_1_5.0256873.pdf
Description
Published version
Size
11.61 MB
Format
Adobe PDF
Checksum (MD5)
0384b13fe029e21e1c0c2716e93df42b
Author(s)
Buehler, Markus J
Date Issued
April 22, 2025
Journal
APL Machine Learning
Publisher
AIP Publishing
Citation
Markus J. Buehler; Graph-aware isomorphic attention for adaptive dynamics in transformers. APL Mach. Learn. 1 June 2025; 3 (2): 026108.
Version
Final published version
Abstract
We present an approach for modifying transformer architectures by integrating graph-aware relational reasoning into the attention mechanism, merging concepts from graph neural networks and language modeling. Building on the inherent connection between attention and graph theory, we reformulate the transformer’s attention mechanism as a graph operation and propose graph-aware isomorphic attention. This method leverages advanced graph modeling strategies, including Graph Isomorphism Networks (GINs), to enrich the representation of relational structures. Our approach improves the model’s ability to capture complex dependencies and generalize across tasks, as evidenced by a reduced generalization gap and improved learning performance. We expand the concept of graph-aware attention to introduce sparse-GIN-attention, a fine-tuning approach that enhances the adaptability of pre-trained foundational models with minimal computational overhead, endowing them with graph-aware capabilities. We show that the sparse-GIN-attention framework leverages compositional principles from category theory to align relational reasoning with sparsified graph structures while modeling hierarchical representation learning that bridges local interactions and global task objectives across diverse domains. Our results demonstrate that graph-aware attention mechanisms outperform traditional attention in both training efficiency and validation performance. These insights bridge graph theory and transformer architectures and uncover latent graph-like structures within traditional attention mechanisms, offering a new lens through which transformers can be optimized. By evolving transformers as hierarchical GIN models, we reveal their implicit capacity for graph-level relational reasoning with profound implications for foundational model development and applications in bioinformatics, materials science, language modeling, and beyond, setting the stage for interpretable and generalizable modeling strategies.
MIT Department
Massachusetts Institute of Technology. Laboratory for Atomistic and Molecular Mechanics
Terms of Use
Creative Commons Attribution-NonCommercial-NoDerivatives
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1063/5.0256873
