Transformers as Empirical Bayes Estimators The Poisson Model
Name
jabbour-mjabbour-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
3.28 MB
Format
Adobe PDF
Checksum (MD5)
5909a4dd81aa50fde9e5f988cdd4838b
Author(s)
Jabbour, Mark
Advisor(s)
Polyanskiy, Yury
Date Issued
February 2025
Publisher
Massachusetts Institute of Technology
Abstract
We study the ability of transformers to perform In Context Learning (ICL) in the setting of Empirical Bayes for the Poison Model. On the theoretical side, we demonstrate the expressibility of transformers by formulating a way to approximate the Robbins estimator, the first empirical Bayes estimator for the Poisson model. On the empirical side, we show that transformers pre-trained on synthetic data can generalize to unseen prior and sequence lengths, outperforming existing methods like Robbins, NPMLE, and ERM monotone in efficiency and accuracy. By studying the internal behavior of the representations of the intermediate layers of these transformers, we found that the representation converges quickly and smoothly over the layers. We also demonstrate that it’s unlikely transformers are implementing Robbin’s or NPMLE estimators in context.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
