Bridging Efficiency and Scalability in Generative Modeling
Name
tong-sytong-phd-eecs-2026-thesis.pdf
Size
64.83 MB
Format
Adobe PDF
Checksum (MD5)
7d11314e8357af3101053c6a12ed94e9
Author(s)
Tong, Shangyuan
Advisor(s)
Jaakkola, Tommi S.
Date Issued
May 2026
Publisher
Massachusetts Institute of Technology
Abstract
Generative modeling has progressed from adversarial frameworks to diffusion and flow-based processes that now achieve state-of-the-art results across many domains. As these models have grown more capable, they have also sharpened a practical tension: the methods that deliver the highest quality often demand substantial cost in training, sampling, or deployment. We argue that the tradeoff between efficiency (strong results at lower computational cost) and scalability (improved quality with additional computation) is not absolute. Across six contributions spanning adversarial, diffusion, and flow-based models, we show that, although these objectives are often treated as competing, better objective design, inference-time search, and distillation can in important settings make generative systems both more efficient and more scalable.
The first part of the thesis develops efficient training formulations. In adversarial generative modeling, we show that standard discriminator objectives exhibit alignment instability: the generator receives nonzero gradients even when it has already matched the data distribution, causing unnecessary oscillation. We introduce pairwise discriminators that eliminate this failure mode by making the aligned state stationary regardless of discriminator quality. We further develop Adversarial Support Alignment, replacing full distribution matching with the weaker but sufficient objective of support matching for domain adaptation under label shift. Turning to diffusion models, we introduce the Stable Target Field, which replaces the single-sample score estimation target with an importance-weighted average over a reference batch, substantially reducing target variance at high noise levels and accelerating convergence. We also present PFGM++, a unified framework connecting diffusion and Poisson flow models through a tunable augmented dimension that controls the tradeoff between training ease and sampling robustness.
The second part introduces Inference-Time Scaling for diffusion models. We show that generation quality can continue to improve even after the gains from additional denoising steps have largely saturated by formulating inference as a search problem: candidate noises and trajectories are evaluated with learned verifiers, and test-time compute yields further quality gains. This establishes a new scaling axis for diffusion models, parallel to recent work on test-time compute in large language models.
The final part bridges efficiency and scalability through distillation. While inference-time search produces high-quality outputs, it is too costly for practical deployment. We present FreeFlow, a data-free method for distilling flow models that addresses what we term Teacher-Data Mismatch: the gap between the data-noised intermediates used in standard distillation and the distributions encountered along the teacher's actual generating trajectories. By anchoring distillation at the prior (the unique point where these two processes must agree) and training through a predictor-corrector framework, FreeFlow distills the teacher into a single-function-evaluation student and achieves state-of-the-art one-step results without external data. The resulting model can also serve as a fast proxy for inference-time search, showing how efficient distillation can make scalable inference more practical.
Together, these contributions show that efficiency and scalability in generative modeling need not always oppose one another: better formulations can improve stability and robustness, inference-time search can convert additional compute into better outputs, and efficient distillation can make those gains practical.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
