Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
Name
zhou-caiz428-sm-eecs-2026-thesis.pdf
Size
851.81 KB
Format
Adobe PDF
Checksum (MD5)
d5074241e2a7bf7c986dba7933bb5dd1
Author(s)
Zhou, Cai
Advisor(s)
Jaakkola, Tommi
Bates, Stephen
Date Issued
May 2026
Publisher
Massachusetts Institute of Technology
Abstract
In this paper we introduce Hierarchical Diffusion Language Models (HDLM) – a novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semantics are surjectively mapped to highlevel tokens with coarse-grained meanings. In the forward process, each token is independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides a general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as a special case. We also propose practical training techniques based on the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
