Hierarchical Reinforcement Learning for Open-Domain Dialog
Name
1909.07547.pdf
Description
Accepted version
Size
1.52 MB
Format
Adobe PDF
Checksum (MD5)
de5a4e4c03188484a038eb5a7263503c
Author(s)
Saleh, Abdelrhman
Jaques, Natasha
Ghandeharioun, Asma
Shen, Judy
Picard, Rosalind W.
Date Issued
2020
Journal
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Citation
Saleh, Abdelrhman, Jaques, Natasha, Ghandeharioun, Asma, Shen, Judy and Picard, Rosalind. 2020. "Hierarchical Reinforcement Learning for Open-Domain Dialog." Proceedings of the AAAI Conference on Artificial Intelligence, 34 (05).
Version
Author's final manuscript
Abstract
Open-domain dialog generation is a challenging problem; maximum likelihood training can lead to repetitive outputs, models have difficulty tracking long-term conversational goals, and training on standard movie or online datasets may lead to the generation of inappropriate, biased, or offensive text. Reinforcement Learning (RL) is a powerful framework that could potentially address these issues, for example by allowing a dialog model to optimize for reducing toxicity and repetitiveness. However, previous approaches which apply RL to open-domain dialog generation do so at the word level, making it difficult for the model to learn proper credit assignment for long-term conversational rewards. In this paper, we propose a novel approach to hierarchical reinforcement learning (HRL), VHRL, which uses policy gradients to tune the utterance-level embedding of a variational sequence model. This hierarchical approach provides greater flexibility for learning long-term, conversational rewards. We use self-play and RL to optimize for a set of human-centered conversation metrics, and show that our approach provides significant improvements – in terms of both human evaluation and automatic metrics – over state-of-the-art dialog models, including Transformers.
MIT Department
Program in Media Arts and Sciences (Massachusetts Institute of Technology)
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1609/AAAI.V34I05.6400
