Towards Human-Centered Optimality Criteria
Name
ghandeharioun-asma_gh-phd-MAS-2021-thesis.pdf
Description
Thesis PDF
Size
9.02 MB
Format
Adobe PDF
Checksum (MD5)
d2123a62f104865788b145f6cfbf7a30
Author(s)
Ghandeharioun, Asma
Advisor(s)
Picard, Rosalind W.
Date Issued
June 2021
Publisher
Massachusetts Institute of Technology
Abstract
Despite the transformational success of machine learning across various applications, examples of deployed models failing to recognize and support human-centered (HC) criteria are abundant. In this thesis, I conceptualize the space of human-machine collaboration with respect to two components: interpretation of people by machines and interpretation of machines by people. I develop several tools that make improvements along these axes.
First, I develop a pipeline that predicts depressive symptoms rated by clinicians from real-world longitudinal data outperforming several baselines. Second, I introduce a novel, model-agnostic, and dataset-agnostic method to approximate interactive human evaluation in open-domain dialog through self-play that is more strongly correlated with human evaluations than other automated metrics commonly used today. While dialog quality evaluation metrics predominantly use word-level overlap or distance metrics based on embedding resemblance to each turn of the conversation, I show the significance of taking into account the conversation's trajectory and using proxies such as sentiment, semantics, and user engagement that are psychologically motivated. Third, I demonstrate an uncertainty measurement technique that helps disambiguate annotator disagreement and data bias. I show that this characterization also improves model performance. Finally, I present a novel method that allows humans to investigate a predictor's decision-making process to gain better insight into how it works. The method jointly trains a generator, a discriminator, and a concept disentangler, allowing the human to ask "what-if" questions. I evaluate it on several challenging synthetic and realistic datasets where previous methods fall short of satisfying desirable criteria for interpretability and show that our method performs consistently well across all. I discuss its applications to detect potential biases of a classifier and identify spurious artifacts that impact predictions using simulated experiments.
Together, these novel techniques and insights provide a more comprehensive interpretation of people by machines and more powerful tools for interpretation of machines by people that can move us closer to HC optimality.
MIT Department
Program in Media Arts and Sciences (Massachusetts Institute of Technology)
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
