Preserving Human Autonomy in AI-Mediated Negotiations
Name
chen-alvinjc-sm-idss-2025-thesis.pdf
Description
Thesis PDF
Size
1.93 MB
Format
Adobe PDF
Checksum (MD5)
ad1d24a67b213118b4ddd05b8ed00b8e
Author(s)
Chen, J. Alvin
Advisor(s)
Susskind, Lawrence
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
The rapid integration of generative artificial intelligence (AI) into negotiation and conflict resolution processes raises critical ethical concerns about the erosion of human autonomy, particularly when AI systems navigate irreconcilable “sacred” values (non-negotiable moral principles) alongside transactional “mundane” interests. This thesis investigates whether generative AI can be designed to recognize and respect important values and beliefs while preserving human agency in decision-making. Drawing on datasets from a repository of large language model (LLM) prompts tested in simulated negotiation scenarios, this study employs a mixed-methods approach to evaluating AI’s efficacy in balancing efficiency with ethical imperatives in negotiation. Quantitative metrics (enumerating the outcomes of two-party negotiations) are analyzed alongside qualitative assessments of values such as transparency and consent, drawn from Kantian ethical frameworks.
My analysis reveals that while AI negotiating bots excel in trades across mundane, tradable interests they struggle to navigate beliefs and values without oversimplifying moral reasoning or obscuring cultural considerations. These findings inform policy recommendations, including a call for human-in-the-loop validation and technical safeguards for protecting important values in efforts to incorporate AI-assistance into negotiations. By bridging technical analysis and ethical theory, I hope this research contributes to improvements in designing autonomy-preserving AI systems for use in a range of negotiating settings, prioritizing human dignity alongside computational efficiency.
MIT Department
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
