2026-09-18T19:36:12Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1624152025-08-22T03:05:59Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
O'Reilly, Una-May
Chiricescu, Silviu
Westenfelder, Finnian Ellis
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
2025-08-21T17:00:28Z
2025-08-21T17:00:28Z
2025-05
2025-06-16T14:46:58.289Z
https://hdl.handle.net/1721.1/162415
https://orcid.org/0009-0004-2533-9884
The Bourne-Again Shell (Bash) command-line interface for Linux systems has complex syntax and requires extensive specialized knowledge. Using the natural language to Bash command (NL2SH) translation capabilities of large language models (LLMs) for command composition alleviates these issues. However, the NL2SH performance of LLMs is difficult to assess due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. We present a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous datasets by 441% and 135%, respectively. Further, we present a novel functional equivalence heuristic that combines command execution with LLM evaluation of command outputs. Our heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heuristics. Evaluation of popular LLMs using our test dataset and heuristic demonstrates that parsing, in-context learning, in-weight learning, and constrained decoding can improve NL2SH accuracy by up to 32%. Additionally, we consider military use cases for NL2SH models and discuss the limitations of current Department of Defense documentation standards for LLMs. We write and publish documentation for our models and datasets to promote safe use. Our findings emphasize the importance of dataset quality, execution-based evaluation, translation method, and proper documentation for advancing NL2SH translation and enabling responsible use. Our code is available at https://github.com/westenfelder/NL2SH.
S.M.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
LLM-Supported Natural Language to Bash Translation
Thesis
application/pdf
Master
Master of Science in Technology and Policy
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="13e83379-ca61-464e-84d1-118a8f654926">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>LLM-Supported Natural Language to Bash Translation</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2025-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Westenfelder, Finnian Ellis</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>The Bourne-Again Shell (Bash) command-line interface for Linux systems has complex syntax and requires extensive specialized knowledge. Using the natural language to Bash command (NL2SH) translation capabilities of large language models (LLMs) for command composition alleviates these issues. However, the NL2SH performance of LLMs is difficult to assess due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. We present a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous datasets by 441% and 135%, respectively. Further, we present a novel functional equivalence heuristic that combines command execution with LLM evaluation of command outputs. Our heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heuristics. Evaluation of popular LLMs using our test dataset and heuristic demonstrates that parsing, in-context learning, in-weight learning, and constrained decoding can improve NL2SH accuracy by up to 32%. Additionally, we consider military use cases for NL2SH models and discuss the limitations of current Department of Defense documentation standards for LLMs. We write and publish documentation for our models and datasets to promote safe use. Our findings emphasize the importance of dataset quality, execution-based evaluation, translation method, and proper documentation for advancing NL2SH translation and enabling responsible use. Our code is available at https://github.com/westenfelder/NL2SH.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>