Artificial Intelligence for Derivative Security Classification: Applications to DoD
Name
ANDREW GELBARD.pdf
Description
Technical Report
Size
1.66 MB
Format
Adobe PDF
Checksum (MD5)
2defd2f2c4646da0a7173eb5f5284c9f
Author(s)
Gelbard, Andrew
Hamilton, Lei
Date Issued
September 10, 2025
Abstract
The accurate classification of government
documents according to their sensitivity (e.g., UNCLASSIFIED,
SECRET, TOP SECRET) is critical for national
security, yet historically has relied on time-intensive
manual review. The current manual classification process
consumes millions of labor hours annually within the
U.S. government, significantly diverting skilled personnel
from essential analytical tasks. This research explores
automating this security classification task using recently
available declassified materials from the DISC
dataset [1], addressing practical challenges such as
noisy Optical Character Recognition (OCR) output,
imbalanced data distributions, and potential leakage
of explicit classification markers within document text.
This dataset contains declassified government documents
sourced from the Digital National Security Archive, providing
authentic textual examples representative of actual
classification scenarios. We evaluate both traditional
machine learning approaches and advanced transformerbased
language models to classify documents accurately
across multiple sensitivity levels. Our results highlight
that transformer-based models, particularly DeBERTa,
effectively improve identification of the minority but
critical TOP SECRET class, achieving recall over 70%
and an overall balanced performance (macro F1 score of
0.75), while traditional methods exhibit similar overall
accuracy but struggle with minority class recall. Despite
promising findings, we caution that conclusions drawn
here remain constrained by limited training data size
and inherent uncertainties in human-labeled documents.
We emphasize the need for larger, rigorously preprocessed
datasets and suggest future research integrating
authoritative classification guidelines directly into model
training, potentially via retrieval-augmented methods.
This work thus contributes a foundational, reproducible
framework that demonstrates significant potential for
machine-assisted security classification, guiding future
research and practical applications in the information
security domain.
Subjects
Air Force Artificial Intelligence Accelerator
Artificial Intelligence
Derivative Security Classification
MIT Department
Lincoln Laboratory
Persistent DSpace Link
