Responsible Computational Text Generation: AI Content Classification and Policy Framework

Abstract

Recent advances in generative AI, particularly in producing human-like text, have blurred the lines between human and AI authorship. Since these AI tools rely on stochastic generation rather than traditional scientific reasoning, concerns about misinformation and reliability have emerged, highlighting the need for AI detection tools and policy guidelines. In response, this study proposes a dual approach: (1) the application of adaptive thresholds to improve the use of AI text detectors and (2) an AI policy framework based on user patterns and opinions. To enhance detector performance, we present a threshold optimization algorithm that adapts to diverse subgroups, such as those based on text lengths and stylistic features, thereby reducing discrepancies in error rates. The commonly used method relies on a single universal threshold, which has led to inconsistent results across various text types because of different probability distributions. Our approach addresses these shortcomings by tailoring thresholds to the specific characteristics of each group. In parallel, the study examines the pressing need for comprehensive AI guidelines, given the rise of misinformation and academic integrity issues. While a few institutions have introduced comprehensive policies, many institutes lack approaches grounded in user patterns and opinions. To remedy this problem, we propose a policy framework based on a user study. The findings of this research will provide practical solutions for more effective AI text classification and a reliable framework for the necessity of AI writing policies.