Detecting AI-Generated Text and Code: An Empirical Study of Cross-Generator and Cross-Domain Generalization

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Large language models (LLMs) now generate fluent natural language and source code, creating challenges for authorship attribution, academic integrity, and software supply-chain security. Most existing detectors for AI-generated content are evaluated separately on natural language or source code, often under matched train–test conditions that can overestimate real-world reliability. We present a paired-prompt benchmark for human-versus-machine detection across English text, Python code, and mixed text–code documents. The benchmark includes 22,141 instances from HC3, CodeSearchNet, MBPP, and HumanEval across training, validation, and test partitions, plus Mix-Eval, a mixed-content set of 997 Jupyter-notebook-style samples. We evaluate RoBERTa-large for text, GraphCodeBERT and CodeBERT-base for code, a unified RoBERTa-base detector trained on both modalities, and zero-shot baselines. Fine-tuned detectors achieve near-perfect in-distribution performance, with AUROC 1.0000±0.0000 and accuracy above 99.5%. Across five instruction-tuned generator families of varying size (3.8B–7B) and architecture, with the human and problem distributions held fixed, cross-generator transfer causes negligible degradation (AUROC spread 0.0002; drops of at most 0.0003). In contrast, domain shift is the main failure mode: on MBPP+HumanEval, GraphCodeBERT drops to 0.85±0.02 AUROC and CodeBERT-base to 0.67±0.02. On Mix-Eval, the unified detector outperforms a routed text–code pipeline by 21 AUROC points (0.96 vs. 0.75), largely because of router failures on mixed inputs. Training-time augmentation improves low-false-positive performance, while legacy supervised detectors show systematic class inversion on modern LLM outputs. These results show that reliable deployment requires cross-domain evaluation, mixed-content testing, and calibration beyond in-distribution accuracy.