Linguistic differences between self-written and generative AI-assisted student texts in Dutch bachelor CS education

Generative AI, such as ChatGPT, has made it possible for students to generate texts and submit them as their own work. This has implications for assessment in formal education, and therefore institutions are exploring ways to detect their use. This study investigates the potential of linguistic analysis for this purpose, focusing on text characteristics such as readability, perplexity and sentiment. A quantitative experiment examined the extent to which student-written texts differ from texts that were written with the help of ChatGPT. For the study, first-year students in a bachelor computer science program at a Dutch university of applied sciences wrote 65 short texts, approximately half of which were written with the help of ChatGPT. Although the multivariate test did not show significant results, univariate analyses indicated that the student-written texts were easier to read, showed higher perplexity and contrary to expectation, less evaluative language. A discriminant analysis showed that text origins could be determined with 94% accuracy using these characteristics. Additionally, 21 teachers classified random subsets of texts. 76% were correctly classified, but 19% of self-written texts were mistakenly identified as written with the help of generative AI. The results suggest that linguistic characteristics can provide an indication of text origin, but both automatic and human analysis should be approached with caution due to their limited accuracy.