Comparative Analysis of ChatGPT's Writing Evaluations and Human Evaluations: The Unveiled Results
In the realm of education, the integration of Artificial Intelligence (AI) is increasingly being explored, particularly in the arena of writing feedback and assessment. A series of recent studies have shed light on the potential benefits and limitations of AI chatbots like ChatGPT in this context.
One notable study, conducted in Türkiye with fourth-grade students, found that ChatGPT's feedback significantly contributed to the development of students' creative writing abilities and their confidence in writing [1]. This suggests a promising role for AI feedback under teacher guidance in primary education curricula.
However, another study, though not yet fully published, is comparing ChatGPT-generated feedback to traditional human supervisor feedback, indicating growing scholarly interest in rigorously comparing AI and human feedback in higher education contexts [5].
While AI chatbots have shown effectiveness in providing writing feedback and positively impacting creative writing skills and writing self-efficacy, they are not without limitations. AI feedback is context-dependent, often focused on specific tasks like essay writing, and may not yet fully replicate the nuanced guidance human teachers provide [3]. For instance, AI may not offer emotional support or personalized feedback that human teachers can provide.
In a study comparing AI-generated assessments to human ones, AI performed marginally better in an assessment where only a score was given, while humans performed better in a study that focused on written feedback [4]. When comparing a couple of different AI systems, AI was found to be more reliable than humans, with human matches at approximately 43% and AI matches between 50 and 82% [4].
The current research used specifically designed prompts that were tested and vetted by experts in using technology for writing instruction, which may not reflect the prompts classroom teachers would write for AI [2]. This underscores the need for ongoing research and development to ensure AI tools are tailored to the needs of educators and students.
The studies did not involve a comparison of AI-generated essays with human-written essays, nor did they compare AI-generated assessments with assessments from other subject areas beyond history [4]. Implementing AI technology in schools requires more training and emphasis on digital literacy for both students and teachers.
In summary, AI chatbots like ChatGPT are an increasingly effective complement to human teachers in providing writing feedback, especially for developmental and creative aspects, but currently work best when integrated with human oversight and instruction [1][3][5]. They offer a promising tool for students to improve their work before submission and can serve as a time-saving tool for teachers. However, the conversation around AI grading may change in the near future as AI technology continues to improve. Unsanctioned use, ethical concerns, and careful consideration of how AI will be implemented in the classroom are important issues to address.
- Teachers in schools could benefit from integrating digital AI tools, such as ChatGPT, into their lessons, as these tools can provide effective writing feedback, particularly for creative and developmental aspects.
- In higher education, there's growing scholarly interest in comparing AI-generated feedback with traditional human feedback to understand their differences and potential benefits.
- AI feedback, while context-dependent and not yet fully capable of offering emotional support or personalized feedback, has demonstrated its ability to enhance student learning in creative writing and writing self-efficacy.
- When it comes to assessments, AI may exhibit superior reliability, but a well-rounded education may still require human involvement, as AI may not yet fully replicate the nuanced guidance that human teachers provide.
