Grading University Students with LLMs Performance and Acceptance of a Canvas-Based Automation

Teachers in higher education spend considerable time grading assignments rather than tutoring students. Large language models (LLMs) could address this by generating human-like grades and feedback for assignments. However, accounts of their practical application are scarce. We wrote Python code to integrate the Canvas learning management system with an API for GDPR-compliant LLM access. We used this AI system to grade and feedback weekly assignments of 58 graduate students (47 study participants) enrolled in an introductory programming course. LLM-grading was fast, cost efficient, and relatively accurate: GPT-4o and human graders agreed perfectly in 80% of 6345 evaluated student answers. Human and LLM-grades were positively correlated (r=.570-.866 for each assignment) but humans awarded higher grades. Disagreements occurred because human graders overlooked student mistakes, and GPT-4o graded ambiguous cases stricter and misgraded some questions. Importantly, we think most LLM-grading mistakes can be reconciled by avoiding certain tasks and improving grading rubrics. Most students (77%) found the LLM-generated feedback helpful, and some (33%) appreciated the rapid grading. However, multiple students (30%) stated that receiving preliminary LLM-grades that were lower than their actual grades made them anxious or upset. We conclude that LLM-based grading and feedback can, and likely should, be used to optimize teachers’ resources. Canvas users can adopt our pipeline by simply adding their own API keys, Canvas IDs, and assignments: https://github.com/lukekorthals/canvas-llm-integration.