Providing meaningful feedback for autograding of programming assignments

Georgiana Haldeman, Andrew Tjang, Monica Babeş-Vroman, Stephen Bartos, Jay Shah, Danielle Yucht, Thu D. Nguyen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Scopus citations

Abstract

Autograding systems are increasingly being deployed to meet the challenge of teaching programming at scale. We propose a methodology for extending autograders to provide meaningful feedback for incorrect programs. Our methodology starts with the instructor identifying the concepts and skills important to each programming assignment, designing the assignment, and designing a comprehensive test suite. Tests are then applied to code submissions to learn classes of common errors and produce classifiers to automatically categorize errors in future submissions. The instructor maps the errors to concepts and skills and writes hints to help students find their misconceptions and mistakes. We have applied the methodology to two assignments from our Introduction to Computer Science course. We used submissions from one semester of the class to build classifiers and write hints for observed common errors. We manually validated the automatic error categorization and potential usefulness of the hints using submissions from a second semester. We found that the hints given for erroneous submissions should be helpful for 96% or more of the cases. Based on these promising results, we have deployed our hints and are currently collecting submissions and feedback from students and instructors.

Original languageEnglish (US)
Title of host publicationSIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education
PublisherAssociation for Computing Machinery, Inc
Pages278-283
Number of pages6
ISBN (Electronic)9781450351034
DOIs
StatePublished - Feb 21 2018
Event49th ACM Technical Symposium on Computer Science Education, SIGCSE 2018 - Baltimore, United States
Duration: Feb 21 2018Feb 24 2018

Publication series

NameSIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education
Volume2018-January

Other

Other49th ACM Technical Symposium on Computer Science Education, SIGCSE 2018
CountryUnited States
CityBaltimore
Period2/21/182/24/18

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Education

Keywords

  • Autograding
  • Concepts/skills-based hints
  • Error categorization

Fingerprint Dive into the research topics of 'Providing meaningful feedback for autograding of programming assignments'. Together they form a unique fingerprint.

Cite this