A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Hieke Keuning, J.T. Jeuring, B.J. Heeren

Research output: Contribution to journalReview articleAcademicpeer-review

Abstract

Formative feedback, aimed at helping students to improve theirwork, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss' feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

Original languageEnglish
Article number3
Number of pages43
JournalACM Transactions on Computing Education
Volume19
Issue number1
DOIs
Publication statusPublished - Jan 2019

Fingerprint

programming
Feedback
Students
literature
student
Labeling
coding
teacher
learning

Keywords

  • ASSIGNMENTS
  • COMPLETION
  • COMPUTER
  • DIAGNOSIS
  • ENVIRONMENT
  • INTELLIGENT TUTORING SYSTEMS
  • KNOWLEDGE
  • SOFTWARE
  • STUDENT PROGRAMS
  • Systematic literature review
  • automated feedback
  • learning programming
  • programming tools

Cite this

@article{fbee68d04a29432d8347631d73ee56d0,
title = "A Systematic Literature Review of Automated Feedback Generation for Programming Exercises",
abstract = "Formative feedback, aimed at helping students to improve theirwork, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss' feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.",
keywords = "ASSIGNMENTS, COMPLETION, COMPUTER, DIAGNOSIS, ENVIRONMENT, INTELLIGENT TUTORING SYSTEMS, KNOWLEDGE, SOFTWARE, STUDENT PROGRAMS, Systematic literature review, automated feedback, learning programming, programming tools",
author = "Hieke Keuning and J.T. Jeuring and B.J. Heeren",
year = "2019",
month = "1",
doi = "10.1145/3231711",
language = "English",
volume = "19",
journal = "ACM Transactions on Computing Education",
issn = "1946-6226",
publisher = "Association for Computing Machinery (ACM)",
number = "1",

}

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. / Keuning, Hieke; Jeuring, J.T.; Heeren, B.J.

In: ACM Transactions on Computing Education, Vol. 19, No. 1, 3, 01.2019.

Research output: Contribution to journalReview articleAcademicpeer-review

TY - JOUR

T1 - A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

AU - Keuning, Hieke

AU - Jeuring, J.T.

AU - Heeren, B.J.

PY - 2019/1

Y1 - 2019/1

N2 - Formative feedback, aimed at helping students to improve theirwork, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss' feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

AB - Formative feedback, aimed at helping students to improve theirwork, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss' feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

KW - ASSIGNMENTS

KW - COMPLETION

KW - COMPUTER

KW - DIAGNOSIS

KW - ENVIRONMENT

KW - INTELLIGENT TUTORING SYSTEMS

KW - KNOWLEDGE

KW - SOFTWARE

KW - STUDENT PROGRAMS

KW - Systematic literature review

KW - automated feedback

KW - learning programming

KW - programming tools

U2 - 10.1145/3231711

DO - 10.1145/3231711

M3 - Review article

VL - 19

JO - ACM Transactions on Computing Education

JF - ACM Transactions on Computing Education

SN - 1946-6226

IS - 1

M1 - 3

ER -