Automated essay scoring in applied games

Reducing the teacher bandwidth problem in online training

W. Westera, Mihai Dascalu, Hub Kurvers, Stefan Ruseti, Stefan Trausan-Matu

Research output: Contribution to journalArticleAcademicpeer-review

23 Downloads (Pure)

Abstract

This paper presents a methodology for applying automated essay scoring in educational settings. The methodology was tested and validated on a dataset of 173 reports (in Dutch language) that students have created in an applied game on environmental policy. Natural Language Processing technologies from the ReaderBench framework were used to generate an extensive set of textual complexity indices for each of the reports. Afterwards, different machine learning algorithms were used to predict the scores. By combining binary classification (pass or fail) and a probabilistic model for precision, a trade-off can be made between validity of automated score prediction (precision) and the reduction of teacher workload required for manual assessment. It was found from the sample that substantial workload reduction can be achieved, while preserving high precision: allowing for a precision of 95% or higher would already reduce the teacher’s workload to 74%; lowering precision to 80% produces a workload reduction of 50%.
Original languageEnglish
Pages (from-to)212-224
Number of pages13
JournalComputers and Education
Volume123
DOIs
Publication statusPublished - Aug 2018

Fingerprint

Bandwidth
Learning algorithms
Learning systems
Students
Processing
Serious games
Statistical Models

Keywords

  • intelligent tutoring systems
  • architectures for educational technology system
  • interactive learning environments
  • simulations
  • distance education and telelearning
  • Natural Language Processing
  • essay scoring
  • games

Cite this

@article{3095f54a32fa4598b31260b4ec0995f4,
title = "Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training",
abstract = "This paper presents a methodology for applying automated essay scoring in educational settings. The methodology was tested and validated on a dataset of 173 reports (in Dutch language) that students have created in an applied game on environmental policy. Natural Language Processing technologies from the ReaderBench framework were used to generate an extensive set of textual complexity indices for each of the reports. Afterwards, different machine learning algorithms were used to predict the scores. By combining binary classification (pass or fail) and a probabilistic model for precision, a trade-off can be made between validity of automated score prediction (precision) and the reduction of teacher workload required for manual assessment. It was found from the sample that substantial workload reduction can be achieved, while preserving high precision: allowing for a precision of 95{\%} or higher would already reduce the teacher’s workload to 74{\%}; lowering precision to 80{\%} produces a workload reduction of 50{\%}.",
keywords = "intelligent tutoring systems, architectures for educational technology system, interactive learning environments, simulations, distance education and telelearning, Natural Language Processing, essay scoring, games",
author = "W. Westera and Mihai Dascalu and Hub Kurvers and Stefan Ruseti and Stefan Trausan-Matu",
year = "2018",
month = "8",
doi = "10.1016/j.compedu.2018.05.010",
language = "English",
volume = "123",
pages = "212--224",
journal = "Computers and Education",
publisher = "Elsevier",

}

Automated essay scoring in applied games : Reducing the teacher bandwidth problem in online training. / Westera, W.; Dascalu, Mihai; Kurvers, Hub; Ruseti, Stefan; Trausan-Matu, Stefan.

In: Computers and Education, Vol. 123, 08.2018, p. 212-224.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Automated essay scoring in applied games

T2 - Reducing the teacher bandwidth problem in online training

AU - Westera, W.

AU - Dascalu, Mihai

AU - Kurvers, Hub

AU - Ruseti, Stefan

AU - Trausan-Matu, Stefan

PY - 2018/8

Y1 - 2018/8

N2 - This paper presents a methodology for applying automated essay scoring in educational settings. The methodology was tested and validated on a dataset of 173 reports (in Dutch language) that students have created in an applied game on environmental policy. Natural Language Processing technologies from the ReaderBench framework were used to generate an extensive set of textual complexity indices for each of the reports. Afterwards, different machine learning algorithms were used to predict the scores. By combining binary classification (pass or fail) and a probabilistic model for precision, a trade-off can be made between validity of automated score prediction (precision) and the reduction of teacher workload required for manual assessment. It was found from the sample that substantial workload reduction can be achieved, while preserving high precision: allowing for a precision of 95% or higher would already reduce the teacher’s workload to 74%; lowering precision to 80% produces a workload reduction of 50%.

AB - This paper presents a methodology for applying automated essay scoring in educational settings. The methodology was tested and validated on a dataset of 173 reports (in Dutch language) that students have created in an applied game on environmental policy. Natural Language Processing technologies from the ReaderBench framework were used to generate an extensive set of textual complexity indices for each of the reports. Afterwards, different machine learning algorithms were used to predict the scores. By combining binary classification (pass or fail) and a probabilistic model for precision, a trade-off can be made between validity of automated score prediction (precision) and the reduction of teacher workload required for manual assessment. It was found from the sample that substantial workload reduction can be achieved, while preserving high precision: allowing for a precision of 95% or higher would already reduce the teacher’s workload to 74%; lowering precision to 80% produces a workload reduction of 50%.

KW - intelligent tutoring systems

KW - architectures for educational technology system

KW - interactive learning environments

KW - simulations

KW - distance education and telelearning

KW - Natural Language Processing

KW - essay scoring

KW - games

U2 - 10.1016/j.compedu.2018.05.010

DO - 10.1016/j.compedu.2018.05.010

M3 - Article

VL - 123

SP - 212

EP - 224

JO - Computers and Education

JF - Computers and Education

ER -