Performance assessment in serious games: Compensating for the effects of randomness

Wim Westera*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41% of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players´ achievements.
Original languageEnglish
Pages (from-to)681-697
Number of pages17
JournalEducation and Information Technologies
Volume21
Issue number3
Early online date20 Aug 2014
DOIs
Publication statusPublished - May 2016

Fingerprint

performance assessment
performance
simulation

Keywords

  • serious gaming
  • performance assessment
  • performance
  • assessment
  • cut score
  • computer-assisted lelarning
  • monte carlo
  • randomness
  • serious
  • gaming

Cite this

@article{d358e0483fef490bbb676f800793545a,
title = "Performance assessment in serious games: Compensating for the effects of randomness",
abstract = "This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41{\%} of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players´ achievements.",
keywords = "serious gaming, performance assessment, performance, assessment, cut score, computer-assisted lelarning, monte carlo, randomness, serious, gaming",
author = "Wim Westera",
year = "2016",
month = "5",
doi = "10.1007/s10639-014-9347-3",
language = "English",
volume = "21",
pages = "681--697",
journal = "Education and Information Technologies",
issn = "1360-2357",
publisher = "Springer US",
number = "3",

}

Performance assessment in serious games: Compensating for the effects of randomness. / Westera, Wim.

In: Education and Information Technologies, Vol. 21, No. 3, 05.2016, p. 681-697.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Performance assessment in serious games: Compensating for the effects of randomness

AU - Westera, Wim

PY - 2016/5

Y1 - 2016/5

N2 - This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41% of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players´ achievements.

AB - This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41% of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players´ achievements.

KW - serious gaming

KW - performance assessment

KW - performance

KW - assessment

KW - cut score

KW - computer-assisted lelarning

KW - monte carlo

KW - randomness

KW - serious

KW - gaming

U2 - 10.1007/s10639-014-9347-3

DO - 10.1007/s10639-014-9347-3

M3 - Article

VL - 21

SP - 681

EP - 697

JO - Education and Information Technologies

JF - Education and Information Technologies

SN - 1360-2357

IS - 3

ER -