Predicting Question Quality using Recurrent Neural Networks

Stefan Ruseti, Mihai Dascalu, Danielle S. McNamara, S. Crossley, Stefan Trausan-Matu, Amy M. Johnson, Renu Balyan, K.J. Kopp

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

Abstract

This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this data set based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.
Original languageEnglish
Title of host publication19th International Conference on Artificial Intelligence in Education (AIED 2018)
EditorsC. P. Rosé, R. Martínez-Maldonado, U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, B.D. Boulay
PublisherSpringer UK
Pages491-502
DOIs
Publication statusPublished - 2018
Externally publishedYes
EventArtificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27 – 20, 2018 - London, United Kingdom
Duration: 27 Jun 201830 Jun 2018
https://aied2018.utscic.edu.au/

Publication series

SeriesLecture Notes in Artificial Intelligence (subseries)
SeriesLecture Notes in Computer Science
ISSN0302-9743

Conference

ConferenceArtificial Intelligence in Education
Abbreviated titleAIED 2018
CountryUnited Kingdom
CityLondon
Period27/06/1830/06/18
Internet address

Keywords

  • Question asking
  • Recurrent neural network
  • Word embeddings

Fingerprint Dive into the research topics of 'Predicting Question Quality using Recurrent Neural Networks'. Together they form a unique fingerprint.

Cite this