Predicting Question Quality using Recurrent Neural Networks

Stefan Ruseti, Mihai Dascalu, Danielle S. McNamara, S. Crossley, Stefan Trausan-Matu, Amy M. Johnson, Renu Balyan, K.J. Kopp

    Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

    Abstract

    This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this data set based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.
    Original languageEnglish
    Title of host publication19th International Conference on Artificial Intelligence in Education (AIED 2018)
    EditorsC. P. Rosé, R. Martínez-Maldonado, U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, B.D. Boulay
    PublisherSpringer UK
    Pages491-502
    DOIs
    Publication statusPublished - 2018
    EventArtificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27 – 20, 2018 - London, United Kingdom
    Duration: 27 Jun 201830 Jun 2018
    https://aied2018.utscic.edu.au/

    Publication series

    NameLecture Notes in Artificial Intelligence
    PublisherSpringer
    NameLecture Notes in Computer Science
    PublisherSpringer
    ISSN (Print)0302-9743

    Conference

    ConferenceArtificial Intelligence in Education
    Abbreviated titleAIED 2018
    CountryUnited Kingdom
    CityLondon
    Period27/06/1830/06/18
    Internet address

    Fingerprint

    student
    learning
    prediction
    experiment
    index
    machine learning

    Keywords

    • Question asking
    • Recurrent neural network
    • Word embeddings

    Cite this

    Ruseti, S., Dascalu, M., McNamara, D. S., Crossley, S., Trausan-Matu, S., Johnson, A. M., ... Kopp, K. J. (2018). Predicting Question Quality using Recurrent Neural Networks. In C. P. Rosé, R. Martínez-Maldonado, U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, ... B. D. Boulay (Eds.), 19th International Conference on Artificial Intelligence in Education (AIED 2018) (pp. 491-502). (Lecture Notes in Artificial Intelligence), (Lecture Notes in Computer Science). Springer UK. https://doi.org/10.1007/978-3-319-93843-1_36
    Ruseti, Stefan ; Dascalu, Mihai ; McNamara, Danielle S. ; Crossley, S. ; Trausan-Matu, Stefan ; Johnson, Amy M. ; Balyan, Renu ; Kopp, K.J. / Predicting Question Quality using Recurrent Neural Networks. 19th International Conference on Artificial Intelligence in Education (AIED 2018). editor / C. P. Rosé ; R. Martínez-Maldonado ; U. Hoppe ; R. Luckin ; M. Mavrikis ; K. Porayska-Pomsta ; B. McLaren ; B.D. Boulay. Springer UK, 2018. pp. 491-502 (Lecture Notes in Artificial Intelligence). (Lecture Notes in Computer Science).
    @inproceedings{c817d0a97ef64c259f3b73741238ff01,
    title = "Predicting Question Quality using Recurrent Neural Networks",
    abstract = "This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this data set based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22{\%}, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6{\%}). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.",
    keywords = "Question asking, Recurrent neural network, Word embeddings",
    author = "Stefan Ruseti and Mihai Dascalu and McNamara, {Danielle S.} and S. Crossley and Stefan Trausan-Matu and Johnson, {Amy M.} and Renu Balyan and K.J. Kopp",
    note = "This publication reflects only the author's view and the European Commission is not responsible for any use that may be made of the information it contains.",
    year = "2018",
    doi = "10.1007/978-3-319-93843-1_36",
    language = "English",
    series = "Lecture Notes in Artificial Intelligence",
    publisher = "Springer UK",
    pages = "491--502",
    editor = "Ros{\'e}, {C. P. } and R. Mart{\'i}nez-Maldonado and U. Hoppe and R. Luckin and M. Mavrikis and K. Porayska-Pomsta and B. McLaren and B.D. Boulay",
    booktitle = "19th International Conference on Artificial Intelligence in Education (AIED 2018)",
    address = "United Kingdom",

    }

    Ruseti, S, Dascalu, M, McNamara, DS, Crossley, S, Trausan-Matu, S, Johnson, AM, Balyan, R & Kopp, KJ 2018, Predicting Question Quality using Recurrent Neural Networks. in CP Rosé, R Martínez-Maldonado, U Hoppe, R Luckin, M Mavrikis, K Porayska-Pomsta, B McLaren & BD Boulay (eds), 19th International Conference on Artificial Intelligence in Education (AIED 2018). Lecture Notes in Artificial Intelligence, Lecture Notes in Computer Science, Springer UK, pp. 491-502, Artificial Intelligence in Education, London, United Kingdom, 27/06/18. https://doi.org/10.1007/978-3-319-93843-1_36

    Predicting Question Quality using Recurrent Neural Networks. / Ruseti, Stefan; Dascalu, Mihai; McNamara, Danielle S.; Crossley, S.; Trausan-Matu, Stefan; Johnson, Amy M.; Balyan, Renu; Kopp, K.J.

    19th International Conference on Artificial Intelligence in Education (AIED 2018). ed. / C. P. Rosé; R. Martínez-Maldonado; U. Hoppe; R. Luckin; M. Mavrikis; K. Porayska-Pomsta; B. McLaren; B.D. Boulay. Springer UK, 2018. p. 491-502 (Lecture Notes in Artificial Intelligence), (Lecture Notes in Computer Science).

    Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

    TY - GEN

    T1 - Predicting Question Quality using Recurrent Neural Networks

    AU - Ruseti, Stefan

    AU - Dascalu, Mihai

    AU - McNamara, Danielle S.

    AU - Crossley, S.

    AU - Trausan-Matu, Stefan

    AU - Johnson, Amy M.

    AU - Balyan, Renu

    AU - Kopp, K.J.

    N1 - This publication reflects only the author's view and the European Commission is not responsible for any use that may be made of the information it contains.

    PY - 2018

    Y1 - 2018

    N2 - This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this data set based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

    AB - This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this data set based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

    KW - Question asking

    KW - Recurrent neural network

    KW - Word embeddings

    U2 - 10.1007/978-3-319-93843-1_36

    DO - 10.1007/978-3-319-93843-1_36

    M3 - Conference article in proceeding

    T3 - Lecture Notes in Artificial Intelligence

    SP - 491

    EP - 502

    BT - 19th International Conference on Artificial Intelligence in Education (AIED 2018)

    A2 - Rosé, C. P.

    A2 - Martínez-Maldonado, R.

    A2 - Hoppe, U.

    A2 - Luckin, R.

    A2 - Mavrikis, M.

    A2 - Porayska-Pomsta, K.

    A2 - McLaren, B.

    A2 - Boulay, B.D.

    PB - Springer UK

    ER -

    Ruseti S, Dascalu M, McNamara DS, Crossley S, Trausan-Matu S, Johnson AM et al. Predicting Question Quality using Recurrent Neural Networks. In Rosé CP, Martínez-Maldonado R, Hoppe U, Luckin R, Mavrikis M, Porayska-Pomsta K, McLaren B, Boulay BD, editors, 19th International Conference on Artificial Intelligence in Education (AIED 2018). Springer UK. 2018. p. 491-502. (Lecture Notes in Artificial Intelligence). (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-319-93843-1_36