Bring It on!

Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench

Marilena Panaite, Mihai Dascalu, Amy M. Johnson, Renu Balyan, Jianmin Dai, Danielle S. McNamara, Stefan Trausan-Matu

    Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

    7 Downloads (Pure)

    Abstract

    Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).
    Original languageEnglish
    Title of host publicationArtificial Intelligence in Education
    Subtitle of host publication19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I
    EditorsC. P. Rosé, R. Martínez-Maldonado, H. U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, B. d. Boulay
    Place of PublicationCham
    PublisherSpringer International Publishing AG
    Pages409-419
    Number of pages11
    ISBN (Electronic)9783319938431
    ISBN (Print)9783319938424
    DOIs
    Publication statusPublished - 2018
    EventInternational Conference, AIED 2018 - London, United Kingdom
    Duration: 27 Jun 201830 Jun 2018
    https://aied2018.utscic.edu.au/

    Publication series

    NameLecture Notes in Computer Science
    PublisherSpringer
    Volume10947
    NameLecture Notes in Artificial Intelligence
    PublisherSpringer

    Conference

    ConferenceInternational Conference, AIED 2018
    Abbreviated titleAIED 2018
    CountryUnited Kingdom
    CityLondon
    Period27/06/1830/06/18
    Internet address

    Fingerprint

    Intelligent systems
    Knowledge based systems
    Processing
    Learning algorithms
    Learning systems
    Students
    Feedback
    Virtual machine

    Keywords

    • Natural language processing
    • Intelligent tutoring systems
    • Self-explanations
    • Support vector machines
    • ReaderBench

    Cite this

    Panaite, M., Dascalu, M., Johnson, A. M., Balyan, R., Dai, J., McNamara, D. S., & Trausan-Matu, S. (2018). Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. In C. P. Rosé, R. Martínez-Maldonado, H. U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, ... B. D. Boulay (Eds.), Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I (pp. 409-419). (Lecture Notes in Computer Science; Vol. 10947), (Lecture Notes in Artificial Intelligence). Cham: Springer International Publishing AG. https://doi.org/10.1007/978-3-319-93843-1_30
    Panaite, Marilena ; Dascalu, Mihai ; Johnson, Amy M. ; Balyan, Renu ; Dai, Jianmin ; McNamara, Danielle S. ; Trausan-Matu, Stefan. / Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. editor / C. P. Rosé ; R. Martínez-Maldonado, ; H. U. Hoppe ; R. Luckin ; M. Mavrikis ; K. Porayska-Pomsta ; B. McLaren ; B. d. Boulay. Cham : Springer International Publishing AG, 2018. pp. 409-419 (Lecture Notes in Computer Science). (Lecture Notes in Artificial Intelligence).
    @inproceedings{693f4195f6b2467daa1fbdcc08552c2d,
    title = "Bring It on!: Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench",
    abstract = "Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59{\%} (adjacent accuracy = 97{\%}; Kappa = .43).",
    keywords = "Natural language processing, Intelligent tutoring systems, Self-explanations, Support vector machines, ReaderBench",
    author = "Marilena Panaite and Mihai Dascalu and Johnson, {Amy M.} and Renu Balyan and Jianmin Dai and McNamara, {Danielle S.} and Stefan Trausan-Matu",
    note = "This publication reflects only the author's view and the European Commission is not responsible for any use that may be made of the information it contains.",
    year = "2018",
    doi = "10.1007/978-3-319-93843-1_30",
    language = "English",
    isbn = "9783319938424",
    series = "Lecture Notes in Computer Science",
    publisher = "Springer International Publishing AG",
    pages = "409--419",
    editor = "Ros{\'e}, {C. P. } and Mart{\'i}nez-Maldonado,, {R. } and Hoppe, {H. U. } and Luckin, {R. } and Mavrikis, {M. } and K. Porayska-Pomsta and McLaren, {B. } and Boulay, {B. d. }",
    booktitle = "Artificial Intelligence in Education",
    address = "Switzerland",

    }

    Panaite, M, Dascalu, M, Johnson, AM, Balyan, R, Dai, J, McNamara, DS & Trausan-Matu, S 2018, Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. in CP Rosé, R Martínez-Maldonado, HU Hoppe, R Luckin, M Mavrikis, K Porayska-Pomsta, B McLaren & BD Boulay (eds), Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. Lecture Notes in Computer Science, vol. 10947, Lecture Notes in Artificial Intelligence, Springer International Publishing AG, Cham, pp. 409-419, International Conference, AIED 2018, London, United Kingdom, 27/06/18. https://doi.org/10.1007/978-3-319-93843-1_30

    Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. / Panaite, Marilena; Dascalu, Mihai; Johnson, Amy M.; Balyan, Renu; Dai, Jianmin ; McNamara, Danielle S.; Trausan-Matu, Stefan.

    Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. ed. / C. P. Rosé; R. Martínez-Maldonado,; H. U. Hoppe; R. Luckin; M. Mavrikis; K. Porayska-Pomsta; B. McLaren; B. d. Boulay. Cham : Springer International Publishing AG, 2018. p. 409-419 (Lecture Notes in Computer Science; Vol. 10947), (Lecture Notes in Artificial Intelligence).

    Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

    TY - GEN

    T1 - Bring It on!

    T2 - Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench

    AU - Panaite, Marilena

    AU - Dascalu, Mihai

    AU - Johnson, Amy M.

    AU - Balyan, Renu

    AU - Dai, Jianmin

    AU - McNamara, Danielle S.

    AU - Trausan-Matu, Stefan

    N1 - This publication reflects only the author's view and the European Commission is not responsible for any use that may be made of the information it contains.

    PY - 2018

    Y1 - 2018

    N2 - Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).

    AB - Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).

    KW - Natural language processing

    KW - Intelligent tutoring systems

    KW - Self-explanations

    KW - Support vector machines

    KW - ReaderBench

    U2 - 10.1007/978-3-319-93843-1_30

    DO - 10.1007/978-3-319-93843-1_30

    M3 - Conference article in proceeding

    SN - 9783319938424

    T3 - Lecture Notes in Computer Science

    SP - 409

    EP - 419

    BT - Artificial Intelligence in Education

    A2 - Rosé, C. P.

    A2 - Martínez-Maldonado,, R.

    A2 - Hoppe, H. U.

    A2 - Luckin, R.

    A2 - Mavrikis, M.

    A2 - Porayska-Pomsta, K.

    A2 - McLaren, B.

    A2 - Boulay, B. d.

    PB - Springer International Publishing AG

    CY - Cham

    ER -

    Panaite M, Dascalu M, Johnson AM, Balyan R, Dai J, McNamara DS et al. Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. In Rosé CP, Martínez-Maldonado, R, Hoppe HU, Luckin R, Mavrikis M, Porayska-Pomsta K, McLaren B, Boulay BD, editors, Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. Cham: Springer International Publishing AG. 2018. p. 409-419. (Lecture Notes in Computer Science). (Lecture Notes in Artificial Intelligence). https://doi.org/10.1007/978-3-319-93843-1_30