Bring It on!: Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench

Marilena Panaite; Mihai Dascalu; Amy M. Johnson; Renu Balyan; Jianmin  Dai; Danielle S. McNamara; Stefan Trausan-Matu

doi:10.1007/978-3-319-93843-1_30

Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench

Marilena Panaite, Mihai Dascalu, Amy M. Johnson, Renu Balyan, Jianmin Dai, Danielle S. McNamara, Stefan Trausan-Matu

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

243 Downloads (Pure)

Abstract

Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).

Original language	English
Title of host publication	Artificial Intelligence in Education
Subtitle of host publication	19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I
Editors	Manolis Mavrikis, Carolyn Penstein Rosé, Bruce McLaren, H. Ulrich Hoppe, Rose Luckin, Kaska Porayska-Pomsta, Benedict du Boulay, Roberto Martinez-Maldonado
Place of Publication	Cham
Publisher	Springer International Publishing AG
Pages	409-419
Number of pages	11
ISBN (Electronic)	9783319938431
ISBN (Print)	9783319938424
DOIs	https://doi.org/10.1007/978-3-319-93843-1_30
Publication status	Published - 2018
Externally published	Yes
Event	International Conference, AIED 2018 - London, United Kingdom Duration: 27 Jun 2018 → 30 Jun 2018 https://aied2018.utscic.edu.au/

Publication series

Series	Lecture Notes in Computer Science
Volume	10947

Series	Lecture Notes in Artificial Intelligence (subseries)

Conference

Conference	International Conference, AIED 2018
Abbreviated title	AIED 2018
Country/Territory	United Kingdom
City	London
Period	27/06/18 → 30/06/18
Internet address	https://aied2018.utscic.edu.au/

Keywords

Intelligent tutoring systems
Natural language processing
ReaderBench
Self-explanations
Support vector machines

Access to Document

10.1007/978-3-319-93843-1_30

panaite2018Proof, 1.41 MBLicence: CC BY-NC-SA

Rage: Realising an Applied Gaming Eco-system
Westera, W. (PI), Georgiadis, K. (CoI), Saveski, G. (CoI), van Lankveld, G. (CoI), Bahreini, K. (CoI), van der Vegt, W. (CoI), Berkhout, J. (CoI), Nyamsuren, E. (CoI), Kluijfhout, E. (CoI) & Nadolski, R. (CoI)
European Union
1/02/15 → 31/07/19
Project: Research

Cite this

Panaite, M., Dascalu, M., Johnson, A. M., Balyan, R., Dai, J., McNamara, D. S., & Trausan-Matu, S. (2018). Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. In M. Mavrikis, C. Penstein Rosé, B. McLaren, H. U. Hoppe, R. Luckin, K. Porayska-Pomsta, B. du Boulay, & R. Martinez-Maldonado (Eds.), Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I (pp. 409-419). Springer International Publishing AG. https://doi.org/10.1007/978-3-319-93843-1_30

Panaite, Marilena ; Dascalu, Mihai ; Johnson, Amy M. et al. / Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. editor / Manolis Mavrikis ; Carolyn Penstein Rosé ; Bruce McLaren ; H. Ulrich Hoppe ; Rose Luckin ; Kaska Porayska-Pomsta ; Benedict du Boulay ; Roberto Martinez-Maldonado. Cham : Springer International Publishing AG, 2018. pp. 409-419 (Lecture Notes in Computer Science, Vol. 10947). (Lecture Notes in Artificial Intelligence (subseries)).

@inproceedings{693f4195f6b2467daa1fbdcc08552c2d,

title = "Bring It on!: Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench",

abstract = "Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students{\textquoteright} practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).",

keywords = "Intelligent tutoring systems, Natural language processing, ReaderBench, Self-explanations, Support vector machines",

author = "Marilena Panaite and Mihai Dascalu and Johnson, {Amy M.} and Renu Balyan and Jianmin Dai and McNamara, {Danielle S.} and Stefan Trausan-Matu",

note = "This publication reflects only the author's view and the European Commission is not responsible for any use that may be made of the information it contains.; International Conference, AIED 2018, AIED 2018 ; Conference date: 27-06-2018 Through 30-06-2018",

year = "2018",

doi = "10.1007/978-3-319-93843-1_30",

language = "English",

isbn = "9783319938424",

series = "Lecture Notes in Computer Science",

publisher = "Springer International Publishing AG",

pages = "409--419",

editor = "Manolis Mavrikis and {Penstein Ros{\'e}}, Carolyn and Bruce McLaren and Hoppe, {H. Ulrich} and Rose Luckin and Kaska Porayska-Pomsta and {du Boulay}, Benedict and Roberto Martinez-Maldonado",

booktitle = "Artificial Intelligence in Education",

address = "Switzerland",

url = "https://aied2018.utscic.edu.au/",

}

Panaite, M, Dascalu, M, Johnson, AM, Balyan, R, Dai, J, McNamara, DS & Trausan-Matu, S 2018, Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. in M Mavrikis, C Penstein Rosé, B McLaren, HU Hoppe, R Luckin, K Porayska-Pomsta, B du Boulay & R Martinez-Maldonado (eds), Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. Springer International Publishing AG, Cham, Lecture Notes in Computer Science, vol. 10947, Lecture Notes in Artificial Intelligence (subseries), pp. 409-419, International Conference, AIED 2018, London, United Kingdom, 27/06/18. https://doi.org/10.1007/978-3-319-93843-1_30

Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. / Panaite, Marilena; Dascalu, Mihai; Johnson, Amy M. et al.
Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. ed. / Manolis Mavrikis; Carolyn Penstein Rosé; Bruce McLaren; H. Ulrich Hoppe; Rose Luckin; Kaska Porayska-Pomsta; Benedict du Boulay; Roberto Martinez-Maldonado. Cham: Springer International Publishing AG, 2018. p. 409-419 (Lecture Notes in Computer Science, Vol. 10947). (Lecture Notes in Artificial Intelligence (subseries)).

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

TY - GEN

T1 - Bring It on!

T2 - International Conference, AIED 2018

AU - Panaite, Marilena

AU - Dascalu, Mihai

AU - Johnson, Amy M.

AU - Balyan, Renu

AU - Dai, Jianmin

AU - McNamara, Danielle S.

AU - Trausan-Matu, Stefan

N1 - This publication reflects only the author's view and the European Commission is not responsible for any use that may be made of the information it contains.

PY - 2018

Y1 - 2018

N2 - Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).

AB - Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).

KW - Intelligent tutoring systems

KW - Natural language processing

KW - ReaderBench

KW - Self-explanations

KW - Support vector machines

U2 - 10.1007/978-3-319-93843-1_30

DO - 10.1007/978-3-319-93843-1_30

M3 - Conference Article in proceeding

SN - 9783319938424

T3 - Lecture Notes in Computer Science

SP - 409

EP - 419

BT - Artificial Intelligence in Education

A2 - Mavrikis, Manolis

A2 - Penstein Rosé, Carolyn

A2 - McLaren, Bruce

A2 - Hoppe, H. Ulrich

A2 - Luckin, Rose

A2 - Porayska-Pomsta, Kaska

A2 - du Boulay, Benedict

A2 - Martinez-Maldonado, Roberto

PB - Springer International Publishing AG

CY - Cham

Y2 - 27 June 2018 through 30 June 2018

ER -

Panaite M, Dascalu M, Johnson AM, Balyan R, Dai J, McNamara DS et al. Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench. In Mavrikis M, Penstein Rosé C, McLaren B, Hoppe HU, Luckin R, Porayska-Pomsta K, du Boulay B, Martinez-Maldonado R, editors, Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part I. Cham: Springer International Publishing AG. 2018. p. 409-419. (Lecture Notes in Computer Science, Vol. 10947). (Lecture Notes in Artificial Intelligence (subseries)). doi: 10.1007/978-3-319-93843-1_30