Scoring Summaries Using Recurrent Neural Networks

Stefan Ruseti, Mihai Dascalu, Amy M. Johnson, Danielle S. McNamara, Renu Balyan, Kathryn S. McCarthy, Stefan Trausan-Matu, Roger Nkambou (Editor), Roger Azevedo (Editor), Julita Vassileva (Editor)

    Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

    Abstract

    Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55% accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary. Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.
    Original languageEnglish
    Title of host publicationIntelligent Tutoring Systems.
    Subtitle of host publicationITS 2018.
    EditorsR. Nkambou, R. Azevedo, J. Vassileva
    PublisherSpringer
    Pages191-201
    Number of pages11
    DOIs
    Publication statusPublished - 17 May 2018
    Event Intelligent Tutoring Systems. ITS 2018. : Lecture Notes in Computer Science. - Montreal, Canada
    Duration: 11 Jun 201815 Jun 2018
    Conference number: vol 10858

    Conference

    Conference Intelligent Tutoring Systems. ITS 2018.
    CountryCanada
    CityMontreal
    Period11/06/1815/06/18

    Fingerprint

    Recurrent neural networks
    Semantics
    Experiments

    Cite this

    Ruseti, S., Dascalu, M., Johnson, A. M., McNamara, D. S., Balyan, R., McCarthy, K. S., ... Vassileva, J. (Ed.) (2018). Scoring Summaries Using Recurrent Neural Networks. In R. Nkambou, R. Azevedo, & J. Vassileva (Eds.), Intelligent Tutoring Systems.: ITS 2018. (pp. 191-201). Springer. https://doi.org/10.1007/978-3-319-91464-0_19
    Ruseti, Stefan ; Dascalu, Mihai ; Johnson, Amy M. ; McNamara, Danielle S. ; Balyan, Renu ; McCarthy, Kathryn S. ; Trausan-Matu, Stefan ; Nkambou, Roger (Editor) ; Azevedo, Roger (Editor) ; Vassileva, Julita (Editor). / Scoring Summaries Using Recurrent Neural Networks. Intelligent Tutoring Systems.: ITS 2018.. editor / R. Nkambou ; R. Azevedo ; J. Vassileva. Springer, 2018. pp. 191-201
    @inproceedings{6beafcfd7e6947a4ad62c873e4675cbc,
    title = "Scoring Summaries Using Recurrent Neural Networks",
    abstract = "Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55{\%} accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary. Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.",
    author = "Stefan Ruseti and Mihai Dascalu and Johnson, {Amy M.} and McNamara, {Danielle S.} and Renu Balyan and McCarthy, {Kathryn S.} and Stefan Trausan-Matu and Roger Nkambou and Roger Azevedo and Julita Vassileva",
    year = "2018",
    month = "5",
    day = "17",
    doi = "10.1007/978-3-319-91464-0_19",
    language = "English",
    pages = "191--201",
    editor = "R. Nkambou and R. Azevedo and J. Vassileva",
    booktitle = "Intelligent Tutoring Systems.",
    publisher = "Springer",

    }

    Ruseti, S, Dascalu, M, Johnson, AM, McNamara, DS, Balyan, R, McCarthy, KS, Trausan-Matu, S, Nkambou, R (ed.), Azevedo, R (ed.) & Vassileva, J (ed.) 2018, Scoring Summaries Using Recurrent Neural Networks. in R Nkambou, R Azevedo & J Vassileva (eds), Intelligent Tutoring Systems.: ITS 2018.. Springer, pp. 191-201, Intelligent Tutoring Systems. ITS 2018. , Montreal, Canada, 11/06/18. https://doi.org/10.1007/978-3-319-91464-0_19

    Scoring Summaries Using Recurrent Neural Networks. / Ruseti, Stefan; Dascalu, Mihai; Johnson, Amy M.; McNamara, Danielle S.; Balyan, Renu; McCarthy, Kathryn S.; Trausan-Matu, Stefan; Nkambou, Roger (Editor); Azevedo, Roger (Editor); Vassileva, Julita (Editor).

    Intelligent Tutoring Systems.: ITS 2018.. ed. / R. Nkambou; R. Azevedo; J. Vassileva. Springer, 2018. p. 191-201.

    Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

    TY - GEN

    T1 - Scoring Summaries Using Recurrent Neural Networks

    AU - Ruseti, Stefan

    AU - Dascalu, Mihai

    AU - Johnson, Amy M.

    AU - McNamara, Danielle S.

    AU - Balyan, Renu

    AU - McCarthy, Kathryn S.

    AU - Trausan-Matu, Stefan

    A2 - Nkambou, Roger

    A2 - Azevedo, Roger

    A2 - Vassileva, Julita

    A2 - Nkambou, R.

    A2 - Azevedo, R.

    A2 - Vassileva, J.

    PY - 2018/5/17

    Y1 - 2018/5/17

    N2 - Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55% accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary. Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.

    AB - Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55% accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary. Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.

    U2 - 10.1007/978-3-319-91464-0_19

    DO - 10.1007/978-3-319-91464-0_19

    M3 - Conference article in proceeding

    SP - 191

    EP - 201

    BT - Intelligent Tutoring Systems.

    PB - Springer

    ER -

    Ruseti S, Dascalu M, Johnson AM, McNamara DS, Balyan R, McCarthy KS et al. Scoring Summaries Using Recurrent Neural Networks. In Nkambou R, Azevedo R, Vassileva J, editors, Intelligent Tutoring Systems.: ITS 2018.. Springer. 2018. p. 191-201 https://doi.org/10.1007/978-3-319-91464-0_19