Assessing User Expertise in Spoken Dialog System Interactions

Eugenio Ribeiro, Fernando Batista, Isabel Trancoso, Jose Lopes, Ricardo Ribeiro, David de Matos

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

Abstract

Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can be used in offline processes of root cause analysis. However, not much effort has been put into automatically identifying the level of expertise of an user, especially in dialog-based interactions. In this paper we present an approach based on a specific set of task related features. Based on the distribution of the features among the two classes – Novice and Expert – we used Random Forests as a classification approach. Furthermore, we used a Support Vector Machine classifier, in order to perform a result comparison. By applying these approaches on data from a real system, Let’s Go, we obtained preliminary results that we consider positive, given the difficulty of the task and the lack of competing approaches for comparison.
Original languageEnglish
Title of host publicationAdvances in Speech and Language Technologies for Iberian Languages
Subtitle of host publicationThird International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings
EditorsAlberto Abad, Alfonso Ortega, António Teixeira, Carmen García Mateo, Carlos D. Martínez Hinarejos, Fernando Perdigão, Fernando Batista, Nuno Mamede
PublisherSpringer Nature Switzerland AG
Pages245-254
ISBN (Electronic)978-3-319-49169-1
ISBN (Print)978-3-319-49168-4
DOIs
Publication statusPublished - Nov 2016
Externally publishedYes
EventInternational Conference on Advances in Speech and Language Technologies for Iberian Languages - Lisbon, Portugal
Duration: 23 Nov 201625 Nov 2016
https://link.springer.com/conference/iberspeech

Publication series

NameLecture Notes in Computer Science book series
PublisherSpringer Nature
Volume10077
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349
NameLecture Notes in Artificial Intelligence book sub series
PublisherSpringer Nature
Volume10077
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceInternational Conference on Advances in Speech and Language Technologies for Iberian Languages
Abbreviated titleIberSPEECH 2016
CountryPortugal
CityLisbon
Period23/11/1625/11/16
Internet address

Fingerprint

Support vector machines
Classifiers

Keywords

  • User expertise
  • Let’s Go
  • SVM
  • Random Forest

Cite this

Ribeiro, E., Batista, F., Trancoso, I., Lopes, J., Ribeiro, R., & de Matos, D. (2016). Assessing User Expertise in Spoken Dialog System Interactions. In A. Abad, A. Ortega, A. Teixeira, C. García Mateo, C. D. Martínez Hinarejos, F. Perdigão, F. Batista, ... N. Mamede (Eds.), Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings (pp. 245-254). (Lecture Notes in Computer Science book series; Vol. 10077), (Lecture Notes in Artificial Intelligence book sub series; Vol. 10077). Springer Nature Switzerland AG. https://doi.org/10.1007/978-3-319-49169-1_24
Ribeiro, Eugenio ; Batista, Fernando ; Trancoso, Isabel ; Lopes, Jose ; Ribeiro, Ricardo ; de Matos, David. / Assessing User Expertise in Spoken Dialog System Interactions. Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings. editor / Alberto Abad ; Alfonso Ortega ; António Teixeira ; Carmen García Mateo ; Carlos D. Martínez Hinarejos ; Fernando Perdigão ; Fernando Batista ; Nuno Mamede. Springer Nature Switzerland AG, 2016. pp. 245-254 (Lecture Notes in Computer Science book series). (Lecture Notes in Artificial Intelligence book sub series).
@inproceedings{a6924e8a08324e3682a0cca3d9b04fe5,
title = "Assessing User Expertise in Spoken Dialog System Interactions",
abstract = "Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can be used in offline processes of root cause analysis. However, not much effort has been put into automatically identifying the level of expertise of an user, especially in dialog-based interactions. In this paper we present an approach based on a specific set of task related features. Based on the distribution of the features among the two classes – Novice and Expert – we used Random Forests as a classification approach. Furthermore, we used a Support Vector Machine classifier, in order to perform a result comparison. By applying these approaches on data from a real system, Let’s Go, we obtained preliminary results that we consider positive, given the difficulty of the task and the lack of competing approaches for comparison.",
keywords = "User expertise, Let’s Go, SVM, Random Forest",
author = "Eugenio Ribeiro and Fernando Batista and Isabel Trancoso and Jose Lopes and Ricardo Ribeiro and {de Matos}, David",
year = "2016",
month = "11",
doi = "10.1007/978-3-319-49169-1_24",
language = "English",
isbn = "978-3-319-49168-4",
series = "Lecture Notes in Computer Science book series",
publisher = "Springer Nature Switzerland AG",
pages = "245--254",
editor = "Abad, {Alberto } and Ortega, {Alfonso } and Teixeira, {Ant{\'o}nio } and {Garc{\'i}a Mateo}, {Carmen } and {D. Mart{\'i}nez Hinarejos}, {Carlos } and Perdig{\~a}o, {Fernando } and Batista, {Fernando } and Mamede, {Nuno }",
booktitle = "Advances in Speech and Language Technologies for Iberian Languages",
address = "Switzerland",

}

Ribeiro, E, Batista, F, Trancoso, I, Lopes, J, Ribeiro, R & de Matos, D 2016, Assessing User Expertise in Spoken Dialog System Interactions. in A Abad, A Ortega, A Teixeira, C García Mateo, C D. Martínez Hinarejos, F Perdigão, F Batista & N Mamede (eds), Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings. Lecture Notes in Computer Science book series, vol. 10077, Lecture Notes in Artificial Intelligence book sub series, vol. 10077, Springer Nature Switzerland AG, pp. 245-254, International Conference on Advances in Speech and Language Technologies for Iberian Languages, Lisbon, Portugal, 23/11/16. https://doi.org/10.1007/978-3-319-49169-1_24

Assessing User Expertise in Spoken Dialog System Interactions. / Ribeiro, Eugenio; Batista, Fernando; Trancoso, Isabel; Lopes, Jose; Ribeiro, Ricardo; de Matos, David.

Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings. ed. / Alberto Abad; Alfonso Ortega; António Teixeira; Carmen García Mateo; Carlos D. Martínez Hinarejos; Fernando Perdigão; Fernando Batista; Nuno Mamede. Springer Nature Switzerland AG, 2016. p. 245-254 (Lecture Notes in Computer Science book series; Vol. 10077), (Lecture Notes in Artificial Intelligence book sub series; Vol. 10077).

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

TY - GEN

T1 - Assessing User Expertise in Spoken Dialog System Interactions

AU - Ribeiro, Eugenio

AU - Batista, Fernando

AU - Trancoso, Isabel

AU - Lopes, Jose

AU - Ribeiro, Ricardo

AU - de Matos, David

PY - 2016/11

Y1 - 2016/11

N2 - Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can be used in offline processes of root cause analysis. However, not much effort has been put into automatically identifying the level of expertise of an user, especially in dialog-based interactions. In this paper we present an approach based on a specific set of task related features. Based on the distribution of the features among the two classes – Novice and Expert – we used Random Forests as a classification approach. Furthermore, we used a Support Vector Machine classifier, in order to perform a result comparison. By applying these approaches on data from a real system, Let’s Go, we obtained preliminary results that we consider positive, given the difficulty of the task and the lack of competing approaches for comparison.

AB - Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can be used in offline processes of root cause analysis. However, not much effort has been put into automatically identifying the level of expertise of an user, especially in dialog-based interactions. In this paper we present an approach based on a specific set of task related features. Based on the distribution of the features among the two classes – Novice and Expert – we used Random Forests as a classification approach. Furthermore, we used a Support Vector Machine classifier, in order to perform a result comparison. By applying these approaches on data from a real system, Let’s Go, we obtained preliminary results that we consider positive, given the difficulty of the task and the lack of competing approaches for comparison.

KW - User expertise

KW - Let’s Go

KW - SVM

KW - Random Forest

U2 - 10.1007/978-3-319-49169-1_24

DO - 10.1007/978-3-319-49169-1_24

M3 - Conference article in proceeding

SN - 978-3-319-49168-4

T3 - Lecture Notes in Computer Science book series

SP - 245

EP - 254

BT - Advances in Speech and Language Technologies for Iberian Languages

A2 - Abad, Alberto

A2 - Ortega, Alfonso

A2 - Teixeira, António

A2 - García Mateo, Carmen

A2 - D. Martínez Hinarejos, Carlos

A2 - Perdigão, Fernando

A2 - Batista, Fernando

A2 - Mamede, Nuno

PB - Springer Nature Switzerland AG

ER -

Ribeiro E, Batista F, Trancoso I, Lopes J, Ribeiro R, de Matos D. Assessing User Expertise in Spoken Dialog System Interactions. In Abad A, Ortega A, Teixeira A, García Mateo C, D. Martínez Hinarejos C, Perdigão F, Batista F, Mamede N, editors, Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings. Springer Nature Switzerland AG. 2016. p. 245-254. (Lecture Notes in Computer Science book series). (Lecture Notes in Artificial Intelligence book sub series). https://doi.org/10.1007/978-3-319-49169-1_24