User-centric evaluation of recommender systems in social learning platforms: Accuracy is just the tip of the iceberg

S. Fazeli, H.J. Drachsler, M.E. Bitter-Rijpkema, F.M.R. Brouns, G.W. van der Vegt, P.B. Sloep

Research output: Contribution to journalArticleAcademicpeer-review

18 Downloads (Pure)

Abstract

Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity.
Original languageEnglish
Pages (from-to)294 - 306
Number of pages13
JournalIEEE Transactions on Learning Technologies
Volume11
Issue number3
DOIs
Publication statusPublished - 2018

Fingerprint

Recommender systems
social learning
electronic learning
evaluation
data analysis
value added
performance
candidacy

Keywords

  • Recommender systems
  • Measurement
  • Prediction algorithms
  • Metadata
  • Crowdsourcing
  • Social network services
  • Tagging
  • evaluation
  • social
  • learning
  • accuracy
  • performance

Cite this

@article{bea2bf9c5c944586aa0a1a10ec2e50c8,
title = "User-centric evaluation of recommender systems in social learning platforms: Accuracy is just the tip of the iceberg",
abstract = "Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity.",
keywords = "Recommender systems, Measurement, Prediction algorithms, Metadata, Crowdsourcing, Social network services, Tagging, evaluation, social, learning, accuracy, performance",
author = "S. Fazeli and H.J. Drachsler and M.E. Bitter-Rijpkema and F.M.R. Brouns and {van der Vegt}, G.W. and P.B. Sloep",
year = "2018",
doi = "10.1109/TLT.2017.2732349",
language = "English",
volume = "11",
pages = "294 -- 306",
journal = "IEEE Transactions on Learning Technologies",
issn = "1939-1382",
publisher = "IEEE",
number = "3",

}

User-centric evaluation of recommender systems in social learning platforms: Accuracy is just the tip of the iceberg. / Fazeli, S.; Drachsler, H.J.; Bitter-Rijpkema, M.E.; Brouns, F.M.R.; van der Vegt, G.W.; Sloep, P.B.

In: IEEE Transactions on Learning Technologies, Vol. 11, No. 3, 2018, p. 294 - 306.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - User-centric evaluation of recommender systems in social learning platforms: Accuracy is just the tip of the iceberg

AU - Fazeli, S.

AU - Drachsler, H.J.

AU - Bitter-Rijpkema, M.E.

AU - Brouns, F.M.R.

AU - van der Vegt, G.W.

AU - Sloep, P.B.

PY - 2018

Y1 - 2018

N2 - Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity.

AB - Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity.

KW - Recommender systems

KW - Measurement

KW - Prediction algorithms

KW - Metadata

KW - Crowdsourcing

KW - Social network services

KW - Tagging

KW - evaluation

KW - social

KW - learning

KW - accuracy

KW - performance

U2 - 10.1109/TLT.2017.2732349

DO - 10.1109/TLT.2017.2732349

M3 - Article

VL - 11

SP - 294

EP - 306

JO - IEEE Transactions on Learning Technologies

JF - IEEE Transactions on Learning Technologies

SN - 1939-1382

IS - 3

ER -