Exploring a prototype framework of web-based and peer-reviewed “European Educational Research Quality Indicators” (EERQI)

Ton Mooij

    Research output: Contribution to journalArticleAcademicpeer-review

    Abstract

    Digitization, the Internet, and information or webometric interdisciplinary approaches are affecting the fields of Scientometrics and Library and Information Science (LIS). These new approaches can be used to improve citation-only procedures to estimate the quality and impact of research. A European pilot to explore this potential was called “European Educational Research Quality Indicators” (EERQI, FP7 # 217549). An interdisciplinary consortium was involved from 2008-2011. Different types of indicators were developed to score 171 educational research documents. Extrinsic bibliometric and citation indicators were collected from the Internet for each document; intrinsic indicators reflecting content-based quality were developed and relevant data gathered by peer review. Exploratory and confirmatory factor analysis and structural modeling were used to explore statistical relationships among latent factors or concepts and their indicators. Three intrinsic and two extrinsic latent factors were found to be relevant. Moreover, the more a document was related to a reviewer’s own area of research, the higher the score the reviewer gave concerning 1) significance, originality, and consistency, and 2) methodological adequacy. The conclusions are that a prototype EERQI framework has been constructed: intrinsic quality indicators add specific information to extrinsic quality or impact indicators, and vice versa. Also, a problem of “objective” impact scores is that they are based on “subjective” or biased peer-review scores. Peer-review, which is foundational to having a work cited, seems biased and this bias should be controlled or improved by more refined estimates of quality and impact of research. Some suggestions are given and limitations of the pilot are discussed. As the EERQI development approach, instruments, and tools are new, they should be developed further.
    Original languageEnglish
    Pages (from-to)1037-1055
    Number of pages19
    JournalScientometrics
    Volume102
    Issue number1
    Early online date7 Sep 2014
    DOIs
    Publication statusPublished - Jan 2015

    Keywords

    • Quality indicators
    • Impact score
    • Prototype framework
    • Web-based versus peer-reviewed
    • Multimethod assessment
    • Educational research
    • Reviewer's bias
    • Structural modeling

    Fingerprint

    Dive into the research topics of 'Exploring a prototype framework of web-based and peer-reviewed “European Educational Research Quality Indicators” (EERQI)'. Together they form a unique fingerprint.

    Cite this