A First Step Towards Synthesizing Rubrics and Video for the Formative Assessment of Complex Skills

. The performance objectives used for the formative assessment of complex skills are generally set through text-based analytic rubrics[1]. Moreover, video modeling examples are a widely applied method of observational learning, providing students with context-rich modeling examples of complex skills that act as an analogy for problem solving [1]. The purpose of this theoretical paper is to synthesize the components of video modeling and rubrics to support the formative assessment of complex skills. Based on theory, we argue that application of the developed Video Enhanced Rubrics (VER) fosters learners’ development of mental models, quality of provided feedback by various actors and finally, the learners mastery of complex skills.


1
Introduction Students in Dutch secondary education follow a standardized curriculum in preparation for further study at the vocational or university level.The standardized curriculum currently mainly tests the knowledge of a student, as opposed to the ability of the student to construct and apply knowledge and to master skills.However, to prepare learners for further education, learners need to manage their learning and be able to construct knowledge and master skills [2].It is important to provide learners with the insight in how to master complex skills, if we expect them to assess and develop complex skills that prepare them for further education.However, several schools in Dutch primary and secondary education have difficulty implementing and evaluating a set of generic complex skills that are beneficial for the learners development also known as 21 st century skills [3].A recent study on the Dutch educational system concludes that current implementations for teaching complex skills lack efficiency, explicitness and structure [4].Reasons for this can found in the individual nature of the school's implementations, resulting in a lack of coordinated and systematic implementation [4].In the assessment of complex skills, rubrics offer teachers a practical and systematical method to analyze the learners performance.Rubrics are effective in assessment settings because they provide transparency of grading criteria to learners [5].For this paper, the term rubrics is used for text-based analytic rubrics.Rubrics provide a detailed description of each level of mastery of a complex skill.This detailed description provides transparent communication of the assessment criteria.The inherent complexity of complex skills can be analyzed through the use of rubrics, making them valuable to teachers.To gain insight into the definition of complex skills, we highlight several characteristics.One of the main characteristics of complex skills is that they are hard to learn, requiring an estimated five hundred hours to acquire [6].Complex skills are skills that are comprised of a set of constituent skills which require conscious processing [7].A single skill, such as typing, differs in complexity from a complex skill such as giving a presentation.Giving a presentation is comprised out of several constituent skills, such as using presentation software, communication and use of information.Textual analytic rubrics describe the inherent complexity of complex skills, but do not provide a modeling example to contextualize the skill and build a complete mental model [1].The focus of this paper is on the synthesis of video and rubrics, that fosters the learners' development of mental models, feedback quality and mastery of complex skills.

Problem definition
Analytic rubrics are one of the main instruments used for the (formative) assessment of complex skills.However, rubrics have several deficiencies we should take into account.Firstly, complex skills are comprised of several constituent skills, generally identified by professionals.Constituent skills must be seen as aspects of the complex skill because a complex skill is more than the sum of its aspects [6].By defining relative proficiency of a complex skill, a rubric describes the quality of the learner's performance in terms of skills mastery levels.However, rubrics provide a purely textual framework to assess the learner's relative proficiency of complex skill aspects.The resulting assessment on aspect-level provides an incomplete and moderately reliable assessment of complex skill [8].In summary, fully assessing complex skills requires an assessment tool with both the ability to assess progress on constituent skill level and assess progress of the coordination, combination and integration of these constituent skills, forming a complete complex skill [1,7].Secondly, constituent skills are formulated as performance objectives in the 4 Component Instructional Design (4CID) method.Using rubrics, the four elements of a performance objective are described on a textual level.These elements are the 1)tools 2)conditions 3)standards and 4)action the learner should perform to meet the performance objective [6].Learners use the performance objectives found in rubrics as a systematic approach to problem solving, enabling low-achieving learners to strategically reach a passing grade [9,10].However, according to the 4CID model a systematic approach to problem solving should not be provided without illustrating relevant examples because a mental model is not only built on the general, abstract knowledge of a skill.A mental model is also built on the concrete case of a professionals' consciously controlled mental processes when applying a complex skill.These concrete cases can be visualized in a video modeling example.Thirdly, the textual form of rubrics inherently lack contextual and dynamic information need by the learner to provide the textual assessment criteria with context and procedural information [11].In summary, imbedding video modeling examples in textual rubrics may create a more complete complex skill description, and secondly may provide a more complete mental model of the complex skill.The accuracy of the created mental model is of importance, as higher accuracy relates to higher performance of the learners' complex skills [12].In summary, we expect video may supplement supportive information and contextualized illustration to textual analytic rubrics.The proposed benefits can be an addition to the transparency, validity, performance and feedback quality's provided by rubrics [1,[13][14][15][16][17][18].

Theoretical framework
Having proposed the introduction of video into the (formative)assessment of complex skills through rubrics, we here explore the theoretical arguments for its effectiveness.
First, we need to be aware of the effects of rubrics and their influence on complex skill assessment and development.The transparent grading criteria found in rubrics is a main contributing factor to their quality and effectiveness, positively influencing the learners' performance and the assessment validity of rubrics [13][14][15][16][17][18].From a learners standpoint, the transparency of rubrics fosters feedback and allows low achieving students to strategically reach a passing grade by providing insight in the minimum requirements per constituent skill [9,10].Furthermore, the transparent assessment criteria found in rubrics reduces anxiety in learners [19][20][21].Another aspect contributing to the transparency of rubrics is the increased accuracy, validity and reliability of the self-and peer assessment of complex skills [5,8,22].One of the methods used to assess the mental model creation of a complex skill is the creation of a contextual map by the learner.By fostering the representation of knowledge through clear textual descriptions, rubrics foster the quality of the learners' conceptual maps [5].Concluding, the transparent skill mastery levels provided by rubrics may foster insight in, feedback on and (formative) assessment of complex skills.
Secondly, by implementing video we are proposing the integration of moving images and audio to textual rubrics.The dynamic superiority effect describes the advantages of video over static visual images [11].This theory states that moving pictures increase the learners' performance because they are remembered better, contain more information, provide more cues to aid retrieval from long term memory, attract more attention from the learner and increase learner engagement [11].Specifically of interest for this paper, the dynamic superiority effect also states that the encoding of a video scene involves extraction of semantic and dynamic information, thereby supplementing the rubrics lack of contextual and dynamic information.In the retrieval of the scene from long term memory, the encoded process is simulated by the learner.This provides a richer mental model of the complex skill through the visualization of the professionals' consciously controlled mental processes when applying a complex skill [1,11].As mentioned in the problem definition, a richer mental model relates to higher performance of the learners' complex skills [12].One aspect of influence on the learning outcomes is the demonstrated mastery level of the provided modeling example.In contradiction to the model-observer similarity (MOS) principle, adults are more effective video models then peers [23].Specifically of interest for the development of the complex skill of collaboration, Kim and Mac-Donough [24] report increased collaborative learning interaction through the implementation of video.The complex skill of presentation has been studied by De Grez et.Al [25], finding learning through video to be more effective then extensive practice in the performance of presentations skills.Concluding, video provides contextual and dynamic information relevant for a complex skill, potentially leading to a richer mental model.However, to benefit from these advantages it is important to specify design considerations behind the combination of video and rubrics.
Thirdly, we focus on the shared aspects of video and rubrics to identify possible overlap.Both rubrics and video have been shown to foster selfassessment, self-regulatory skills and self-efficacy [5,14,17,19,[26][27][28][29][30][31].More importantly for this paper, self-regulated learning and self-efficacy act as a motivational predictor of performance on complex tasks and its constituent processes, such as search, information processing and memory processes that affect learning [29,32,33].Specifically for the development of the complex skill of presenting, De Grez [33] stresses the importance of self-regulated learning as he found significantly increased learning gains as a result of implementing 'self-generated focused learning goals'.Moreover, the self-efficacy fostered by rubrics and video is critical in the development of presentation skills and can result in significant learning gains [34].Concluding, we expect that the combined use of video and rubrics during the process of mastering a complex skill has a positive effect on self-regulated learning, which contributes to the performance of complex skills.
Forth, we subsequently examine theory to facilitate the synthesis of video and rubrics.The cognitive theory of multimedia learning (CTML) states that learners perform significantly better at the transfer of complex skills to new situations through a combined use of video and text, as opposed to text alone [35].Within this theory, several principles for the effective implementation are described.The CTML principles aim to achieve three goals, namely to 1) reduce extraneous processing, 2) manage essential processing and 3) foster generative processing.Firstly, reducing extraneous load is achieved by lightening the load of the instruction.This can be done by excluding extraneous information, signaling, highlighting and avoiding redundancy.Secondly, the goal of managing essential processing is to limit the intrinsic load of the multimedia instruction to the learners' cognitive capacity, preventing essential cognitive overload.Relying mainly on the dual channel theory, this principle states that the visual channel is overloaded by simultaneously observing a video and reading on screen text [36].This can be remedied by offload-ing the visual processing of the text to auditory processing by implementing narration.In order to achieve the final goal, fostering generative processing, the principles based on social cues are introduced.These principles activate a social response, increasing active cognitive processing.According to dual channel theory, active cognitive processing is essential in transferring information from the sensory memory into working memory.This fosters the quality of the learning outcome [37].The stability of the multimedia effect has been recently studied, finding even a relatively simple multimedia implementation to be substantially and persistently more beneficial for retaining knowledge as compared to text alone [38].However, the inherent complexity of a complex skill is challenging for the limited amount of working memory available to the learner.If the principles are implemented incorrectly, cognitive load is increased and mental model creation is limited, impacting performance [12].To ensure effective implementation of video and text, Mayer's [35] CTML relies upon dual channel theory, cognitive load theory and the integrated theory of text and picture recognition.In addition to the principles of CTML, several studies have been done on partial aspects of instructional multimedia design.Eitel & Scheiter's [39] review consisting of 42 studies regarding text and picture sequencing states that it is helpful for comprehension if the medium that contains the least complexity is presented first.Presenting the least complex medium (either text or picture) first may facilitate processing of the complex information presented in the second medium.These findings are in line with both CTML and the 4CID models.The CTML's pre-training principle states that simple information can be used to prime the learner for complex learning [35].The 4CID model ranges task classes and supportive information from simple to complex to accommodate the sequencing principle [40].

Conclusion.
Analytic text-based analytic rubrics mainly contribute to the development of complex skill on a cognitive level, providing rich feedback, anxiety reducing transparency and performance enhancing insight into the performance levels of a complex skill.However, despite these advantages, three problems regarding formative assessment of complex skills using rubrics where defined.The first problem indicates the aspect level of rubric assessment and states that a complex skill is more than the sum of its identified aspects.We expect that the implementation of video combined with text-based rubrics will help to unify the individual aspects of rubrics into a complex skill.We expect video address the first and third problem by 'filling the gaps' between the individual aspects of the complex skill as video provides the learner with the opportunity to personally encode semantic and dynamic information from the modeling example of the complex skill, supplementing the information provided by the rubric and providing personalization.The second problem indicates that the information that a rubric provides when used as a systematic approach to problem solving by a learner is insufficient to form an accurate mental model of a complex skill.For a video to convey the appropriate information to form a mental model, it is of importance that the modeling example conveys the mastery of the complex task in such a manner that the learner can encode the rich mental model found in the actions of the professional performing the task.Concluding, we expect video to provide a rich enhancement to text-based rubrics for the specific use of (formatively) assessing complex skills.However, to ensure effective multimedia implementation it is of importance to adhere to CTML principles.In summary, we have taken a first step towards a synthesis of video and analytic text-based rubrics.We will focus further study on the development of design guidelines for effective implementation of video and analytic text-based rubrics for the (formative) assessment of complex skills in the XXXXX project.Information on the XXXXX project can be found on www.XXXXX.nl.