Systematic Review of Design-Based Research Progress: Is a Little Knowledge a Dangerous Thing?

Sufficient attention and resources have been allocated to design-based research (DBR) to warrant review concerning if and how its potential has been realized. Because the DBR literature clearly indicates that this type of research strives toward both the development of an intervention to address a problem in practice and empirical investigation yielding theoretical understanding that can inform the work of others, thoughtful assessment of DBR progress must devote substantial attention to each of these aspects. This requires an in-depth analysis of full-text reports of DBR, framed by a refined conceptualization of the intended outputs of DBR, and ideally, complemented by empirical investigation involving design-based research participants directly.

genre of inquiry is fulfilling its promise.Terry Anderson and Julie Shattuck (2012) tackled this challenge in a recent issue of Educational Researcher.After carefully describing key characteristics of DBR, the authors then present an analysis of the 47 most-cited DBR articles of the last decade.Their findings reveal that DBR is being utilized increasingly, especially in K-12 contexts with technological interventions, and that most interventions yield (potentially) improved learning outcomes or student attitudes.We agree wholeheartedly with Anderson and Shattuck about the importance of assessing what the last decade of DBR has brought to the field of education and concur with most of the characteristics of DBR that they sketch, which align well with others cited in literature (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003;Reinking & Bradley, 2008;Wang & Hannafin, 2005).This work seems thoughtfully conducted and we are grateful to the authors for their review and portrait of the field.However, as fellow scholars interested in DBR, we feel compelled to point out that the Anderson and Shattuck (2012) review still leaves the question of if, and to what extent, DBR is meeting its potential, largely unanswered.
Anderson and Shattuck, as well as others, note that DBR strives toward two main goals, simultaneously: advancing theoretical understanding and development of an intervention in the real world (DBRC, 2003).These twin pursuits represent both a defining feature (McKenney & Reeves, in press) and a key challenge of DBR (Phillips & Dolle, 2006).
While their review does report that nearly all articles addressed a dual focus on theory building and practical applications, the authors provide extremely limited justification for this claim by way of example, and in their overview of major results.We contend that attending to the issues of if, how, and to what extent, individual and collective studies are having impact, i.e. informing theoretical understanding and serving to improve practice, is central to the question of whether or not DBR is meeting its potential.Following several concerns about the conceptualization of DBR, we express our skepticism about the usefulness of a systematic review of scientific literature as the sole source of information for assessing the progress of a nascent field.

Conceptualizing DBR and its potential
To a great extent, we concur with the conceptualization of DBR as presented by Anderson & Shattuck (2012), which characterizes DBR as: situated in real educational contexts; focusing on the design and testing of interventions; using mixed methods; involving multiple iterations; stemming from partnership between researchers and practitioners; yielding design principles; different from action research and concerned with an impact on practice.However, we find it misleading to refer to DBR as a "new research methodology" (p.16) since DBR uses existing quantitative and qualitative research methods and follows established norms for sampling, data collection and data analysis.It is not so much the methods, as it is the goals, which set DBR apart from other genres of research (Bereiter, 2002).Researchers using DBR have characterized their work as belonging to what Stokes (1997) referred to as Pasteur's Quadrant (cf.Roschelle, Bakia, Toyama & Patton, 2011), thus blending the goals of basic and applied science.
McKenney and Reeves (2012) discuss five types of research goals (descriptive, interpretive, predictive, development, and action) and note that the theoretical understanding emerging from DBR can be descriptive, explanatory or predictive in nature; it can also be more prescriptive/normative, providing guidance for future design efforts.
Moreover, we feel compelled to point out an important characteristic missing from Anderson & Shattuck's characterization of DBR: Departing from a problem.The idea that DBR is initiated to address problems that are both scientifically and practically significant has been repeatedly addressed in the literature (Edelson, 2002;Ejersbo et al., 2008) and remains, in our opinion, central to the approach.Thus, thoughtful examination of DBR progress would have to include examination of the kinds of problems tackled, what renders them -from theoretical and practical perspectives -both researchable and research-worthy (McKenney & Reeves, 2012), and with what results.We submit that greater insight could be gleaned from the studies reviewed if a more nuanced framework were used to examine the kinds of solutions developed, the types of theoretical understanding generated and, because these are inextricably linked to their root causes, the nature of problems that are being tackled by DBR.

Literature review for DBR progress: Opportunities and constraints
Because DBR studies depart from a problem and then pursue both knowledge and interventions that address it, we view the systematic review conducted by Anderson and Shattuck (2012) as only partially suitable for assessing the DBR progress.This has to do with two main concerns: methodological fit given the stated purpose; and decisions yielding the data set.
One of the main goals of DBR is to generate theoretical understanding that can be of value to others.Given that theoretical understanding is most often shared through scientific literature, it would seem that a systematic review is well suited to the task.It is therefore disappointing that this area receives very little treatment, and no discussion of the finding that only 33% of the studies yielded "new understandings about educational phenomena."At the same time, comparatively more attention is given to the interventions designed.While Anderson and Shattuck do note that many studies show beneficial effects (i.e.14% show improvements in student attitudes, epistemology or motivation; 33% indicate increases in student learning; and/or 20% demonstrate potential for improved student learning), we find it overly optimistic to view them as indicative of genuinely improving practice.This is because many individual DBR projects reported in literature are insufficiently mature to warrant reports of their specific impact on practice, due to the fact that they are, among other factors, still in the midst of empirical tuning (notably, this was the case with 79% of the studies analyzed); conducted under exceptionally favorable circumstances, thereby limiting the ecological validity of the results (Linn, 2000;Wong, Boticki, Sun, & Looi, 2011); or predominantly driven by the researchers with insufficient practitioner input despite researchers' efforts to involve practitioners (Leeman & Wardekker, 2011).Thus, many published studies are currently able to demonstrate impact potential but not genuine impact on practice.This raises the question of whether the field has matured to a point where assessment of progress through systematic review is responsible, or if such a process risks encouraging premature conclusions about the state of DBR in education.There are individual DBR studies that have yielded clear improvements in practice (e.g.Barab, Gresalfi, & Ingram-Goble, 2010;Clark & Dede, 2009;Swan, 2007), but because far more published studies appear to ongoing than completed, we submit that either (a) the search strategy would have to specifically seek out mature studies; or (b) examination of literature would also have to be accompanied by field-based investigation involving practitioners to seek evidence of any genuine impact resulting from DBR.
While a systematic search through scientific literature may be suited to gaining some insight into the kinds of theoretical understanding and practical impact generated by DBR studies, we find it necessary to comment on several decisions made in the Anderson and Shattuck study which detract from its ability to accurately estimate DBR progress.First, the search strategy relied on use of the term, "design-based research" and, in contrast with recommended practice (cf.Cooper, 2009) provided no alternatives such as "design research", "formative research" or "design experiments" despite the fact that many experts have pointed to the abundance of terminologies that have been used to describe this nascent field (Reinking & Bradley, 2008;van den Akker, 1999).Second, citation analysis is a logical starting point but remains potentially weak as the sole basis for narrowing selection, given known flaws with most systems, e.g.no distinctions between positive and negative citations (Togia & Tsigilis, 2006).Third, given the focus on DBR, the choice to (a) lump scientific books into the category of "grey literature" (e.g. the Kelly, Lesh and Baek [2008] which was made possible by a grant from the National Science Foundation and contains contributions from leading experts in the field can hardly be termed "grey"); and then (b) omit all books seems dubious.Several books on or related to DBR have been published, three of which have been widely cited in the literature though admittedly, one does fall outside the time frame of this study (Kelly, Lesh, & Baek, 2008;van den Akker, Branch, Gustafson, Nieveen, & Plomp, 1999;van den Akker, Gravemeijer, McKenney, & Nieveen, 2006).Especially when it comes to conceptualizing the process of DBR, we feel that omission of these sources weakens the findings of the review considerably.To give an impression of their impact on theoretical understanding, we provide here the Google Scholar citation analysis of one chapter from each of these books derived the Publish or Perish system used by Anderson and Shattuck: van den Akker (1999) was cited 289 times; Reeves (2006) 117 times, andKali (2008) 20 times.Given that the median number of citations in the reported review set appears to have been 55 (as gleaned from the online supplement), we find it difficult to justify omission of publications like these.Finally, because abstract quality is notoriously varied (cf.Hahs-Vaughn & Onwuegbuzie, 2010), we find the choice to base several analyses and conclusions on abstracts alone to be dubious.We support Anderson and Shattuck's (2012) recommendation for further analyses based on the full texts, but also express our concern that the findings from this study could be misleading due to the potential presence of poor quality abstracts in the data set.

Conclusion
In sum, we applaud Anderson & Shattuck for their (2012) investigation into the state of the art of DBR.We value their results and find it useful to observe that the field is growing and what the studies focus on, as well as to note that iterative development and testing of interventions in practice appears promising.However, we find this review insufficient for assessing "a decade of progress," and have cited both conceptual and methodological reasons for this.Because the DBR literature emphasizes that inquiry departs from a problem, we assert that assessment of DBR progress must attend to the nature of problems being tackled by DBR, while also typifying the kinds of theoretical understanding and solutions to practical problems that are generated.Most of our methodological concerns related to the selection of the data set.Perhaps we are particularly sensitive to the methodological limitations of literature review for assessing DBR progress due to our own struggles in this area (Ormel, Pareja Roblin, McKenney, Voogt & Pieters, in press), which we have experienced as being both insightful and humbling.Noting heated discussions concerning systematic reviews (Evans & Benefield, 2000;Hammersley, 2001) and the candid portrayal of how reviewers grapple with their tasks (Nind, 2006), we suspect that the kinds of concerns addressed here are not necessarily unique to a systematic review of DBR.In any case, we urge those interested in assessing DBR progress over the last decade to seek nuanced judgment of how well the field is meeting its two main goals of advancing theoretical understanding and benefitting practice.Where completed studies are not available from which impact data could be gleaned, a review of scientific literature alone is insufficient.A lack of completed studies should give pause to consider if progress can yet be measured well, and if so, how field-based investigation involving DBR participants to gather evidence of any genuine impact on practice resulting from DBR projects can be accomplished.