Schlagworte
Mediendidaktik
Systematik
Bildungstechnologie
Zitationsvorschlag
Lizenz
Copyright (c) 2023 Katja Buntins, Svenja Bedenlier, Victoria Marín, Marion Händel, Melissa Bond
Dieses Werk steht unter der Lizenz Creative Commons Namensnennung 4.0 International.
Abstract
Methoden der Evidenzsynthese werden in den Sozialwissenschaften immer beliebter, auch im Bereich der Bildungstechnologie, wo die Menge der durchgeführten Sekundärforschung exponentiell zugenommen hat. Während Übersichten einen wertvollen Einblick in den Stand des Feldes geben können, wurden Fragen zur methodischen Strenge und Transparenz aufgeworfen. In dieser tertiären Übersichtsarbeit wird daher untersucht, wie transparent und reproduzierbar die Berichterstattung über die Methoden der Evidenzsynthese im Bereich der Bildungstechnologie in den verschiedenen Arten von Übersichtsarbeiten ist. Zu diesem Zweck wurde eine kritische Analyse einer Stichprobe von 446 Evidenzsynthesen in drei Sprachen (Englisch, Deutsch und Spanisch) durchgeführt. Systematic Review, Meta-Analyse und Literaturreview als die am häufigsten verwendeten Review-Typen wurden für die weitere Analyse. In der Stichprobe wurden Indikatoren für die Replizierbarkeit in kritischen Phasen der Überprüfung identifiziert und nach Überprüfungstyp (Forschungsfrage, Suchstrategie, Datenextraktion und Synthese) analysiert. Die Ergebnisse zeigen, dass die methodische Transparenz bei der Datenextraktion und -synthese erheblich verbessert werden kann, wobei bestimmte Arten von Übersichten bei den Indikatoren schlechter abschneiden als andere. Es werden Empfehlungen zur Verbesserung der methodischen Transparenz und Strenge der Evidenzsynthese im Bereich der Bildungstechnologie gegeben.
Literatur
Abrami, Philip C., Peter A. Cohen, and Sylvia d’Apollonia. 1988. «Implementation Problems in Meta-Analysis». Review of Educational Research 58 (2): 151–79. https://doi.org/10.3102/00346543058002151.
Alexander, Patricia A. 2020. «Methodological Guidance Paper: The Art and Science of Quality Systematic Reviews». Review of Educational Research 90 (1): 6–23. https://doi.org/10.3102/0034654319854352.
Atlam, Hany F., Muhammad Ajmal Azad, Madini O. Alassafi, Abdulrahman A. Alshdadi, and Ahmed Alenezi. 2020. «Risk-Based Access Control Model: A Systematic Literature Review». Future Internet 12 (6). https://doi.org/10.3390/fi12060103.
Baumeister, Roy F., and Mark R. Leary. 1997. «Writing Narrative Literature Reviews». Review of General Psychology 1 (3): 311–20. https://doi.org/10.1037/1089-2680.1.3.311.
Belur, Jyoti, Lisa Tompson, Amy Thornton, and Miranda Simon. 2018. «Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making». Sociological Methods & Research 50 (2): 837–65. https://doi.org/10.1177/0049124118799372.
Bond, Melissa. 2020. «Schools and Emergency Remote Education During the COVID-19 Pandemic: A Living Rapid Systematic Review». Asian Journal of Distance Education 15 (2): 191–247. https://doi.org/10.5281/zenodo.4425683.
Bond, Melissa, Katja Buntins, Svenja Bedenlier, Olaf Zawacki-Richter, and Michael Kerres. 2020. «Mapping Research in Student Engagement and Educational Technology in Higher Education: A Systematic Evidence Map». International Journal of Educational Technology in Higher Education 17 (1): 1–30. https://doi.org/10.1186/s41239-019-0176-8.
Borenstein, Michael, ed. 2009. Introduction to Meta-Analysis: John Wiley & Sons, Ltd.
Borenstein, Michael, Larry V. Hedges, Julian P. T. Higgins, and Hannah R. Rothstein. 2010. «A Basic Introduction to Fixed-Effect and Random-Effects Models for Meta-Analysis». Research Synthesis Methods 1 (2): 97–111. https://doi.org/10.1002/jrsm.12.
Borrego, Maura, Margaret J. Foster, and Jeffrey E. Froyd. 2014. «Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields». Journal of Engineering Education. 103 (1): 45–76. https://doi.org/10.1002/jee.20038.
Bramer, Wichor M., Dean Giustini, and Bianca Kramer. 2016. «Comparing the Coverage, Recall, and Precision of Searches for 120 Systematic Reviews in Embase, MEDLINE, and Google Scholar: A Prospective Study». Systematic Reviews 5 (1): 39. https://doi.org/10.1186/s13643-016-0215-7.
Bramer, Wichor M., Dean Giustini, Bianca Kramer, and P. F. Anderson. 2013. «The Comparative Recall of Google Scholar Versus PubMed in Identical Searches for Biomedical Systematic Reviews: A Review of Searches Used in Systematic Reviews». Systematic Reviews 2 (1): 115. https://doi.org/10.1186/2046-4053-2-115.
Braun, Virginia, and Victoria Clarke. 2006. «Using Thematic Analysis in Psychology». Qualitative Research in Psychology 3 (2): 77–101. https://doi.org/10.1191/1478088706qp063oa.
Bulfin, Scott, Michael Henderson, Nicola F. Johnson, and Neil Selwyn. 2014. «Methodological Capacity Within the Field of «Educational Technology» Research: An Initial Investigation». British Journal of Educational Technology 45 (3): 403–14. https://doi.org/10.1111/bjet.12145.
Buntins, Katja, Michael Kerres, and Anna Heinemann. 2022. «A Scoping Review of Research Instruments for Measuring Student Engagement: In Need for Convergence». International Journal of Educational Research Open 2. https://doi.org/10.1016/j.ijedro.2021.100099.
Buntins, Katja, Miriam Mulders, and Nadine Schröder. 2023. «Cognitive Biases in Screening Processes – Search Strategies in Educational Technology Research a Systematic Review on Learning with Virtual Reality». MedienPädagogik 54: 103–124. https://doi.org/10.21240/mpaed/54/2023.11.19.X.
Campbell, Allison, Brian Taylor, Jessica Bates, and Una O’Connor-Bones. 2018. «Developing and Applying a Protocol for a Systematic Review in the Social Sciences». New Review of Academic Librarianship 24 (1): 1–22. https://doi.org/10.1080/13614533.2017.1281827.
CASP. 2022. «CASP Checklists. Critical Appraisal Skills Programme». https://casp-uk.net/casp-tools-checklists/.
Cooper, Chris, Rebecca Lovell, Kerryn Husk, Andrew Booth, and Ruth Garside. 2018a. «Supplementary Search Methods Were More Effective and Offered Better Value Than Bibliographic Database Searching: A Case Study from Public Health and Environmental Enhancement». Research Synthesis Methods 9 (2): 195–223. https://doi.org/10.1002/jrsm.1286.
Cooper, Chris, Joanna Varley-Campbell, Andrew Booth, Nicky Britten, and Ruth Garside. 2018b. «Systematic Review Identifies Six Metrics and One Method for Assessing Literature Search Effectiveness but No Consensus on Appropriate Use». Journal of Clinical Epidemiology 99:53–63. https://doi.org/10.1016/j.jclinepi.2018.02.025.
Cooper, Harris, and L. V. Hedges, eds. 1994. Handbook of Research Synthesis: Russell Sage Foundation.
Crompton, Helen, Mildred V. Jones, and Diane Burke. 2022. «Affordances and Challenges of Artificial Intelligence in K-12 Education: A Systematic Review». Journal of Research on Technology in Education, 1–21. https://doi.org/10.1080/15391523.2022.2121344.
Durlak, J. A., and M. W. Lipsey. 1991. «A Practitioner’s Guide to Meta-Analysis». American Journal of Community Psychology 19 (3): 291–332. https://doi.org/10.1007/BF00938026.
Eysenbach, G., J. Tuische, and T. L. Diepgen. 2001. «Evaluation of the Usefulness of Internet Searches to Identify Unpublished Clinical Trials for Systematic Reviews». Medical Informatics and the Internet in Medicine 26 (3): 203–18. https://doi.org/10.1080/14639230110075459.
Glass, Gene V., Barry McGaw, and Mary Lee Smith. 1981. «Meta-analysis in social research». SAGE Publications.
Glass, Gene V., and M. L. Smith. 1979. «Meta-Analysis of Research on Class Size and Achievement». Educational Evaluation and Policy Analysis 1 (1): 2–16. https://doi.org/10.3102/01623737001001002.
Gough, David, Phil Davies, Gro Jamtvedt, Etienne Langlois, Julia Littell, Tamara Lotfi, Edoardo Masset et al. 2020. «Evidence Synthesis International (ESI): Position Statement». Systematic Reviews 9 (1): 155. https://doi.org/10.1186/s13643-020-01415-5.
Hammersley, Martyn. 2001. «On ‘Systematic’ Reviews of Research Literatures: A ‘Narrative’ Response to Evans & Benefield». British Educational Research Journal 27 (5): 543–54. https://doi.org/10.1080/01411920120095726.
Hartling, Lisa, Andrea Milne, Michele P. Hamm, Ben Vandermeer, Mohammed Ansari, Alexander Tsertsvadze, and Donna M. Dryden. 2013. «Testing the Newcastle Ottawa Scale Showed Low Reliability Between Individual Reviewers». Journal of Clinical Epidemiology 66 (9): 982–93. https://doi.org/10.1016/j.jclinepi.2013.03.003.
Hew, Khe Foon, Min Lan, Ying Tang, Chengyuan Jia, and Chung Kwan Lo. 2019. «Where Is the ‘Theory’ Within the Field of Educational Technology Research?» British Journal of Educational Technology 50 (3): 956–71. https://doi.org/10.1111/bjet.12770.
Higgins, Julian P. T., Douglas G. Altman, Peter C. Gøtzsche, Peter Jüni, David Moher, Andrew D. Oxman, Jelena Savović, Kenneth F. Schulz, Laura Weeks, and Jonathan A. C. Sterne. 2011. «The Cochrane Collaboration Tool for Assessing Risk of Bias in Randomised Trials». BMJ 343. https://doi.org/10.1136/bmj.d5928.
Higgins, Julian P. T., J. Thomas, J. Chandler, M. Cumpston, T. Li, Matthew J. Page, and Vivian A. Welch. 2022. Cochrane Handbook for Systematic Reviews of Interventions, Version 6.3: Cochrane.
Ioannidis, John P. A., Marcus R. Munafò, Paolo Fusar-Poli, Brian A. Nosek, and Sean P. David. 2014. «Publication and Other Reporting Biases in Cognitive Sciences: Detection, Prevalence, and Prevention». Trends in Cognitive Sciences 18 (5): 235–41. https://doi.org/10.1016/j.tics.2014.02.010.
Katrak, Persis, Andrea E. Bialocerkowski, Nicola Massy-Westropp, V. S. Kumar, and Karen A. Grimmer. 2004. «A Systematic Review of the Content of Critical Appraisal Tools». BMC Medical Research Methodology 4 (1): 1–11. https://doi.org/10.1186/1471-2288-4-22.
Kimmons, Royce, and Joshua M. Rosenberg. 2022. «Trends and Topics in Educational Technology, 2022 Edition». TechTrends 66 (2): 134–40. https://doi.org/10.1007/s11528-022-00713-0.
Kolbe, Richard H., and Melissa S. Burnett. 1991. «Content-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity». Journal of Consumer Research 18 (2): 243–50. https://doi.org/10.1086/209256.
Krippendorff, Klaus. 2018. Content Analysis: An Introduction to Its Methodology: SAGE Publications.
Lai, Jennifer W. M., and Matt Bower. 2020. «Evaluation of Technology Use in Education: Findings from a Critical Analysis of Systematic Literature Reviews». Journal of Computer Assisted Learning 36 (3): 241–59. https://doi.org/10.1111/jcal.12412.
Landis, J. Richard, and Gary G. Koch. 1977. «The Measurement of Observer Agreement for Categorical Data». Biometrics 33 (1): 159–74. https://doi.org/10.2307/2529310.
Lombard, Matthew, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. 2002. «Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability». Human Communication Research 28 (4): 587–604. https://doi.org/10.1111/j.1468-2958.2002.tb00826.x.
Marín, Victoria I., Katja Buntins, Svenja Bedenlier, and Melissa Bond. 2023. «Invisible Borders in Educational Technology Research? A Comparative Analysis». Educational Technology Research and Development: ETR & D 71: 1349–1370. https://doi.org/10.1007/s11423-023-10195-3.
Meline, Timothy. 2006. «Selecting Studies for Systemic Review: Inclusion and Exclusion Criteria». Contemporary Issues in Communication Science and Disorders 33 (Spring): 21–27. https://doi.org/10.1044/cicsd_33_S_21.
Methley, Abigail M., Stephen Campbell, Carolyn Chew-Graham, Rosalind McNally, and Sudeh Cheraghi-Sohi. 2014. «PICO, PICOS and SPIDER: A Comparison Study of Specificity and Sensitivity in Three Search Tools for Qualitative Systematic Reviews». BMC Health Services Research 14:579. https://doi.org/10.1186/s12913-014-0579-0.
Moher, David, Alessandro Liberati, Jennifer Tetzlaff, and Douglas G. Altman. 2009. «Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement». PLoS Medicine 6 (7): e1000097. https://doi.org/10.1371/journal.pmed.1000097.
Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer et al. 2021. «The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews». BMJ 372. https://doi.org/10.1136/bmj.n71.
Pigott, Therese D., and Joshua R. Polanin. 2020. «Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review». Review of Educational Research 90 (1): 24–46. https://doi.org/10.3102/0034654319877153.
Polanin, Joshua R., Brandy R. Maynard, and Nathaniel A. Dell. 2017. «Overviews in Education Research: A Systematic Review and Analysis». Review of Educational Research 87 (1): 172–203. https://doi.org/10.3102/0034654316631117.
Pollock, Alex, and Eivind Berge. 2018. «How to Do a Systematic Review». International Journal of Stroke 13 (2): 138–56. https://doi.org/10.1177/1747493017743796.
Shea, Beverley J., Jeremy M. Grimshaw, George A. Wells, Maarten Boers, Neil Andersson, Candyce Hamel, Ashley C. Porter, Peter Tugwell, David Moher, and Lex M. Bouter. 2007. «Development of AMSTAR: A Measurement Tool to Assess the Methodological Quality of Systematic Reviews». BMC Medical Research Methodology 7 (1): 10. https://doi.org/10.1186/1471-2288-7-10.
Singh, Jatinder. 2013. «Critical Appraisal Skills Programme». Journal of pharmacology and Pharmacotherapeutics 4 (1): 76. https://doi.org/10.4103/0976-500X.107697.
Sotola, Lukas K. 2022. «Garbage In, Garbage Out? Evaluating the Evidentiary Value of Published Meta-Analyses Using Z-Curve Analysis». Collabra: Psychology 8 (1): 32571. https://doi.org/10.1525/collabra.32571.
Stock, William A. 2008. «Systematic Coding for Research Synthesis». The Handbook of Research Synthesis 236: 125–38. Russell Sage Foundation.
Tamim, Rana M., Robert M. Bernard, Eugene Borokhovski, Philip C. Abrami, and Richard F. Schmid. 2011. «What Forty Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study». Review of Educational Research 81 (1): 4–28. https://doi.org/10.3102/0034654310393361.
Thomas, J., Graziosi, S., Brunton, J., Ghouze, Z., O›Driscoll, P., Bond, M., and Koryakina, A. 2023. «EPPI Reviewer: advanced software for systematic reviews, maps and evidence synthesis». EPPI Centre Software. UCL Social Research Institute. London. https://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4.
Viera, Anthony J., and Joanne M. Garrett. 2005. «Understanding Interobserver Agreement: The Kappa Statistic». Family Medicine 37 (5): 360–63. https://pubmed.ncbi.nlm.nih.gov/15883903/.
Zawacki-Richter, Olaf, Michael Kerres, Svenja Bedenlier, Melissa Bond, and Katja Buntins, eds. 2022. Systematic Reviews in Educational Research. Springer Fachmedien: Wiesbaden. https://doi.org/10.1007/978-3-658-27602-7.