Erprobung der Vorhersage-basierten Evaluationsmethode für das Instrument der Lernerfolgsevaluation
Proving a prediction-based evaluation method to estimate student learning outcome
by Binia-Laureen Grebener née Entgelmeier
Date of Examination:2022-05-17
Date of issue:2022-04-21
Advisor:Prof. Dr. Tobias Raupach
Referee:Prof. Dr. Tobias Raupach
Referee:PD Dr. Dr. Philipp Kanzow
Referee:Prof. Dr. Ralf Dressel
Files in this item
This file will be freely accessible after 25.05.2022.
EnglishBackground: Evaluations play an important role for teaching curricula. Therefore, student evaluations of teaching are often used to draw conclusions about teaching quality. In undergraduate medical education it is important to evaluate the learning outcome but low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome. Therefore, it was compared to the evaluation tool of comparative student self-assessments that has been developed at Goettingen University Medical Centre and is used to evaluate courses in the clinical phase of undergraduate medical education so far. Methods: Before and after a cardiorespiratory module, 128 fourth-year medical students at Goettingen University Medical Centre provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Data were collected at these two defined points in time and pre-post performance gain was compared across all three modalities. Furthermore, the lower limits of response rate required to obtain reliable learning outcome data (number needed to evaluate) based on student predictions and self-assessments were calculated. Results: Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Hence, this study identified a significant correlation between performance gain derived from student self-assessments and performance gain derived from predictions (r = 0,952; p < 0,001) meaning a high agreement between both measuring methods. Also, the number needed to evaluate based on student predictions and self-assessments were comparable with no significant difference between the two. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. The performance gains derived from self-assessments as well as derived from predictions showed both a significant correlation to the performance gain derived from the objective examinations (r= 0,709; p < 0,001 and r = 0,704; p < 0,001). For learning objectives with a high objective learning outcome, less student ratings were needed to obtain stable results from the two evaluation methods compared to learning objectives with low objective learning outcome. Conclusions: Student self-assessments and predictions are equally valid sources of learning outcome measures. Furthermore, low response rates are sufficient to produce stable results for both methods.
Keywords: evaluation; learning outcome; comparative self-assessment; predictions