Advancing Explainable AI in Medicine: Novel Frameworks for Robustness, Evaluation, and Human-Centered Interpretability
by Jacqueline Michelle Metsch née Jacqueline Michelle Beinecke
Date of Examination:2025-04-29
Date of issue:2025-06-23
Advisor:Prof. Dr. Anne-Christin Hauschild
Referee:Prof. Dr. Anne-Christin Hauschild
Referee:Prof. Dr. Alexander Ecker
Persistent Address:
http://dx.doi.org/10.53846/goediss-11333
Files in this item
Name:Dissertation_Metsch.pdf
Size:10.3Mb
Format:PDF
Description:Dissertation
The following license files are associated with this item:
Abstract
English
The advancement of artificial intelligence (AI) has led to its increasing usage in various fields, including medicine, where AI models, particularly deep neural networks (DNNs), have shown great performance. However, despite their predictive power, integrating AI into clinical practice remains challenging due to transparency, ethical, and accountability concerns. DNNs are often referred to as ªblack-boxº models because human users can not understand their decision-making processes. This lack of transparency results in a lack of trust, as clinicians are hesitant to rely on AI decisions without a clear understanding of how these are made. Furthermore, regulatory frameworks such as the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), and the European Union AI Act mandate transparency and accountability in AI decision-making, emphasising the need for explainability in AI systems. Explainable AI (XAI) has emerged as a research area aimed at addressing these challenges by providing insights into the decision-making of AI models, thereby increasing interpretability and trust. However, despite the progress in XAI research, several issues remain that prevent its integration into medicine. Some of these issues include (1) a lack of robustness in XAI methods, where explanations can vary significantly due to small input perturbations, hyperparameter adjustments, or differences in explanation methods (2) inconsistencies in the evaluation of XAI methods, as there is no widely accepted standard for evaluating explanations, leading to difficulties in comparing different XAI methods across various data types and AI models and (3) the need for human-centered approaches that facilitate interaction between AI developers and domain experts to ensure that explanations are not only technically valid but also meaningful. To address these issues, this thesis presents four main contributions. First, a novel sample-wise rank normalisation technique is introduced to improve the robustness of post-hoc XAI methods. This normalisation method reduces sensitivity to small fluctuations in raw relevance attributions while preserving the relative importance of features. By standardising XAI explanations, this method allows for more consistent comparisons between different XAI methods across different data types and AI models. Second, a comprehensive evaluation framework, BenchXAI, is developed to systematically benchmark XAI methods across multiple biomedical data types, including clinical data, medical imaging and signal data, and biomolecular data. Unlike previous evaluation approaches that primarily rely on qualitative heatmaps or a limited set of evaluation metrics, BenchXAI incorporates a Monte Carlo Cross Validation (MCCV) approach to ensure that XAI evaluations are made across multiple randomised training and test splits. This provides a more robust assessment of XAI methods. The benchmarking study conducted using BenchXAI demonstrates that while certain XAI methods, such as DeepLift, DeepLiftShap, GradientShap, and Integrated Gradients, perform consistently well across various data types, others exhibit high variability, underscoring the need for standardisation in XAI evaluation. Third, this thesis proposes a novel ensemble majority vote approach to address the robustness issue across different XAI methods. Individual XAI methods often introduce biases based on their underlying explanation mechanisms, leading to inconsistent or contradictory explanations. By aggregating explanations from multiple XAI methods using an ensemble approach, this method reduces individual biases and enhances the stability of explanations. This ensemble approach is applied to a large-scale electrocardiogram (ECG) classification task, where it identifies clinically relevant features associated with different cardiovascular diseases. Furthermore, the ensemble approach highlights significant differences in explanations for true and false positive classifications, showing its potential for AI model debugging and improving overall classification performance. Finally, to bridge the gap between AI researchers and domain experts, this thesis introduces CLARUS, an interactive XAI platform. CLARUS is tailored for graph neural networks (GNNs) and allows domain experts to explore patient-specific protein-protein interaction (PPI) networks, visualise XAI explanations, and interactively modify graph structures to test counterfactual scenarios. This interactive approach enables domain experts to improve their understanding of how AI models make decisions. By allowing users to perform manual counterfactual analysis Ð such as investigating the effects of removing or modifying specific gene interactions Ð CLARUS allows domain experts to validate AI predictions. Unlike existing XAI platforms that primarily focus on static visualisations, CLARUS allows for real-time retraining, making it a valuable tool for human-in-the-loop AI evaluation. Despite these advancements, challenges remain in further enhancing and expanding XAI methods. Future research should focus on improving normalisation techniques to preserve both robustness and magnitude in XAI explanations. Additionally, expanding the BenchXAI framework to support more advanced AI models, such as attention-based models and transformer networks, will be essential for better benchmarking studies. Furthermore, usability studies on CLARUS are also necessary to assess its effectiveness in real-world settings and optimise its design. In conclusion, this thesis advances the field of XAI by addressing key limitations in robustness, evaluation, and human-centeredness, ultimately contributing to the broader goal of increasing trust in AI systems. By providing more stable and interpretable explanations, developing standardised benchmarking frameworks, and fostering better collaboration between AI researchers and domain experts, these contributions help bridge the gap between AI research and clinical implementation.
Keywords: Artificial Intelligence; Explainable AI; XAI in Medicine
