A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems
This paper reviews how explainable AI methods are evaluated in clinical decision support systems, emphasizing the importance of making AI understandable for healthcare professionals.
Content & Liability Disclaimer
This article and its accompanying video are automated summaries derived from A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems by Alessandro Gambetti, Qiwei Han, Hong Shen, Cláudia Soares. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.
The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.
This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.
- 1 By aligning technical advancements in XAI with rigorous human-centered evaluations, we aim to steer research toward developing trustworthy and deployable CDSS solutions.
- 2 Artificial Intelligence (AI) is increasingly being integrated into Clinical Decision Support Systems (CDSS) to improve timely diagnosis, optimize treatment plans, and reduce healthcare costs [Sendak et al., 2020] .
- 3 Estimates suggest that AI adoption in healthcare could reduce U.S. healthcare expenditures by 5-10%, translating to 200-360 billion in annual savings while maintaining quality and access [Sahni et al., 2023] .
- 4 These trends indicate that the role of AI in medical decision-making will continue to expand [Lorenzini and others, 2023] .
Introduction
A key challenge accompanying this expansion is the need for trust and transparency in AI-driven medical decisions [Qin and others, 2024] . Trust in CDSS is critical because diagnostic errors can have life-threatening consequences [Zhang and Zhang, 2023] .
Regulatory frameworks such as the General Data Protection Regulation (GDPR) further reinforce the requirement for explainability in automated decisions.
Prior studies have largely emphasized algorithmic innovation over evaluation of clinical usability [Amann et al., 2022] .
These three levels of XAI evaluations imply a trade-off between fidelity and cost, where application-grounded evaluations provide the most realistic insights but are expensive, human-grounded evaluations offer a balance of generalizability and affordability but lack domain specificity, and proxy evaluations.
Research Question
By aligning technical advancements in XAI with rigorous human-centered evaluations, we aim to steer research toward developing trustworthy and deployable CDSS solutions.
Methodology
System performance is evaluated using machine learning metrics pertinent to the specific task. To this end, human-centered evaluations are conducted to assess the overall effectiveness and efficiency of the system in the specific task.
Study Design
This is typically achieved through common white-box models such as linear models, in which coefficients can be considered the marginal effects on the output in regression problems, and tree methods as decision trees, in which their hierarchical structure allows for direct observation and understanding of the decision-making process.
Importantly, note that intrinsic interpretable models are by definition model specific, e.g., interpreting linear regression coefficients is specific to that kind of model only.
How PDFdigest Helps You Understand Research
Instant Paper Analysis
Get structured summaries and key findings from dense PDFs in seconds.
Visual Explanations
Turn complex methods, figures, and results into clearer visual breakdowns.
AI-Powered Q&A
Ask focused questions and get answers grounded in the paper.
Results & Findings
Artificial Intelligence (AI) is increasingly being integrated into Clinical Decision Support Systems (CDSS) to improve timely diagnosis, optimize treatment plans, and reduce healthcare costs [Sendak et al., 2020] . Estimates suggest that AI adoption in healthcare could reduce U.S. healthcare expenditures by 5-10%, translating to 200-360 billion in annual savings while maintaining quality and access [Sahni et al., 2023] .
- Artificial Intelligence (AI) is increasingly being integrated into Clinical Decision Support Systems (CDSS) to improve timely diagnosis, optimize treatment plans, and reduce healthcare costs [Sendak et.
- Estimates suggest that AI adoption in healthcare could reduce U.S. healthcare expenditures by 5-10%, translating to 200-360 billion in annual savings while maintaining quality and access.
- The rapid growth of AI-driven CDSS has also attracted significant investment, with venture capital funding in AI for health exceeding $11 billion in 2024 [Goldsack and.
- These trends indicate that the role of AI in medical decision-making will continue to expand [Lorenzini and others, 2023] .
- Although deep learning and other black-box models achieve high accuracy, clinicians remain hesitant to rely on AI-generated recommendations [Longoni et al., 2019; Gaczek and others, 2023].
Dependency: Model Specific vs Model Agnostic Model Specific methods are limited to certain families of model architectures, and their use is coupled to the inner workings of those models.
Despite extensive research in XAI for healthcare [Loh et al., 2022] , a critical gap remains: limited systematic evidence exists on how XAI explanations influence real-world clinical adoption and decision-making [Ghassemi et al., 2021] .
Practical Applications
Finally, some deep learning models may have intrinsic interpretability components in their structure, e.g., attention weights in Transformers, highlighting the importance of different input features. In practical medical applications, socio-technical gaps may arise between the CDSS explainability factors provided by XAI techniques and end-users’ perceptions of their utility [Ackerman, 2000] .
Application-grounded evaluations involve testing real tasks with real experts and should be employed whenever possible because of their fidelity to real-world scenarios.
For example, in the medical domain, physicians may be evaluated on specific application tasks.
This absence highlights a nonnegligible research gap: future work may design evaluation protocols that may incorporate patient perspectives, ensuring that explanations are not only useful by clinicians but also easily communicable to patients (when possible).
Clinical Decision Support Systems and Explainable AI Taxonomy
This section provides an overview of CDSS and presents a taxonomy of XAI methods, detailing the criteria based on type, dependency, and scope. It emphasizes the need for human-centered evaluations to ensure the clinical relevance and interpretability of CDSS.
Source Paper Figures and Captions
Frequently Asked Questions
This raises a central question: How should XAI techniques be evaluated to ensure usability, trustworthiness, and effectiveness in clinical settings? By aligning technical advancements in XAI with rigorous human-centered evaluations, we aim to steer research toward developing trustworthy and deployable CDSS solutions.
System performance is evaluated using machine learning metrics pertinent to the specific task. To this end, human-centered evaluations are conducted to assess the overall effectiveness and efficiency of the system in the specific task.
Artificial Intelligence (AI) is increasingly being integrated into Clinical Decision Support Systems (CDSS) to improve timely diagnosis, optimize treatment plans, and reduce healthcare costs [Sendak et al., 2020] . Estimates suggest that AI adoption in healthcare could reduce U.S. healthcare expenditures by.
Explainable AI (XAI) has therefore emerged to enhance transparency, allowing medical professionals to understand model reasoning [Loh et al., 2022] . As the explainability evaluation of these models is endogenous, it is relatively straightforward to trace back how a prediction could be.
Dependency: Model Specific vs Model Agnostic Model Specific methods are limited to certain families of model architectures, and their use is coupled to the inner workings of those models. This absence highlights a nonnegligible research gap: future work may design evaluation protocols.
This paper reviews how explainable AI methods are evaluated in clinical decision support systems, emphasizing the importance of making AI understandable for healthcare professionals.