Explainable AI in Healthcare: A Holistic View of Technical Approaches, Regulatory Frameworks, Reliable Clinical Implementation Insights and Limitations

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Opaque deep learning models in biomedical applications pose challenges related to bias, fairness, and regulatory compliance. This paper presents a narrative review of explainable artificial intelligence (XAI) in healthcare, specifically medical diagnosis, based on a targeted selection guided by a narrative review of recent literature, from 2019 to 2026, conducted on Scopus, Web of Science, and PubMed databases. Covered topics comprise model interpretability, including image-based models and multimodal models, with special focus on large language models (LLMs) as a hard-to-interpret system. The review situates XAI within key regulatory frameworks, including the European Union (EU) Artificial Intelligence (AI) Act, highlighting how explainability supports legal requirements for transparency, auditability, and accountability. It examines widely used XAI methods such as Shapley Additive Explanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME), attention visualization, neurosymbolic reasoning, and others. These techniques help identify bias, detect spurious correlations, and analyze hallucinations in LLM deployment. They present an effort to, within a feasible range, minimize these events and increase confidence in the predictions provided, without dispensing with clear and proper validation within the medical classification loop. While emphasizing the benefits of XAI, the review also addresses key limitations, including limited explanation fidelity, cognitive overload, and the risk of misleading interpretations. Overall, it provides a concise synthesis of how explainability intersects with ethics, regulation, and system design to support oversight and transparency in AI healthcare. Additionally, it provides suggestions for the incorporation of technical XAI methods within current legislation and guidelines to provide a robust framework for AI model deployment.