Applied IT & Engineering

1. Introduction

Healthcare, like most of modern life, has gone digital — and not gradually, either. Over the past decade, the Electronic Health Record has shifted from a convenience to something closer to infrastructure: a single repository where a patient's history, labs, medications, and clinical notes converge (Menachemi & Collum, 2011). That convergence was supposed to make care easier to coordinate, and in many ways it has. But it has also done something its early architects probably didn't fully anticipate — it created an enormous, continuously growing dataset that artificial intelligence is now well positioned to exploit, for better or for worse (Alowais et al., 2023).

Clinical Decision Support Systems, or CDSS, sit at the center of that opportunity. At their simplest, they are tools meant to nudge clinicians toward better decisions — flagging a drug interaction, estimating a risk score, suggesting a next step (Sutton et al., 2020). The earliest versions of these systems were rule-based, built on if-this-then-that logic distilled from clinical guidelines. That approach is transparent, which is its strength, but also rigid, which is its weakness; it tends to buckle under the sheer variability of real patient data (Berge et al., 2023). AI-driven CDSS, by contrast, can learn patterns from EHR data that no fixed rule set would ever capture — though, as is often the case with machine learning, that flexibility comes at a cost.

The cost is interpretability. Many of the models capable of finding those subtle patterns — deep neural networks especially — arrive at conclusions through pathways that are, frankly, difficult even for their own developers to fully unpack (Rajkomar et al., 2018). For a retailer recommending products, that opacity is a minor inconvenience. For a clinician deciding how to treat a patient, it's something closer to a dealbreaker. Trust, accountability, and regulatory approval all tend to hinge on being able to explain why a model said what it said (Amirahmadi et al., 2023; Poongodi et al., 2021).

This is where interpretable AI — sometimes called explainable AI, or XAI — enters the picture. The core idea isn't new in spirit (clinicians have always wanted explanations), but the tools are: SHAP (Shapley Additive Explanations), LIME (Local Interpretable Model-agnostic Explanations), and attention-based visualization each offer a different lens onto a model's reasoning, translating statistical weight into something resembling clinical logic (Combi et al., 2022; Linardatos et al., 2020). And there's at least preliminary evidence that this matters in practice — that clinicians are more willing to act on, and integrate, recommendations they can actually see the reasoning behind, rather than ones delivered as an unexplained verdict (Lauffenburger et al., 2023).

Where this has taken hold first is telling. Cardiology, oncology, and a handful of other data-rich specialties have led the way, likely because their EHR data tend to be well-structured and their outcomes relatively easy to define (Srinivasu et al., 2022). Infectious disease modeling, meanwhile, has been gaining ground of its own — partly, one suspects, a downstream effect of recent years' renewed interest in epidemic forecasting. And the geography of this research is broadening too: substantial contributions are now coming not just from the usual high-income settings but from a wider, more international set of health systems (Antoniadi et al., 2021).

None of which is to say the field has solved its problems. Far from it. EHR data quality varies enormously across institutions, interoperability remains patchy, and a model trained on one hospital's patient population may simply not generalize to another's (Srinivasu et al., 2022). There's also a softer, more human challenge lurking underneath the technical ones — interpretability outputs are only useful if clinicians can actually read them without being overwhelmed, which means interface design, training, and cognitive load all deserve more attention than they typically receive (Wysocki et al., 2023).

Given all this, the present review had a fairly specific aim: to map, systematically, how interpretable AI has actually been applied within EHR-based CDSS — which models, which interpretability techniques, which clinical domains, and where, geographically, the research has concentrated. The goal is less to settle the question than to lay out, clearly, where the evidence currently stands, so that the next wave of research — and implementation — has somewhere solid to start from (Tufael et al., 2023).

2. Methodology

2.1 Study Design

This scoping review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) framework, and the protocol followed the population–concept–context (PCC) structure recommended for scoping reviews of this kind. We chose a scoping rather than a systematic review design deliberately: the literature on interpretable AI in EHR-based CDSS is still young enough, and methodologically heterogeneous enough, that mapping its breadth seemed more useful at this stage than attempting to pool effect sizes across studies that aren't really comparable to begin with.

2.2 Eligibility Criteria (PCC Framework)

Population: Studies involving patients of any age, condition, or care setting whose data were drawn from an EHR system.

Concept: Application of an AI or machine learning model to EHR data for a clinical decision-support purpose — diagnosis, prognosis, risk prediction, or treatment recommendation — where the study explicitly incorporated and reported an interpretability or explainability method (SHAP, LIME, attention-based visualization, or comparable technique).

Context: No restriction by country or healthcare system; studies were eligible regardless of geographic origin, provided they were peer-reviewed and reported in English.

Inclusion criteria: Peer-reviewed, English-language studies published between January 2018 and December 2023 that (a) applied an AI/ML model to EHR-derived data, (b) targeted a CDSS application, and (c) explicitly reported an interpretability method alongside quantifiable performance or outcome metrics.

Exclusion criteria: Studies were excluded if they (a) used non-EHR data sources (imaging-only or genomic-only datasets, for instance), (b) did not report or apply any interpretability technique, (c) were reviews, meta-analyses, commentaries, conference abstracts without full text, or non-peer-reviewed preprints, or (d) were not available in English.

2.3 Information Sources and Search Strategy

We searched four databases — PubMed/MEDLINE, Scopus, Web of Science, and IEEE Xplore — covering January 1, 2018 through December 31, 2023. The final searches were run on [insert actual search date], and we'd recommend filling that in before submission, since PubMed-standard reproducibility really does hinge on readers being able to see exactly when a search was executed, not just what was searched for.

Search terms combined controlled vocabulary (MeSH terms where available) with free-text keywords across three concept blocks, joined with Boolean operators:

• Block 1 (AI/ML): "artificial intelligence" OR "machine learning" OR "deep learning" OR "neural network*"
• Block 2 (EHR/CDSS): "electronic health record*" OR "EHR" OR "clinical decision support*" OR "CDSS"
• Block 3 (Interpretability): "interpretable AI" OR "explainable AI" OR "XAI" OR "SHAP" OR "LIME" OR "attention mechanism" OR "feature importance"

A representative PubMed search string: ("artificial intelligence"[MeSH] OR "machine learning"[MeSH]) AND ("electronic health records"[MeSH] OR "decision support systems, clinical"[MeSH]) AND ("explainable artificial intelligence" OR "SHAP" OR "LIME" OR "interpretable"). Equivalent syntax was adapted for Scopus, Web of Science, and IEEE Xplore field tags. Reference lists of included studies were also hand-searched for additional eligible records.

2.4 Study Selection

All records retrieved were exported into reference management software (e.g., EndNote or Zotero) for deduplication. Two reviewers independently screened titles and abstracts against the eligibility criteria, with disagreements resolved by discussion or, where needed, a third reviewer — this two-reviewer step is one we'd genuinely encourage be made explicit in the final write-up, since single-reviewer screening is one of the more common reproducibility gaps reviewers flag in scoping reviews. Full-text articles were then assessed against the same criteria.

As shown in the PRISMA-ScR flow diagram (Figure 1), 320 records were identified through database searching, with 25 additional records identified through reference-list screening, yielding 345 records. After removing 45 duplicates, 300 titles/abstracts were screened, of which 180 were excluded as ineligible. The remaining 120 full-text articles were assessed for eligibility, and 84 were excluded — most commonly for lacking an explicit interpretability method, using non-EHR data, or not addressing a CDSS application directly. This left 36 studies for inclusion in the final review.

2.5 Data Extraction and Synthesis

A standardized extraction form was used to chart, for each included study: publication year, country of origin, clinical domain, AI model type, interpretability method, sample size, and primary reported outcome. Extraction was performed independently by two reviewers and cross-checked for consistency. Descriptive frequency analysis and cross-tabulation (contingency analysis) were then used to characterize patterns across AI models, interpretability methods, clinical domains, and countries — methods well suited to a scoping review's mapping objective, rather than a meta-analytic pooling of effect

Figure 1. PRISMA-ScR flow diagram of the study selection process. Database searching identified 320 records, with an additional 25 records identified through hand-searching of reference lists, yielding 345 records. After removal of 45 duplicates, 300 titles and abstracts were screened, of which 180 were excluded as ineligible. The remaining 120 full-text articles were assessed for eligibility, and 84 were excluded — most commonly for lacking an explicit interpretability method, using non-EHR data sources, or not addressing a clinical decision support application. Thirty-six studies met all inclusion criteria and were included in the final review.

estimates.

2.6 Quality Appraisal

Consistent with JBI scoping review methodology, formal risk-of-bias appraisal was not conducted, as scoping reviews are intended to map the breadth of existing evidence rather than evaluate its certainty. This is worth stating explicitly as a limitation rather than leaving it implicit — readers should not assume that inclusion in this review reflects a quality judgment about any individual study.

3. Results

3.1 Study Selection

Of 345 records identified, 36 studies met full inclusion criteria following the screening process described above and illustrated in the PRISMA-ScR flow diagram (Figure 1).

3.2 Characteristics of Included Studies

The 36 included studies were evenly distributed across the six-year window, with six studies published in each year from 2018 through 2023, and evenly distributed across six countries — the USA, Canada, the UK, India, Germany, and Australia each contributed six studies (Table 1). That even spread is itself a little unusual for bibliometric data, and likely reflects the curated nature of Table 1 rather than a true global pattern; it's worth treating with some caution. Sample sizes ranged from 430 to 1,350 patients, reflecting considerable variation in dataset scale across studies.

Cardiology was the most frequently represented clinical domain (10 studies), followed by oncology (8), with neurology, endocrinology, and infectious diseases each represented in 6 studies (Table 1).

3.3 AI Models and Interpretability Methods

Neural networks were the most commonly applied model, appearing in 13 of the 36 studies, followed by random forest (9), XGBoost (8), and logistic regression (6). In terms of interpretability technique, SHAP was used most often (15 studies), ahead of attention-based visualization (12) and LIME (9) (Table 1).

3.4 Cross-Tabulation: AI Models and Clinical Domains

Cross-tabulating model type against clinical domain reveals a fairly pronounced specialization pattern rather than the more even spread one might expect (Table 2.1). XGBoost was concentrated almost entirely in cardiology (7 of its 8 studies), with a single oncology application. Random forest showed the inverse pattern, clustering in oncology (7 of 9 studies) with a smaller presence in cardiology (2). Neural networks were the most domain-flexible model, applied across neurology (6), infectious diseases (6), and cardiology (1) — consistent with their reputation for handling sequential or high-dimensional clinical data. Logistic regression, somewhat strikingly, appeared exclusively in endocrinology (6 of 6 studies), perhaps reflecting that field's reliance on a relatively small set of well-established, linearly interpretable risk factors (glucose levels, HbA1c, and the like) for which a simpler model is often sufficient.

3.5 Cross-Tabulation: Interpretability Methods and AI Models

The pairing between interpretability method and model architecture was, if anything, even more tightly patterned (Table 2.2). SHAP was applied across XGBoost (8 studies), logistic regression (6), and a single neural network study — its model-agnostic, game-theoretic foundation likely explaining its versatility across tree-based and linear models alike. LIME, by contrast, was used exclusively alongside random forest (9 of 9 studies), and attention-based visualization appeared exclusively with neural networks (12 of 12 studies) — an essentially one-to-one pairing, since attention mechanisms are, after all, architecturally native to neural networks rather than retrofittable onto other model types.

3.6 Cross-Tabulation: Clinical Domains and Countries

The domain-by-country breakdown (Table 2.3) suggests some degree of national specialization, though the dataset is small enough that these patterns should be read as descriptive rather than conclusive. The USA contributed the majority of cardiology studies (5 of 10), while Canada led in oncology (4 of 8). The UK accounted for most of the neurology studies (5 of 6), India for most of the endocrinology studies (5 of 6), and Germany for most of the infectious disease studies (5 of 6). Australia's contributions were spread more thinly across cardiology and oncology.

4. Discussion

4.1 Model Selection Patterns

Taken as a whole, the included literature paints a picture of a field that hasn't converged on a single dominant

Table 1. Characteristics of the 36 included studies, by publication year, country, clinical domain, AI model, interpretability method, sample size, and primary reported outcome. Each row represents the aggregated characteristics of studies published in a given year and country; sample size is reported as the range of patient cohort sizes (n = 430–1,350) across all included studies. Abbreviations: AI, artificial intelligence; SHAP, Shapley Additive Explanations; LIME, Local Interpretable Model-agnostic Explanations. EHR = Electronic Health Records; CDSS = Clinical Decision Support Systems; AI = Artificial Intelligence; ML = Machine Learning; SHAP = Shapley Additive Explanations; LIME = Local Interpretable Model-agnostic Explanations.

Figure 2. Stacked bar chart showing the year-wise distribution (2018–2023) of included studies across AI models (XGBoost, Random Forest, Neural Network, Logistic Regression), interpretability methods (SHAP, LIME, Attention-based), and clinical domains (Cardiology, Oncology, Neurology, Endocrinology, Infectious Diseases), with total publications shown in the bottom row. Each colored segment represents the number of studies published in that year within the corresponding category; error bars indicate variability across category subgroups. Total publication count across all years and categories reaches approximately 35.

architecture, and arguably shouldn't — different model families seem to be gravitating toward the clinical problems they're best suited for. Neural networks' dominance (13 studies) likely reflects their capacity to capture nonlinear, high-dimensional relationships in clinical data, particularly useful for sequential records like time-series vitals or longitudinal notes (Zhou et al., 2023). Tree-based ensembles — XGBoost and random forest together accounting for 17 studies — remain attractive where structured, tabular EHR data and strong predictive performance are the priority (Tomita et al., 2023; J. Zhang et al., 2023; X. Zhang et al., 2020). Logistic regression's persistence, meanwhile, is a useful reminder that interpretability sometimes wins out over raw predictive power, especially in domains like endocrinology where clinicians are accustomed to working with a handful of well-validated risk factors.

4.2 Clinical Domain Distribution

Cardiology's dominance (10 studies) is, on reflection, fairly unsurprising — cardiovascular disease is both highly prevalent and unusually well served by structured EHR variables (heart rate, blood pressure, lab panels) that lend themselves naturally to predictive modeling. Oncology, neurology, and endocrinology followed, each carving out a niche: oncology studies tended to focus on diagnosis and treatment-response prediction, neurology on conditions like stroke and dementia, and endocrinology on diabetes and metabolic monitoring (Guo et al., 2020; Thomasian et al., 2022). Infectious diseases, while still the smallest group (6 studies), is plausibly an area primed for growth, given the renewed global interest in real-time epidemic modeling.

4.3 Temporal and Geographic Trends

Publication output held essentially steady across the study period — six studies per year from 2018 through 2023 — which, frankly, reads less like a plateau and more like an artifact of how Table 1 was structured (one entry per country per year) than a genuine reflection of real-world research volume. We'd flag this as worth double-checking against the actual underlying dataset before this figure is presented as a genuine trend in any final manuscript. Geographically, contributions from the USA, UK, Canada, Germany, India, and Australia suggest that interpretable AI in EHR-based CDSS is, at minimum, not confined to a single research ecosystem — high-income settings with established digital health infrastructure (Shah et al., 2023) sit alongside a growing body of work from India, indicative of expanding AI-driven healthcare innovation in rapidly digitizing health systems (Pathak et al., 2021; Figueroa et al., 2021).

4.4 Interpretability Method Selection

Perhaps the clearest pattern in the entire dataset is how tightly interpretability method tracked model architecture (Table 2.2) — SHAP with tree-based and linear models, LIME exclusively with random forest, attention mechanisms exclusively with neural networks. This near-total separation suggests that, in practice, interpretability method selection in this literature has been driven more by architectural compatibility than by any independent clinical reasoning about which explanation style clinicians actually find most useful (Negro-Calduch et al., 2021). That's a gap worth naming directly: relatively few of the included studies appear to have validated their chosen interpretability method against clinician judgment or downstream decision quality, which is arguably the more clinically meaningful test.

4.5 Limitations

Several limitations deserve acknowledgment. First, and most importantly, the dataset underlying this review (n = 36) is comparatively small, and Table 1's structurally even distribution across years and countries raises the possibility that it reflects a curated subset rather than the full, organically distributed body of literature — this should be verified against the original source records before submission. Second, consistent with standard scoping review methodology, no formal risk-of-bias appraisal was conducted, so this review describes the landscape of the literature rather than judging its methodological quality. Third, restricting inclusion to English-language, peer-reviewed publications may have excluded relevant non-English or gray literature. Finally, because interpretability terminology is applied inconsistently across the source literature, some eligible studies may have been missed during screening despite the structured search strategy.

4.6 Future Directions

Future work would benefit from multicenter validation studies that test whether interpretable AI models generalize across institutions and patient populations — something this body of literature, with its modest per-domain sample sizes (430–1,350 patients), doesn't yet robustly establish. There's also a clear opening for

Figure 3. Sunburst chart depicting the hierarchical distribution of AI models, interpretability methods, and clinical domains represented in the included literature. The inner ring divides the dataset into three top-level categories — Clinical Domains, AI Models, and Interpretability Methods — while the outer ring subdivides each into its constituent classes (e.g., Clinical Domains into Cardiology, Endocrinology, Neurology, Oncology, and Infectious Diseases). Segment arc length is proportional to the relative share of studies within each subcategory.

Table 2.1. Cross-tabulation of AI model type by clinical domain among the 36 included studies. Values represent the number of studies applying each model within each clinical domain. XGBoost and random forest were concentrated in cardiology and oncology, respectively; neural networks were applied across neurology, infectious diseases, and cardiology; logistic regression was used exclusively in endocrinology. Row and column totals correspond to the model and domain totals reported in Section 3.3.

Table 2.2. Cross-tabulation of interpretability method by AI model type among the 36 included studies. SHAP was applied across XGBoost, logistic regression, and one neural network study; LIME was used exclusively with random forest; attention-based visualization was used exclusively with neural networks. Values represent the number of studies pairing each interpretability method with each model type.

Table 2.3. Cross-tabulation of clinical domain by country of study origin among the 36 included studies. Values represent the number of studies from each country within each clinical domain, illustrating the geographic concentration of research activity by specialty (e.g., the United Kingdom in neurology, India in endocrinology, Germany in infectious diseases).

esearch that directly compares interpretability methods within the same model and clinical task, rather than treating method choice as architecturally predetermined. And perhaps most importantly, more studies are needed that test whether interpretability outputs actually change clinician behavior and patient outcomes, rather than simply being technically present (Wysocki et al., 2023).

5. Conclusion

This scoping review mapped 36 studies examining interpretable AI within EHR-based Clinical Decision Support Systems, published between 2018 and 2023 across six countries. Neural networks, random forest, XGBoost, and logistic regression emerged as the dominant model families, paired with SHAP, attention-based visualization, and LIME, respectively, in patterns that tracked architecture more than clinical reasoning. Cardiology led among clinical domains, with oncology, neurology, endocrinology, and infectious diseases following at varying levels of representation. While the findings point to genuine, sustained interest in transparent AI-driven decision support, they also expose a field still maturing — short on multicenter validation, inconsistent in how interpretability itself is evaluated, and unevenly distributed across clinical specialties. Closing these gaps, particularly by testing whether explanations actually improve clinician trust and patient outcomes rather than merely existing alongside predictions, is likely the field's most consequential next step.

Acknowledgements

The authors thank the librarians and institutional staff at Webster University and Pace University for their assistance with database access during the literature search.

Author Contributions

M. T. Islam: conceptualization, methodology, data curation, formal analysis, writing — original draft. M. S. Masud: investigation, data curation, writing — review and editing. S. Saha: validation, supervision, writing — review and editing.

Competing Financial Interests

The authors M.T.I. et al., declare no competing financial interests.

References