Assessment Criteria of Digital Health Interventions Focusing on Healthcare Providers: A Scoping Review
DOI:
https://doi.org/10.30953/thmt.v10.629Keywords:
Assessment criteria, digital health intervention, eHealth, healthcare providers, telemedicineAbstract
Objective: Digital Health Interventions (DHIs) are increasingly integrated into healthcare systems and widely adopted by healthcare providers (HPs). The lack of consistent evaluation standards makes it difficult to identify high-quality DHIs. This study aims to provide an overview of the criteria used to assess DHIs focusing on HPs.
Methods: A scoping review was conducted following the methodological framework proposed by Arksey and O’Malley and adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. Searches were conducted in PubMed, Web of Science, Scopus, and Cochrane databases for English-language studies published between February 2019 and February 2024, that evaluated DHIs focusing on HPs.
Results: The results revealed that there is no single consensus on how to assess DHIs, but rather multiple approaches have been proposed. From these studies, we derived a unified framework for evaluating DHIs focusing on HPs, encompassing assessment criteria grouped into ten key areas: data governance, usability, information management, health impact, patient-centered care, technical aspects, global context, functionality, sustainability, and design.
Discussion: The proposed multidimensional framework offers a structured basis for consistent evaluation and informed implementation of DHIs focusing on HPs.
Conclusions: The results of this scoping review lead to two main conclusions: first, there is a lack of a standardized framework for assessing DHIs; second, the development of a holistic framework is needed to enhance the effectiveness and sustainability of DHIs for HPs.
Downloads
References
The Fifty-eighth World Health Assembly, WHA 58/28 Document. Geneva: World Health Organization; 2005. Available from: https://apps.who.int/gb/ebwha/pdf_files/WHA58REC1/english/A58_2005_REC1-en.pdf (accessed 15 February 2024).
Classification of digital health interventions. Geneva: World Health Organization; 2018. (WHO/RHR/18.06).
Fifty-eighth World Health Assembly, eHealth Report A58/21. Geneva: World Health Organization; 2005. Available from: https://apps.who.int/gb/archive/pdf_files/WHA58/A58_21-en.pdf (accessed 15 February 2024).
Magnol M, Eleonore B, Claire R, et al. Use of eHealth by patients with rheumatoid arthritis: observational, cross-sectional, multicenter study. J Med Internet Res. 2021 Jan 29;23(1):e19998. doi:10.2196/19998. PMID: 33512320.
Knitza J, Simon D, Lambrecht A, et al. Mobile health usage, preferences, barriers, and eHealth literacy in rheumatology: patient survey study. JMIR Mhealth Uhealth. 2020 Aug 12;8(8):e19661. doi:10.2196/19661. PMID: 32678796.
Perrin Franck C, Babington-Ashaye A, Dietrich D, et al. iCHECK-DH: guidelines and checklist for the reporting on digital health implementations. J Med Internet Res. 2023 May 10;25:e46694. doi:10.2196/46694. PMID: 37163336.
Jacob C, Lindeque J, Müller R, et al. A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process. NPJ Digit Med. 2023 Dec 15;6(1):232. doi:10.1038/s41746-023-00982-w. PMID: 38102323.
The ongoing journey to commitment and transformation: digital health in the WHO European Region, 2023. Copenhagen: WHO Regional Office for Europe; 2023.
Hongsanun W, Insuk S. Quality assessment criteria for mobile health apps: a systematic review. Walailak J Sci Technol. 2020;17(8):745–759. doi:10.48048/wjst.2020.6482.
Jacob C, Lindeque J, Klein A, et al. Assessing the quality and impact of eHealth tools: systematic literature review and narrative synthesis. JMIR Hum Factors. 2023 Mar 23;10:e45143. doi:10.2196/45143. PMID: 36843321.
Ribaut J, DeVito Dabbs A, Dobbels F, et al. Developing a comprehensive list of criteria to evaluate the characteristics and quality of eHealth smartphone apps: systematic review. JMIR Mhealth Uhealth. 2024 Jan 15;12:e48625. doi:10.2196/48625. PMID: 38224477.
Onukwugha FI, Smith L, Kaseje D, et al. The effectiveness and characteristics of mHealth interventions to increase adolescents' use of sexual and reproductive health services in Sub-Saharan Africa: a systematic review. PLoS One. 2022 Jan 21;17(1):e0261973. doi:10.1371/journal.pone.0261973. PMID: 35061757.
Fadahunsi KP, O'Connor S, Akinlua JT, et al. Information quality frameworks for digital health technologies: systematic review. J Med Internet Res. 2021 May 17;23(5):e23479. doi:10.2196/23479. PMID: 33835034.
Munn Z, Pollock D, Khalil H, Alexander L, McInerney P, Godfrey CM, et al. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022;20(4):950-952. doi:10.11124/JBIES-21-00483. PMID:35249995
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. doi:10.1080/1364557032000119616
Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009 Jul 21;6(7):e1000097. doi:10.1371/journal.pmed.1000097. Epub 2009 Jul 21. PMID: 19621072.
Shetty J, Shetty A, Mundkur SC, et al. Economic burden on caregivers or parents with Down syndrome children: a systematic review protocol. Syst Rev. 2023;12:3. doi:10.1186/s13643-022-02165-2
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77‑101. doi:10.1191/1478088706qp063oa.
Critical Appraisal Skills Program. CASP Qualitative Checklist. 2018. Available from: https://casp-uk.net/casp-tools-checklists/ (accessed 15 March 2024).
Institute JB. JBI’s critical checklist for quasi-experimental studies. Available from: https://jbi.global/critical-appraisal-tools (accessed 20 March 2024).
El Joueidi S, Bardosh K, Musoke R, et al. Evaluation of the implementation process of the mobile health platform ‘WelTel’ in six sites in East Africa and Canada using the modified consolidated framework for implementation research (mCFIR). BMC Med Inform Decis Mak. 2021;21:293. doi:10.1186/s12911-021-01644-1.
Arrossi S, Paolino M, Orellana L, et al. Mixed-methods approach to evaluate an mHealth intervention to increase adherence to triage of human papillomavirus-positive women who have performed self-collection (the ATICA study): study protocol for a hybrid type I cluster randomized effectiveness-implementation trial. Trials. 2019;20:148. doi:10.1186/s13063-019-3229-3.
Lagan S, Aquino P, Emerson MR, et al. Actionable health app evaluation: translating expert frameworks into objective metrics. npj Digit Med. 2020;3:100. doi:10.1038/s41746-020-00312-4
Jones C, O'Toole K, Jones K, et al. Quality of psychoeducational apps for military members with mild traumatic brain injury: an evaluation utilizing the Mobile Application Rating Scale. JMIR Mhealth Uhealth. 2020;8(8):e19807. URL: https://mhealth.jmir.org/2020/8/e19807. doi:10.2196/19807
Roberts AE, Davenport TA, Wong T, et al. Evaluating the quality and safety of health-related apps and e-tools: adapting the Mobile App Rating Scale and developing a quality assurance protocol. Internet Interv. 2021 Mar 17;24:100379. doi:10.1016/j.invent.2021.100379. PMID: 33777705.
Sedhom R, McShea MJ, Cohen AB, et al. Mobile app validation: a digital health scorecard approach. npj Digit Med. 2021;4:111. doi:10.1038/s41746-021-00476-7.
Tan YY, Woulfe F, Chirambo GB, et al. Framework to assess the quality of mHealth apps: a mixed-method international case study protocol. BMJ Open. 2022;12:e062909. doi:10.1136/bmjopen-2022-062909.
Woulfe F, Fadahunsi KP, O'Grady M, et al. Modification and validation of an mHealth app quality assessment methodology for international use: cross-sectional and eDelphi studies. JMIR Form Res. 2022 Aug 19;6(8):e36912. doi:10.2196/36912. PMID: 35984688.
Little LM, Pickett KA, Proffitt R, et al. Keeping pace with 21st century healthcare: a framework for telehealth research, practice, and program evaluation in occupational therapy. Int J Telerehabil. 2021 Jun 22;13(1):e6379. doi:10.5195/ijt.2021.6379. PMID: 34345350.
Alon N, Stern AD, Torous J. Assessing the Food and Drug Administration's risk-based framework for software precertification with top health apps in the United States: quality improvement study. JMIR Mhealth Uhealth. 2020 Oct 26;8(10):e20482. doi:10.2196/20482. PMID: 32927429.
Ndlovu K, Mars M, Scott RE. Validation of an interoperability framework for linking mHealth apps to electronic record systems in Botswana: expert survey study. JMIR Form Res. 2023 May 2;7:e41225. doi:10.2196/41225. PMID: 37129939.
Khowaja K, Al-Thani D. New checklist for the heuristic evaluation of mHealth apps (HE4EH): development and usability study. JMIR Mhealth Uhealth. 2020 Oct 28;8(10):e20353. doi:10.2196/20353. PMID: 33112252.
Vokinger KN, Nittas V, Witt CM, et al. Digital health and the COVID-19 epidemic: an assessment framework for apps from an epidemiological and legal perspective. Swiss Med Wkly. 2020 May 17;150:w20282. doi:10.4414/smw.2020.20282. PMID: 32418194.
Haig M, Main C, Chávez D, et al. A value framework to assess patient-facing digital health technologies that aim to improve chronic disease management: a Delphi approach. Value Health. 2023 Oct;26(10):1474-1484. doi:10.1016/j.jval.2023.06.008. Epub 2023 Jun 28. PMID: 37385445.
van Haasteren A, Vayena E, Powell J. The mobile health app trustworthiness checklist: usability assessment. JMIR Mhealth Uhealth. 2020 Jul 21;8(7):e16844. doi:10.2196/16844. PMID: 32706733.
Jahnel T, Pan C-C, Pedros Barnils N, Muellmann S, Freye M, Dassow H-H, Lange O, Reinschluessel AV, Rogowski W, Gerhardus A. Developing and evaluating digital public health interventions using the Digital Public Health Framework DigiPHrame: a framework development study. J Med Internet Res. 2024;26:e54269. doi:10.2196/54269
Davis MM, Freeman M, Kaye J, et al. A systematic review of clinician and staff views on the acceptability of incorporating remote monitoring technology into primary care. Telemed J E Health. 2014 May;20(5):428-438. doi:10.1089/tmj.2013.0166. PMID: 24731239.
Moharra M, Almazán C, Decool M, et al. Implementation of a cross-border health service: physician and pharmacists' opinions from the epSOS project. Fam Pract. 2015 Oct;32(5):564-567. doi:10.1093/fampra/cmv052. PMID: 26148726.
Rouleau G, Wu K, Ramamoorthi K, Boxall C, Liu RH, Maloney S, Zelmer J, Scott T, Larsen D, Wijeysundera HC, Ziegler D, Bhatia S, Kishimoto V, Steele Gray C, Desveaux L. Mapping theories, models, and frameworks to evaluate digital health interventions: scoping review. J Med Internet Res. 2024;26:e51098. doi:10.2196/51098
Weirauch V, Soehnchen C, Burmann A, Meister S. Methods, indicators, and end-user involvement in the evaluation of digital health interventions for the public: scoping review. J Med Internet Res. 2024;26:e55714. doi:10.2196/55714.
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Issam El Kouarty, MD, Samia El Hilali, MD, Abdelmajid Sahnoun, MD, Majdouline Obtel, PhD
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors retain copyright of their work, with first publication rights granted to Telehealth and Medicine Today (THMT).
THMT is published under a Creative Commons Attribution-NonCommercial 4.0 International License.
