ORIGINAL RESEARCH

The Impact of Secure Messaging Telehealth Service on the Quality of Healthcare

Dong-Gil Ko, PhD symbol

Department of Operations, Business Analytics, and Information Systems, University of Cincinnati, Cincinnati, Ohio, USA

Keywords: patient satisfaction, quality of healthcare, secure messaging, telehealth

Abstract

Background: Secure messaging within electronic health records (EHRs) is the fastest-growing component of virtual care in the telehealth ecosystem. It is a cost-effective way to communicate medical advice and an alternative to an in-person, video, or telephone visits. Secure messaging is shown to have a positive effect on the management of chronic diseases. It can improve health awareness and literacy and also improve communication and care coordination. Healthcare providers have experienced a 200% increase in secure messages over the past few years, and in some cases, it is associated with avoidance of medical visits and burnout among healthcare providers. Such paradoxical outcomes raise questions about the value of secure messages and whether they positively impact the quality of healthcare outcomes.

Objective: To conduct an empirical assessment and evaluate the extent to which and for whom secure messages improve the quality of healthcare.

Methods: EHRs of adult patients with type 2 diabetes mellitus and an active patient portal account who visited an academic medical center in the Midwest US between 2015 and 2020 were included in the analysis. Patients must have initiated a secure message, made a follow-up in-person appointment, and completed the CAHPS (Consumer Assessment of Healthcare Providers and Systems) patient satisfaction survey.

Results: The final regression analysis included 1,332 patients with no missing data. The results reveal a negative relationship between patients who discuss their secure messages during their encounters and patient satisfaction even after controlling for patient-specific (age, gender, and insurance), provider-specific (responsiveness), and medical conditions (average blood sugar and mean arterial pressure). The results also show a stronger negative effect for the underrepresented minorities (Blacks and Hispanics).

Summary: The negative relationship between secure message exchanges and patient satisfaction may stem from differences in providers’ subject matter expertise and their mental models that are incongruent with patients’ general health knowledge. This suggests that patients may struggle to understand the value of new information, assimilate it, and apply it. Furthermore, effective communication is fundamental to patient satisfaction, and healthcare providers may be challenged to instill greater understanding during their average 18-min encounters. The proliferation of digital divide research in healthcare indicates that Afro-Americans and Hispanic Americans are at a disadvantage; therefore, it is not unreasonable to suggest that the health literacy rate differs, possibly creating greater uncertainty and ambiguity. More research probing the role and value of secure messaging is needed as healthcare providers begin to charge patients for this telehealth service.

Plain Language Summary

This study evaluated the impact of secure messaging telehealth service on the quality of healthcare using EHR of adult patients with type 2 diabetes mellitus. Based on the findings in 1,332 patients, there was a negative association between secure message exchanges and healthcare quality, and this effect was stronger for underrepresented minorities. The results suggest that personal health literacy may be a vital key to improving the quality of healthcare, and underrepresented minorities may be at a disadvantage due to delayed access to digital technologies. These findings underscore the importance of conducting additional research examining the value of secure messaging as healthcare providers threaten to eliminate the only remaining free telehealth service.

 

Citation: Telehealth and Medicine Today © 2023, 8: 435 - https://doi.org/10.30953/thmt.v8.435

Copyright: © 2023 The Authors. This is an open-access article distributed in accordance with the Creative Commons Attribution Non-Commercial (CC BY-NC 4.0) license, which permits others to distribute, adapt, enhance this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0.

Received: August 23, 2023; Accepted: September 9, 2023; Published: September 27, 2023

Funding Statement: This research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors. This research was completed voluntarily.

Financial and non-Financial Relationships and Activities: No relevant disclosures.

Corresponding Author: Dong-Gil Ko, Email: kodg@ucmail.uc.edu

 

Among American adults, 83.4% visit providers annually,1 and 68% of patients use telehealth—an increase from 45% 2 years earlier.2 During the same period, the number of secure messages doubled.35 Telehealth is a health-related service and a platform for the distribution of information using health information technologies. These include video visits, phone visits, “e-visits,” and secure messaging, with each technology becoming increasingly more accessible and important for managing healthcare.6 Secure messaging is the only free telehealth service where patients can prompt questions and facilitate health knowledge exchanges with their healthcare providers. It is an area calling for additional research as secure message exchanges enhance healthcare quality.

Secure messaging is a custom-tailored approach designed to address each patient’s unique needs. Patients send secure messages to their healthcare providers in order to ask for nonurgent advice about their health and medical conditions, as well as to improve health literacy that occurs through collaborative knowledge exchanges. Because of incongruent mental models between the patients (less knowledgeable) and providers (subject matter experts), patients may rely on the free secure messaging telehealth service to close the “knowledge gap.”7 This health knowledge-seeking behavior highlights the provider–patient interpersonal dynamics related to an external capability focused on patient satisfaction.

Patient satisfaction is defined as a patient’s judgment of the overall experience after an encounter or receiving a medical service.8 It is a de facto indicator of healthcare quality9 and is tied to in-person encounters. Establishing relationships associating healthcare quality with telehealth services would require identifying secure messaging exchanges that “spilled” into in-person encounters. In other words, the secure message exchanges that occur before in-person encounters could be integral in impacting patient satisfaction.

Secure messaging telehealth service has been suggested to impact the underrepresented minorities differentially. The rapid penetration of broadband and mobile internet use has contributed to maximizing equal access. While the delayed access to telehealth services has disadvantaged the underrepresented minorities and made it challenging for them to gain health literacy at the same rate as Americans of European descent, secure message exchanges were found to supplement their interactions with their providers by enabling them to address their concerns more thoroughly and accurately.10

This study performed a retrospective analysis using data from adult patients with type 2 diabetes mellitus (T2DM) at an academic medical center located in the urban Midwest US and reports the use of secure message telehealth service by evaluating the extent to which and for whom secure message exchanges improved the quality of healthcare. Study design, data, and analytical approaches are described, followed by results, discussion, and concluding remarks.

METHODS

The University Institutional Review Board approved this study. Electronic health records (EHRs), including the contents of secure messages, came from the academic medical center (an urban hospital in the Midwest US), where diabetes is the largest patient population served, requiring frequent routine care. Data were fully deidentified per HIPAA (Health Insurance Portability and Accountability Act of 1996) privacy requirements after the unstructured data were analyzed and linked with structured data. No identifying information of individuals or covered entities was utilized.

The author was given access to the EHRs of adult patients with T2DM, including the industry-approved healthcare quality data—the CAHPS survey of patient satisfaction and a performance metric associated with healthcare improvements, profitability, and the industry’s transition to value-based care. Patients diagnosed with T2DM between October 1, 2015 (the switching date from ICD-9 to ICD-10 [International Classification of Diseases, 9th and then 10th Revision, Clinical Modification] mandated by the Centers for Medicare & Medicaid Services) and August 30, 2020 (the cut-off date for data collection) were examined in this study. The EHR contains a coding scheme, ICD-10-CM, which must be used for all electronic healthcare transactions. Included in this study is the ICD-10-CM code E11.x indicating patients diagnosed with T2DM and a specification of additional levels of T2DM (if any).

To be included in the final data analysis, (i) secure messages had to be associated with the patient’s next in-person encounter; (ii) a patient satisfaction survey had to be linked to the same encounter; and (iii) there were no missing data. By taking a more restrictive approach to ensure the in-person encounter is related to the secure message exchanges, the timestamp of the “Patient Appointment Schedule Request” must not precede but must be within a day of the last secure message, the encounter date must be the same as the appointment date, the chief complaint of the encounter is related to the secure message exchanges, and there are no missing data.

Deidentified patient and encounter identifications and dates allow data from different areas of the EHR to be linked without ambiguity. Content analysis was performed on the secure messages and compared with the appointment’s chief complaint, which describes the symptom, problem, condition, diagnosis, or other factors that led the patient to seek medical attention. The relationship was determined based on matching words associated with a symptom (e.g., headache), problem (e.g., unable to sleep), condition (e.g., glucose and blood pressure), diagnosis (e.g., diabetes and depression), or another factor (e.g., body areas such as abdomen or knee).

After data cleansing, pre-processing, and evaluation, there were 1,332 unique patients with complete T2DM data. The dependent variable, or the patient satisfaction score, came from the CAHPS survey using a 10-point scale question: Using any number from 0 to 10, where 0 is the worst provider possible, and 10 is the best provider possible, what number would you use to rate this provider?” The academic medical center converted the patient satisfaction score to an internal performance rating (1–10) to make operational and evaluation decisions. Covariates and independent variables are defined in Table 1.

Table 1. Covariates and independent variables among the 1,332 patients with unique type 2 diabetes mellitus and complete data
Variable Defined as…
1. COVID-19 For the forced transformation to online due to the pandemic. It takes a “0” for encounters that occurred before March 11, 2020. It takes a “1” for encounters beginning March 11, 2020. The state where the medical center is located declared a state of emergency with effect from March 11, 2020.
2. Age Biological age of the patient
3. Gender Male or female at birth
4. Insurance Whether or not private health access
5. Hemoglobin A1c Used to diagnose T2DM and monitor blood glucose levels. Hba1c reflects an average blood glucose level for the past 2 to 3 months. Higher HbA1c reflects a higher risk of diabetes complications. Less than 5.7% is normal, 5.7% to 6.4% is diagnosed as prediabetes, and above 6.4% on two separate tests indicates diabetes.
6. Mean arterial pressure (MAP) Readings outside this range reflect too low or too high pressure as an indication of global perfusion pressure necessary for organ perfusion and oxygen delivery. Physicians rely on MAP, as opposed to systolic and diastolic blood pressure, to determine whether there are sufficient blood flow, resistance, and pressure to supply blood to all major organs. MAP readings between 60 mmHg and 100 mmHg are considered normal.
7. Underrepresented minorities Afro-Americans and Hispanic Americans
8. Secure message exchanges or nonurgent medical advice requests Originated by patients, followed by subsequent responses. One secure message exchange is defined as a secure message sent and received.
9. Provider responsiveness The amount of time it takes providers to respond to patients’ secure messages. It is a system-captured measure equivalent to “waiting time” experienced during an in-person encounter that has been shown to negatively impact patient satisfaction.11
Hba1c: hemoglobin A1c; MAP: mean arterial pressure; T2DM: type 2 diabetes mellitus.

The impact of secure messages on the quality of healthcare was modeled using the following regression equation:

yi = α + β1x1 + C’γ1 + M’δ1 + N’ζ1 + K’η1 + ɛi.

where yi represents patient satisfaction, and x1 is a under-represented minority (URM). C is a vector of patient covariates, including the COVID-19 period, patient age, gender, insurance, and provider responsiveness. M is the T2DM patients’ health condition HbA1c and Mean arterial pressure (MAP) (a more accurate indicator of perfusion to vital organs than systolic blood pressure). N is the patients’ secure message exchanges with providers. K is an interaction effect between URM and secure message exchanges.

RESULTS

In this study, a new perspective on the linkage between secure message exchanges and patient satisfaction was developed using disparate data sources involving 1,332 patients with T2DM. Table 2 presents summary statistics of all variables used in this study. Table 3 shows correlations. The average patient satisfaction was converted to a performance rating is 0.87 (SD = 0.338). Although 86.1% of the appointments in the dataset occurred during the pandemic, it should be noted that the data used came from the patient’s most recent appointment. The median age is 63 years, with an average of 61.32 years (SD = 12.53 years). Females comprised 54.5% of the sample, and 44.1% carried private insurance. The average HbA1c for the dataset is 7.21% (SD = 1.438%), with 80.3% of the sample population having normal MAP readings. In the sample population, 21.5% are URM. On average, there were 1.72 secure message exchanges (SD = 0.954) ranging from 1 to 7 exchanges (recall, 1 exchange = a message sent and received) before a patient scheduled an appointment through the patient portal.

Table 2. Descriptive statistics of all variables used in this study
Variables Sample size Mean Std. Dev. Range
Performance rating 1,475 0.868 0.338 0–1
COVID-19 1,475 0.861 0.346 0–1
Age 1,475 61.32 12.526 19–96
Female 1,475 1.55 0.498 1–2
Insurance 1,474 1.44 0.497 1–2
Hemoglobin A1c 1,475 7.207 1.438 4.1–15.0
Mean arterial pressure 1,389 1.803 0.398 1–2
Provider responsiveness 1,475 0.612 4.277 0–96.09
Under-represented minority 1,412 1.22 0.411 1–2
Secure message exchanges 1,475 1.72 0.954 1–7
Std. Dev.: Standard deviation.

 

Table 3. Correlations among variables
Variable 1 2 3 4 5 6 7 8 9 10
1. Performance rating 1
2. COVID-19 0.052* 1
3. Age 0.133** 0.012 1
4. Female −0.025 0.027 −0.168* 1
5. Insurance −0.028 0.012 −0.422** 0.033 1
6. Hemoglobin A1c 0.019 0.011 −0.176** 0.042 0.112** 1
7. Mean arterial pressure 0.026 0.019 0.134** 0.040 −0.063* −0.090** 1
8. Provider responsiveness 0.038 −0.065* 0.035 −0.018 −0.035 −0.006 −0.028 1
9. Under-represented minority −0.010 0.064* −0.182** 0.235** 0.023 0.060* −0.135** −0.043 1
10. Secure message exchanges −0.045 0.010 −0.053* −0.009 0.022 0.008 −0.016 −0.064* 0.027 1
*p < 0.05; **p < 0.01.

This study revealed that patients are less satisfied with their providers as they increasingly seek medical advice through secure messaging. The regression analysis (Table 4—Main) shows a statistically significant and negative relationship between secure message exchanges and patient satisfaction (b = −0.057, p < 0.05) even after controlling for patient-specific factors (age, gender, and insurance), medical conditions (HbA1c and MAP), and a provider-specific factor (provider responsiveness – a proxy for waiting time). Moreover, the effect for the URM is strikingly stronger (Table 4—Main): there is a stronger statistically significant and negative relationship between underrepresented minorities seeking medical advice through secure messaging and patient satisfaction (b = −0.071, p < 0.05).

Table 4. Regression analysis
Variable Main (Beta) Robustness check (Beta)
COVID-19 0.046 0.039
Age 0.134** 0.133**
Female −0.001 0.004
Insurance (private) 0.028 0.028
Hemoglobin A1c 0.037 0.038
Mean arterial pressure 0.009 0.010
Provider responsiveness 0.040 0.039
Under-represented minority 0.039 0.036
Secure message exchanges −0.057* −0.059*
Interaction effect (URM * SME) −0.071* −0.07*
Adjusted gross income 0.001
Digital literacy 0.025
F-statistics 3.725** 3.204**
Degrees of freedom 10, 1,321 12, 1312
Sample size (N) 1,332 1,325
URM (under-represented minority) *SME (secure message exchanges); *p <0.05. **p < 0.001.

To further assess whether the findings observed in this study are influenced by other factors that disproportionately impact healthcare quality, an additional robustness test was performed using socioeconomic status and digital literacy. Using the 2019 IRS Individual Income Tax Statistics ZIP Code (mail delivery area) Data, the average adjusted gross income and the average number of individuals who use computers to prepare their income tax returns at each patient’s ZIP Code level were used as proxies for socioeconomic status and digital literacy, respectively. The overall results remained consistent, suggesting evidence of structural validity (Table 4 – Robustness Check).

DISCUSSION

There are two plausible explanations for the statistically significant and negative relationships between secure message exchanges and the quality of healthcare. First, the knowledge management framework describes mental models as stocks of prior knowledge that differentiate patients and providers7—patients have general health knowledge, whereas providers are subject matter experts. In other words, patients’ and providers’ mental models are incongruent, suggesting that the patient’s ability to recognize the value of new information, assimilate it, and apply it differs from the provider’s absorptive capacity.12 For patients with T2DM with their low absorptive capacity, we explore in search of discovering a “breadth” of information. In contrast, medically trained and certified providers with high absorptive capacity exploit by applying known certainties, or “depth” of information, to achieve refinements or efficiency gains.13 Varying information search and assimilation processes and behaviors inherent in different mental models introduce communication and knowledge “stickiness” that inhibit effective knowledge transfer. One of the best methods for overcoming communication and knowledge stickiness is to close the “knowledge gap” through learning that occurs through collaborative knowledge exchanges.12

Second, effective communication is fundamental to patient satisfaction,14 and prior studies have pointed to communication science as a potential source that explains why telehealth is unable to realize its potential fully. As patients seek medical advice through secure messaging, providers might find it difficult to use more straightforward language, provide specific examples, or articulate more precise explanations through the digital medium. The added work of portal interaction is seen as an increase in workload and a missed opportunity to take care of another patient, prompting providers to place less emphasis on communication effectiveness. The consequences of not addressing it early can have a negative downstream effect. Due to different mental models and limited training in communication skills, it is possible that providers have not adequately equipped their patients with the medical advice response they were seeking. This study examined patients’ encounters that are directly related to secure message exchanges; therefore, providers and patients are anticipated to discuss the medical advice request during their 18-min encounters. In doing so, many problems may arise, including less time to address other medical needs and increased confusion associated with communication barriers. Patients with limited medical knowledge may form a negative perception from such encounters, which is ultimately captured in the CAHPS Survey. The application of communication science begins with empathetically responding to medical advice requests through secure messaging. Application of the Centers for Disease Control and Prevention’s Clear Communication Index can improve communication, enhance clarity, and aid understanding of medical knowledge.

The results also suggest that there is a stronger negative association between underrepresented minorities (Afro-Americans and Hispanic Americans) who seek medical advice using secure messaging and patient satisfaction. Prior research consistently reported that patients of European descent have the advantage of experiencing and gaining access to telehealth sooner and thereby benefit from accelerated digitalization in healthcare.15 An underlying assumption is that early access to telehealth provides opportunities to gain general and personal health knowledge and make informed decisions. The focus of this study was not on the widespread digital divide argument since all 1,332 T2DM patients have access to the secure messaging feature. Instead, the high absorptive capacity that comes with experience enables more knowledgeable patients to recognize words better, detect patterns, or understand their medical conditions relative to their less knowledgeable counterparts. The proliferation of digital divide research in healthcare makes it clear that the lack of access to telehealth has placed underrepresented minorities at a disadvantage, making it challenging for them to gain health literacy at the same rate as Americans of European descent.16 It is reasonable to suggest that providers may not have effectively converted their “tacit” knowledge to “explicit” knowledge that patients can understand. Alternatively, the health information provided does not effectively cater to patients with low literacy. This is more challenging in asynchronous secure message exchanges, which may have prompted patients to schedule appointments. The underrepresented minorities’ low absorptive capacity and health literacy, coupled with high expectations from the encounter, place an added burden on the providers’ ability to effectively communicate to patients’ satisfaction.

Limitations

Despite the valuable insights gained, this study is not without limitations. This study examined one disease within one health system. The results should be interpreted cautiously as the findings may not generalize to other health systems and locations. However, this study sample of 21.5% underrepresented minorities is representative of the local population and comparable to 15.6% in the state and 28.5% in the urban area surrounding the academic medical center. This study was also limited in the data it received. In particular, access to patient clinical notes would offer greater details about the continuity of care and the role of secure messages in managing the quality of healthcare and outcomes.

CONCLUSIONS

The digitization of the U.S. healthcare systems continues to be a priority, given the expected capabilities enabled by telehealth to improve health, prevent or delay complications, and reduce disparities. This study set out to answer a fundamental question: “Does and for whom secure messages lead to improved quality of healthcare?” According to this study, the answer is “No.” The more patients seek nonurgent medical advice through secure messages, the more they are likely to experience dissatisfaction with their related in-person encounters, and the effects are stronger for the underrepresented minorities. The findings observed in this study offer empirical evidence about the relationship between secure message exchanges and patient satisfaction and shed new insight that closing the “knowledge gap” or developing personal health literacy may be a vital key to improving healthcare quality.

The communication science theoretical lens offers a direction for shrinking the knowledge gap. Namely, providers must obtain training in developing the necessary skills to communicate more effectively and take advantage of patients’ proactive requests for medical advice. The knowledge management theoretical lens explains the challenges associated with communicating with those with different mental models developed based on prior experience and knowledge. At its core, both patients and providers must continuously learn. The additional challenge experienced by providers to convert “tacit” knowledge may be futile if patients do not continue to develop their health literacy. When both “boats rise,” providers can more easily tailor their communication, and patients can better assimilate and understand new information. Though the digital divide has allowed Americans of European descent to get a head start and the underrepresented minorities are “catching up,”15 an implication arising from this study is that both patients and providers must be invested in developing patients’ health literacy if we are to improve the quality of healthcare.

More study is needed to establish a causal link between secure messaging and the quality of healthcare. In doing so, there is an opportunity to innovatively leverage the growing telehealth services to make a lasting impact. Future secure messaging research could also improve patient satisfaction, communication, and health literacy by applying natural language processing and machine learning techniques to offer eighth-grade-level recommended responses using population real-world evidence in the EHR. As of November 14, 2021, healthcare providers threaten to charge for secure messaging telehealth services, and providers have begun implementing secure messaging billing models.5

CONTRIBUTORS

Dr Ko conceived and designed the study; acquired, analyzed, and interpreted the data; and wrote the manuscript.

ACKNOWLEDGMENTS

The contributions of the “Ko’s Lab” at the Lindner College of Business, University of Cincinnati, played a pivotal role in the successful completion of this research and the development of this manuscript.

REFERENCES

  1. Center for Disease Control and Prevention. FastStats—Physician office visits [Internet]. 2019. Available from: https://www.cdc.gov/nchs/fastats/physician-visits.htm [cited 6 May 2023].
  2. ONC Tech Forum: Insights into API use to enable data sharing between EHRs and apps | HealthIT.gov [Internet]. Available from: https://www.healthit.gov/news/events/onc-tech-forum-insights-api-use-enable-data-sharing-between-ehrs-and-apps [cited 6 May 2023].
  3. Holmgren AJ, Downing NL, Tang M, Sharp C, Longhurst C, Huckman RS. Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. J Am Med Inform Assoc. 2022;29(3):453–60. doi: 10.1093/jamia/ocab268
  4. Charleson K. Telehealth statistics and trends: A 2021 report [Internet]. The Checkup. 2021. Available from: https://www.singlecare.com/blog/news/telehealth-statistics/ [cited 3 September 2023].
  5. Raths D. Healthcare Innovation. Health systems begin charging for some portal message responses [Internet]. Available from: https://www.hcinnovationgroup.com/clinical-it/patient-portals/article/21287423/health-systems-begin-charging-for-some-portal-message-responses [cited 6 May 2023].
  6. Bao C, Bardhan I, Singh H, Meyer B, Kirksey K. Patient–provider engagement and its impact on health outcomes: a longitudinal study of patient portal use. MISQ. 2020;44(2):699–723. doi: 10.25300/MISQ/2020/14180
  7. Ko DG, Dennis AR. Profiting from knowledge management: The impact of time and experience. Inform Syst Res. 2011; 22(1): 134–52. doi: 10.1287/isre.1090.0247
  8. Ancarani A, Di Mauro C, Giammanco MD. Patient satisfaction, managers’ climate orientation and organizational climate. Jonsson P, Johansson M, editors. Int J Operat Prod Manage. 2011;31(3):224–50. doi: 10.1108/01443571111111900
  9. Salzarulo PA, Bretthauer KM, Côté MJ, Schultz KL. The impact of variability and patient information on healthcare system performance. Prod Operat Manage. 2011;20(6):848–59. doi: 10.1111/j.1937-5956.2010.01210.x
  10. Ordaz OH, Croff RL, Robinson LD, Shea SA, Bowles NP. Optimization of primary care among Black Americans using patient portals: qualitative study. J Med Internet Res. 2021;23(6):e27820. doi: 10.2196/27820
  11. Ko DG, Mai F, Shan Z, Zhang D. Operational efficiency and patient-centered healthcare: a view from online physician reviews. J Operat Manage. 2019;65(4):353–79. doi: 10.1002/joom.1028
  12. Ko DG, Kirsch LJ, King WR. Antecedents of knowledge transfer from consultants to clients in enterprise system implementations. MIS Q. 2005;29(1):59–85. doi: 10.2307/25148668
  13. March JG. Exploration and exploitation in organizational learning. Organ Sci. 1991;2(1):71–87. doi: 10.1287/orsc.2.1.71
  14. Shiraly R, Mahdaviazad H, Pakdin A. Doctor-patient communication skills: a survey on knowledge and practice of Iranian family physicians. BMC Fam Pract. 2021;22(1):130. doi: 10.1186/s12875-021-01491-z
  15. Mai F, Ko DG, Shan Z, Zhang D. The impact of accelerated digitization on patient portal use by underprivileged racial minority groups during COVID-19: Longitudinal study. J Med Internet Res. 2023;25(1):e44981. https://doi.org/10.2196/44981
  16. Smith SG, O’Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc. 2015;22(4):888–95. https://doi.org/10.1093/jamia/ocv025

Copyright Ownership: This is an open-access article distributed in accordance with the Creative Commons Attribution Non-Commercial (CC BY-NC 4.0) license, which permits others to distribute, adapt, enhance this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0.