Healthcare Professionals and Telehealth Usability during COVID-19

Dr. Jing Xu, Ph.D.1 symbol; Dr. Hanadi Y. Hamadi, Ph.D.1 symbol; Dr. Kristen K. Hicks-Roof, Ph.D., RDN, LDN, FAND2 symbol; Dr. Robert J. Zeglin, Ph.D., MA, BS, CST, NCC, LMHC3 symbol; Ms. Chloe E. Bailey, MHA, BHA1 symbol; Dr. Mei Zhao, Ph.D.1 symbol

1Department of Health Administration, Brooks College of Health, University of North Florida, Jacksonville, Florida; 2Department of Nutrition & Dietetics Brooks College of Health, University of North Florida, Jacksonville, Florida; 3Department of Public Health, Brooks College of Health, University of North Florida, Jacksonville, Florida.

Abstract

Objective: During the coronavirus disease 2019 (COVID-19) pandemic, many other health providers needed to rapidly adopt telehealth services to ensure continuity of patient care, without the opportunity to extensively evaluate the usability of the adopted technology. Therefore, this study aims to examine health professionals’ telehealth usability during COVID-19 in Florida.

Design: This cross-sectional study employed the Telehealth Usability Questionnaire (TUQ) to licensed healthcare providers in Florida in June 2020.

Setting and Participants: A total of 399,660 selected health professionals with Florida licensure were recruited from open-access Florida healthcare to participate in a Qualtrics web-based survey. A total of 1,868 health professionals completed the survey. Multiple linear and mixed regression models were applied to analyze the overall and subdomain scores from TUQ.

Main Outcome Measures: Telehealth usability.

Results: The analysis of the overall TUQ score showed younger, female healthcare professionals, and participants who reported an increase in telehealth usage during pandemic had a significantly higher overall TUQ score. Compared with the score from physicians and nurses, the scores from the mental health group and social work group were significantly higher, while the score rehabilitation group was significantly lower. Analysis of the subdomain scores was consistent with the overall scores.

Conclusion: The findings from this study indicate that the health professionals’ telehealth usability is related to age, gender, and the change of telehealth usage during the COVID-19 pandemic. While pandemics represent only one possible impetus for the healthcare system to swiftly switch to telehealth platforms, each profession should consider providing adequate resources to accommodate the need for change.

Keywords: COVID-19, Florida, Health Professionals, Telehealth, Telehealth Usability, Telemedicine

 

Citation: Telehealth and Medicine Today 2021, 6: 270 - http://dx.doi.org/10.30953/tmt.v6.270

Copyright: © 2021 The Authors. This is an open access article distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, adapt, and enhance this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See http://creativecommons.org/licenses/by-nc/4.0.

Published: 18 November 2021

Competing interests and funding: No funding was received from any source for preparing this article for publication.

Corresponding Author: Hanadi Y. Hamadi, Email: h.hamadi@unf.edu

 

INTRODUCTION

At the end of December 2019, patients were admitted to hospitals in Wuhan, Hubei Province, China, with symptoms of pneumonia of unknown origin. This was the first time the new coronavirus disease 2019 (COVID-19) was encountered, where reports predicted the early onset of a potential outbreak of this coronavirus.1 By March 11th, the World Health Organization declared COVID-19 a global pandemic.2 States, counties, and cities nationwide issued stay at home orders, while many businesses and restaurants are closed for the most part to flatten the curve and avoid a surge of patients flooding healthcare facilities across the country.3 By July 29th of 2020, Florida became a new epicenter for COVID-19 at 441,977 total confirmed cases with 68.1% of cases occurring within the month of July.4 As a precaution and ways to mitigate the spread of COVID-19, many hospitals and healthcare providers limited and even halted medically unnecessary and elective procedures5–7 and access to care.8–10 Therefore, there is a dire need to examine the ability of health providers to use these technologies to improve their care delivery remotely. During the start of the pandemic, telemedicine visits were found to have increased by 683% in the months of March and April alone with a staggering 700 additional virtual visits a day.

Telehealth Utilization Prior to COVID-19

Telehealth technologies define a vast array of activities that allow the delivery of care and the interaction of provider-to-patient or provider-to-provider through synchronous (phone and video) and asynchronous (store and forward such as patient portals) communication and virtual agents (telemonitoring through wearable devices).11 Currently, there is a great need for telehealth services during these trying times as many organizations have expanded their telehealth capabilities to serve the patients to the best of their ability while keeping them safe at home.12 It allows for services that have been traditionally conducted by acute care hospitals and health provider offices during face-to-face outpatient visits to be delivered to underserved communities and populations.13 By December of 2016, The Agency for Health Care Administration reported that less than half (44.8%) of hospitals in Florida were found to have telehealth services available to their patients.14

However, telehealth services are not new, more recently, the biggest need for these types of services is to manage chronic diseases.12 An estimated 100 million Americans suffer from chronic diseases such as heart disease, cancer, stroke, diabetes, arthritis, asthma, hypertension or high blood pressure, emphysema, bronchitis, depression, anxiety, and others. These chronic diseases account for 75% of healthcare expenditures.15 In Florida, 46.7% of all Medicare enrollees ages 65 and more reported having at least four chronic conditions, indicating that the large aging population within the state is contributing to the higher proportion of chronic conditions.16 Traditionally, these illnesses are managed by the patient visiting the primary care offices episodically; however, using telehealth services would offer more benefits such as a reduction in hospitalizations, readmission, length of stay, and costs.17,18 Telehealth also offers opportunities for consistent communication between patients and clinicians with the use of tools such as smartphones, tablets, or computers that possess webcams.19 Furthermore, telehealth technologies can be extremely effective when it comes to reducing the cost of healthcare by saving an estimated $19–$121 per virtual visit. These savings come from reduced operational overhead and technology adaption, which is of great necessity during a pandemic that has created a healthcare recession that resulted in over 1.4 million healthcare jobs lost in April 2020 and 30,000 jobs in January 2021.20,21

Telehealth Policies during COVID-19

Furthermore, in March 2020, Centers for Medicare & Medicaid Services (CMS) has provided mass waivers for telehealth reimbursements, allowing for healthcare facilities to provide these services for new and established patients during times of public health emergencies. Additionally, these emergency regulations promote broader access to patients than ever before and allow more providers to take part in conducting their care through virtual work.22 One of the telehealth’s largest obstacles before the pandemic was regarding certain state governments requiring health providers to hold a license in all states they provide care or consultative services in, whether virtually or in-person. Yet, the recent emergency legislation has bypassed most states’ requirements, with some mandating only an emergency credential application to perform virtual consultations in the new COVID-19 era.23 This act has significantly broadened accessibility for health providers and patients. For example, many surveyed physicians have reported that the lifted telehealth restrictions have eased their workload more efficiently, leading to increased care quality and greater satisfaction with newly adopted technology.24 Ultimately, these policies support the promise of telehealth with its proven effectiveness to connect more patients with the quality care they deserve and open the door to have telehealth and make its place in the future of healthcare.

Purpose of This Study

There is an increasing need to use telehealth technologies and increase policy incentivizing its use now more than ever during the COVID-19 pandemic. Yet, it is important to also examine the usability of these sudden adoptions or utilizations of telehealth technologies. Usability refers to a product’s ability to be used by specified users to achieve specific goals such as efficiency, effectiveness, and satisfaction within a particular context of use, which can provide evaluation and overall improved effectiveness of both the technology and delivered services.25 Telehealth’s usability is an important issue for many patients particularly the elderly and those with chronic conditions, such as cardiovascular conditions, who may need to use telehealth services more frequently.26 It is imperative that the operational interface and end-user experience are easy to use for those who will require more access to these platforms, especially in times of a pandemic when sudden adjustments to virtual care may be the only option for nonemergency patients.27 Therefore, the objective of this paper is to report on health professionals’ telehealth usability in a COVID-19 hotspot area.

METHODS

This study was a cross-sectional survey disseminated online using Qualtrics™.

Study and Instrument Design

The Telehealth Usability Questionnaire (TUQ)28 was implemented as part of the survey. Final survey questions included a modified version of the TUQ, including adjusted language from patient-focused to provider-focused. Five experts in healthcare reviewed and approved the modified survey for face validity. The TUQ is designed to evaluate the usability of telehealth implementation and services, including 21 questions covering six subdomains: usefulness (three questions), ease of use & learnability (three questions), interface quality (four questions), interaction quality (four questions), reliability (three questions), and satisfaction and future use (four questions). Each question is assessed by the participants on a seven-point Likert scale (1 = strongly disagree and 7 = strongly agree), in which a higher score consistently indicates a more positive answer to each question. The overall composite score is calculated as an arithmetic mean of all questions with nonmissing answers. The composite score for each subdomain is calculated in the same fashion using the questions from the specific domain. At the beginning of the survey, participants gave anonymous consent to use their responses for research purposes. This study was deemed exempt by the Institutional Review Board (IRB) at the researchers’ University.

Participants

Health professionals licensed in Florida were invited to participate in this study. Participation in this study was voluntary, there were no cash incentives for completing this survey. The following healthcare professionals were selected to complete the survey: medical doctor, osteopathic physician, advanced practice registered nurse, registered nurse, physician assistant, physical therapist, occupational therapist, registered dietitian, psychologist, licensed mental health counselor, and licensed clinical social worker. Specific professions were selected based on the availability of the Florida Healthcare Practitioner Data Portal.29 A total of 399,660 surveys were disseminated via email in June 2020, including an initial and one follow-up email. Participants completed a demographic profile including primary profession, years as a licensed provider, gender, age, ethnicity, and geographic location of the practice.

Data Analysis

All data summaries and statistical analyses were performed using the SAS software version 9.4 (SAS Institute, Cary, NC).30 In the analysis, the nine individual professions were categorized into five groups based on the practice area: physician and nurses (medical doctor, nurse, and physician assistant), rehabilitation (physical therapist and occupational therapist), mental health (licensed mental health counselor and licensed psychologist), social work (social w1orker), and dietetics (registered dietitian).

The analysis was conducted for the overall TUQ score first, and then for the subdomain scores. The overall score was analyzed using a multiple linear regression model; the scores from six subdomains were analyzed using a linear mixed model. In both models, profession group, gender, age, and the change of telehealth usage during the COVID-19 pandemic were included as fixed covariates. In the subdomain analysis, the correlations between residual errors from the same participants on different subdomains were accounted for by the specified covariance structure. The Kenward and Roger method will be used for the calculation of the denominator degrees of freedom.31,32

RESULTS

Descriptive Statistics

A total of 1,868 healthcare professionals are included in the sample. Table 1 summarizes the demographic information including profession group, gender, years of experience, and the change of telehealth usage during the pandemic. In all the survey participants, 78% are female healthcare professionals, indicating a higher likelihood for females to use telehealth compared with their male counterparts. A total of 1,722 healthcare professionals were included in the analysis due to missing values in the response variables.

Table 1. Descriptive statistics: gender, years of experience, and telehealth usage by health profession, N = 1,868
Physician and nurses Rehabilitation Mental Health Social work Dietetics
Gender, n = 1,868
Female 75.8 (943) 81.1 (133) 80.1 (213) 83.3 (105) 95.7 (67)
Male 24.2 (301) 18.9 (31) 19.9 (51) 16.7 (21) 4.3 (3)
Experience as a licensed provider, n = 1,863
0–5 years 9.8 (122) 14.6 (24) 18.9 (50) 12.7 (16) 17.1 (12)
6–10 years 11.1 (138) 3.0 (5) 17.0 (45) 9.5 (12) 7.1 (5)
11–15 years 13.0 (162) 7.3 (12) 16.3 (43) 9.5 (12) 18.6 (13)
16–20 years 12.1 (150) 14.6 (24) 14.8 (39) 14.3 (18) 10.0 (7)
21+ years 53.8 (669) 59.1 (97) 33.0 (87) 54.0 (68) 47.1 (33)
Telehealth usage change during pandemic, n = 1,837
Decreased 3.6 (45) 1.8 (3) 2.7 (7) 0.8 (1) 2.9 (2)
No change 54.7 (680) 65.9 (108) 27.3 (72) 31.0 (39) 42.9 (30)
Increased 40.0 (498) 29.3 (48) 70.1 (185) 66.7 (84) 50.0 (35)

Multivariate Analysis

For the overall TUQ score, profession group, gender, age, and the change of telehealth usage during the COVID-19 pandemic all had a significant impact (Table 2). Specifically, younger and female healthcare professionals participating in the survey had a significantly higher overall TUQ score versus elder and male participants. In addition, individuals who reported to have increased usage during the pandemic also had a significantly higher score. The differences in the overall TUQ scores among different profession groups were also significant. Compared with the score from physician and nurses (estimated average = 4.21), the scores from the mental health group (4.78) and social work group (4.62) were significantly higher and the score from the dietetics group (4.39) was higher but not significant, while the score rehabilitation group (3.93) was significantly lower.

Table 2. Estimated overall Telehealth Usage Questionnaire (TUQ) scores by health profession, gender, and telehealth usage
Estimate (95% CI)
Profession
Physician and nurses 4.21 (4.10, 4.32)
Rehabilitation 3.98 (3.73, 4.13)
Mental health 4.78 (4.61, 4.94)
Social work 4.62 (4.40, 4.85)
Dietetics 4.39 (4.11, 4.68)
Gender
Male 4.32 (4.16, 4.48)
Female 4.45 (4.33, 4.57)
Usage
My use of telehealth decreased 4.05 (3.75, 4.35)
My use of telehealth has not changed 4.27 (4.17, 4.38)
My use of telehealth increased 4.83 (4.73, 4.93)

The scores from all six subdomains demonstrated a similar pattern to the overall score across the profession groups from the highest to the lowest, indicating the reliability and consistency of the questionnaire (Table 3). It is worth mentioning that the absolute scores were varying from different subdomains. Specifically, usefulness, ease of use and learnability, satisfaction, and future use domains received the highest scores, and reliability domain received the lowest score, while interface quality and interaction quality domains were in the middle for each profession group. It is also estimated that the female participants scored significantly higher in three of the six subdomains: interface quality, interaction quality, and reliability.

Table 3. Estimated Telehealth Usage Questionnaire subdomain scores by health profession
Physician and nurses Rehabilitation Mental health Social work Dietetics
Estimate (95% CI) Estimate (95% CI) Estimate (95% CI) Estimate (95% CI) Estimate (95% CI)
Usefulness 4.57 (4.48, 4.65) 4.22 (4.01, 4.43) 5.13 (4.97, 5.29) 4.96 (4.73, 5.19) 4.79 (4.48, 5.10)
Ease of use & learnability 4.48 (4.39, 4.56) 4.09 (3.88, 4.29) 5.16 (4.99, 5.32) 4.95 (4.72, 5.18) 4.73 (4.42, 5.04)
Interface quality 4.17 (4.08, 4.25) 3.93 (3.72, 4.14) 4.67 (4.51, 4.84) 4.59 (4.36, 4.82) 4.40 (4.09, 4.71)
Interaction quality 4.20 (4.11, 4.28) 3.91 (3.70, 4.12) 4.65 (4.49, 4.82) 4.54 (4.30, 4.77) 4.17 (3.86, 4.48)
Reliability 3.54 (3.45, 3.62) 3.35 (3.14, 3.56) 3.79 (3.63, 3.95) 3.67 (3.44, 3.91) 3.60 (3.28, 3.91)
Satisfaction and future use 4.32 (4.24, 4.41) 4.07 (3.86, 4.28) 5.23 (5.07, 5.40) 4.99 (4.76, 5.22) 4.70 (4.38, 5.01)

DISCUSSION

The findings from this study indicate that the health professionals’ telehealth usability in Florida was significantly related to health professional categories, age, gender, and the change of telehealth usage during the COVID-19 pandemic. Compare with physicians and nurses, mental health professionals and social workers reported higher telehealth usability, while rehabilitation clinicians reported much lower usability.

Mental health professionals in the sample, including clinical mental health counselors and psychologists, evidenced the highest TUQ scores. These professions have a lengthy history utilizing telehealth modalities, with discussions of best practices around such beginning in earnest in the 1990s.32–34 As such, professionals in these fields are likely to receive at least some training in telehealth practice.35,36 For example, the Council for Accreditation of Counseling and Related Education Programs (CACREP) requires counseling programs to include training on “technology-assisted [counseling] relationships”.37 Technology-assisted relationships, including telehealth, are therefore not anomalous for most mental health professionals.38–40 This may be why a recent study by McClellan and Florell41 noted that 89% of mental health clinicians in their sample had favorable or neutral attitudes toward telehealth. The results of the current study add helpful context to this, showing that mental health professionals have significantly more favorable views of telehealth than other medical professionals.

Mental health professionals also evidenced higher scores on all TUQ subdomains. The highest subdomain scores were in relation to usefulness and intention to use in the future. The typical tasks of mental health clinicians almost exclusively require less hands-on contact than other professionals, and therefore, may find the telehealth platform less hindering to their normal work duties. Mental health counselors and psychologists providing primarily psychotherapy can, with notably few exceptions, provide the same services via telehealth platforms that they can in face-to-face settings.42 This fact may more easily assuage any concerns these clinicians have about the usability of telehealth platforms.

Services provided by physicians, nurses, and rehabilitation clinicians require a lot of hands-on examinations, and it would be harder for these groups to practice completely virtually. However, with better telehealth tools, these health providers are able to address this gap in care delivery. Examples of such tools found to be conductive to improve care delivery include the use virtual tools that allow for both communication and data sharing among health providers and between health provider and patient,43 and for health organizations to partner with local communities to create a community telehealth network to better serve rural and vulnerable populations.44 Furthermore, there is a need to ensure that telecommunication infrastructures continue to grow as the demand for telehealth services increases.45 Nevertheless, both hospitals and healthcare providers have created innovative alternatives during COVID-19. For example, individual specialties and subspecialties developed virtual examination protocols and having patients proceed through sets of physical activities and manipulating readily available props in the home.46 Studies have shown that clinicians can successfully use several elements of a standard musculoskeletal assessment to evaluate patients with mild low back pain47 and knee pain48 in countries where telehealth is widely available. Studies also indicate that a systematic virtual examination can aid in triaging and managing common musculoskeletal conditions.49 These best practice protocols are still evolving with more use of telehealth services.

Worth noting, psychiatrists, who are trained physicians with an additional psychiatry rotation, were coded as physicians in the present study. Their work, which is largely hands-off medication consultations, can be completed more easily via telehealth modalities than many of their physician colleagues. Telepsychiatry has proliferated over the past decade,50 especially during the COVID-19 pandemic.38,51 As such, psychiatrists may represent a bit of an anomaly within the physician community, evidencing the flexibility and benefits that the other mental health professionals in the study did.

Younger and female healthcare professionals participating in this study showed significantly higher telehealth usability, which is consistent with the literature that telemedicine use among physicians was more prominent among younger doctors. Physicians 45 or younger reported they were practicing some form of telehealth at a higher rate than physicians 46 or older.52 Of course, this trend may change due to the abrupt adaptations needed from the COVID-19 pandemic. In April 2020, already nearly half of physicians using telehealth, up from just 18% in 2018.52 At the same time, the findings on the female healthcare professionals are corresponding with the reports that women and younger adults are more likely to use telehealth services.53 Motivation strategies for older and male healthcare professionals to use telehealth services should be something the healthcare leaders to consider in the future.

Lastly, the healthcare professionals who reported to have increased usage during the pandemic showed significantly higher telehealth usability. This is understandable since practice makes perfect. This is also consistent with previous studies that healthcare providers engaged in more telehealth activities in the realms of telemedicine and there was a significant shift in their perceptions, indicating greater openness and willingness to adopt telehealth services.54 The COVID-19 pandemic forced an array of healthcare providers to quickly adapt and find ways to provide their usual services by using online tools. However, we should also realize the challenges related to telehealth services regarding efficiency, safety, quality, outcomes, as well as the adequacy of existing regulations from both the federal and state governments.

Connolly, Miller55 highlighted the relationship between acceptance of telehealth and the use of telehealth. In light of the COVID-19 pandemic, and especially the subsequent need for a rapid transition to telehealth, it is prudent for training programs in all health professions to consider including at least some training in telehealth, following the model set by the mental health fields, to promote acceptance of the modality of care. Pandemics represent only one possible impetus for the healthcare system to swiftly switch to telehealth platforms; each profession should consider providing their trainees with the best possible chance of successfully making such a switch. The remaining concerns among all healthcare professions surround the platform’s technical stability.

LIMITATIONS

This study has several limitations to consider. Healthcare professionals were contacted through their registered emails with the Florida Department of Health; likewise, healthcare facilities may have blocked these emails containing external URLs, with concerns of spam or cybercrimes. Additionally, a 1% response rate was less than preferred. This sample may not be generalizable to the entire 400,000+ healthcare professionals practicing in Florida. Lastly, this study was a cross-sectional study, only capturing data from one point in time, thus not capturing the changes across the span of the COVID-19 pandemic. Future research should evaluate how health professional’s telehealth usability changes over time.

CONCLUSIONS

The United States healthcare system has faced incredible demands for both in-person care and adaptations to virtual care. Educators and policymakers must consider the importance of the education and training of different modalities of care (e.g., telehealth). Telehealth can provide easy access to high-quality care regardless of geography, work, or personal constraints, but healthcare professionals need to be provided the tools for success. It is important for policymakers and health administrators to view telehealth as a vital tool in future pandemic planning and response.56

REFERENCES

  1. Li H, Liu S-M, Yu X-H, Tang S-L, Tang C-K. Coronavirus disease 2019 (COVID-19): Current status and future perspective. Int J Antimicrob Agents. 2020;55(5):105951. https://doi.org/10.1016/j.ijantimicag.2020.105951
  2. World Health Organization. WHO Director-General’s opening remarks at the media briefing on COVID-19-11 March 2020. 2020. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020. Accessed 7 October 2021.
  3. Gostin LO, Wiley LF. Governmental public health powers during the COVID-19 pandemic: Stay-at-home orders, business closures, and travel restrictions. JAMA. 2020;323(21):2137–8. https://doi.org/10.1001/jama.2020.5460
  4. Sen-Crowe B, Sutherland M, McKenney M, Elkbuli A. The Florida COVID-19 mystery: Lessons to be learned. Am J Emerg Med. 2020; 46: 661–3. https://doi.org/10.1016/j.ajem.2020.08.009. Accessed 7 October 2021
  5. Fu SJ, George EL, Maggio PM, Hawn M, Nazerali R. The consequences of delaying elective surgery: Surgical perspective. Ann Surg. 2020;272(2):e79–e80. https://doi.org/10.1097/SLA.0000000000003998
  6. Elster E, Potter BK, Chung K. Response to COVID-19 by the surgical community. Surgery. 2020;167(6):907–908. https://doi.org/10.1016/j.surg.2020.03.011
  7. Donley G, Chen B, Borrero S. The legal and medical necessity of abortion care amid the COVID-19 pandemic. J Law Biosci. 2020;7(1):Isaa013. https://doi.org/10.1093/jlb/lsaa013
  8. Bloem BR, Dorsey ER, Okun MS. The coronavirus disease 2019 crisis as catalyst for telemedicine for chronic neurological disorders. JAMA Neurol. 2020;77(8):927–928. https://doi.org/10.1001/jamaneurol.2020.1452
  9. Stachura ME, Bolch EB, Piette KF, Ebberwein JF. Chronic condition management using remote monitoring and telehomecare. Chronic illness and long-term care: Breakthroughs in research and practice: IGI global; 2019. p. 669–86. https://doi.org/10.4018/978-1-5225-7122-3.ch033
  10. Lu JF, Chi MJ, Chen CM. Advocacy of home telehealth care among consumers with chronic conditions. J Clin Nurs. 2014;23(5–6):811–19. https://doi.org/10.1111/jocn.12156
  11. Mehrotra A, Jena AB, Busch AB, Souza J, Uscher-Pines L, Landon BE. Utilization of telemedicine among rural Medicare beneficiaries. JAMA. 2016;315(18):2015–16. https://doi.org/10.1001/jama.2016.2186
  12. Nouri S, Khoong EC, Lyles CR, Karliner L. Addressing equity in telemedicine for chronic disease management during the Covid-19 pandemic. NEJM Catalyst Innov Care Deliv. 2020;1(3). https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0123. Accessed 7 October 2021.
  13. Ward MM, Merchant KA, Carter KD, Zhu X, Ullrich F, Wittrock A, et al. Use of telemedicine for ED physician coverage in critical access hospitals increased after CMS policy clarification. Health Aff. 2018;37(12):2037–44. https://doi.org/10.1377/hlthaff.2018.05103
  14. The Agency For Health Care A. Florida report on telehealth utilization and accessibility tallahassee [Internet]. FL: The Agency For Health Care Administration; 2016. Available from: https://b.ahca.myflorida.com/Publications/docs/Telehealth_Report_Final_12-28-16.pdf
  15. Robinson KT, Bergeron CD, Mingo CA, Meng L, Ahn S, Towne Jr SD, et al. Factors associated with pain frequency among adults with chronic conditions. J Pain Symp Manag. 2017;54(5):619–27. https://doi.org/10.1016/j.jpainsymman.2017.07.043
  16. United Health Foundation. Explore multiple chronic conditions—Ages 65+ in Florida | 2020 Senior Health: United Health Foundation [Internet]; 2019. Available from: https://www.americashealthrankings.org/explore/senior/measure/mult_chronic_conditions_sr_a/state/FL
  17. Orozco-Beltran D, Sanchez-Molla M, Sanchez JJ, Mira JJ, ValCronic Research G. Telemedicine in primary care for patients with chronic conditions: The valCronic quasi-experimental study. J Med Internet Res. 2017;19(12):e400. https://doi.org/10.2196/jmir.7677
  18. Moore MA, Coffman M, Jetty A, Klink K, Petterson S, Bazemore A. Family physicians report considerable interest in, but limited use of, telehealth services. J Am Board Fam Med. 2017;30(3):320–30. https://doi.org/10.3122/jabfm.2017.03.160201
  19. Ensina LA, Lee HD, Takaki WSR, Maciejewski NAR, Spolaôr N, Wu FC. Heuristics-based responsiveness evaluation of a telemedicine computational web system. IEEE Latin Am Trans. 2019;17(03):444–52. https://doi.org/10.1109/TLA.2019.8863315
  20. Nord G, Rising KL, Band RA, Carr BG, Hollander JE. On-demand synchronous audio video telemedicine visits are cost effective. Am J Emerg Med. 2019;37(5):890–4. https://doi.org/10.1016/j.ajem.2018.08.017
  21. U.S. Bureau of Labor Statistics. Economic news release: Employment situation summary. 2020. https://www.bls.gov/news.release/empsit.nr0.htm. Accessed 7 October 2021.
  22. Perrin PB, Pierce BS, Elliott TR. COVID-19 and telemedicine: A revolution in healthcare delivery is at hand. Health Sci Rep. 2020;3(2):e166. https://doi.org/10.1002/hsr2.166
  23. Gadzinski AJ, Gore JL, Ellimoottil C, Odisho AY, Watts KL. Implementing telemedicine in response to the COVID-19 pandemic. Philadelphia, PA: Wolters Kluwer; 2020.
  24. Miner H, Fatehi A, Ring D, Reichenberg JS. Clinician telemedicine perceptions during the COVID-19 pandemic. Telemed e-Health. 2020;27(5):508–12. https://doi.org/10.1089/tmj.2020.0295
  25. Abran A, Khelifi A, Suryn W, Seffah A. Usability meanings and interpretations in ISO standards. Softw Qual J. 2003;11(4):325–38. https://doi.org/10.1023/A:1025869312943
  26. Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems—A literature survey. Int J Med Inform. 2016;93:57–69. https://doi.org/10.1016/j.ijmedinf.2016.06.004
  27. Narasimha S, Agnisarman S, Chalil Madathil K, Gramopadhye A, McElligott JT. Designing home-based telemedicine systems for the geriatric population: An empirical study. Telemed J E Health. 2018;24(2):94–110. https://doi.org/10.1089/tmj.2017.0047
  28. Parmanto B, Lewis Jr AN, Graham KM, Bertolet MH. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabilitation. 2016;8(1):3. https://doi.org/10.5195/ijt.2016.6196
  29. Florida Health. Health care practitioner data portal 2020 [Internet]. Available from: https://mqa-internet.doh.state.fl.us/downloadnet/Licensure.aspx
  30. SAS Institute. Base SAS 9.4. Cary, NC: SAS Institute Inc; 2013.
  31. Kenward MG, Roger JH. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics. 1997;53:983–97. https://doi.org/10.2307/2533558
  32. Harper SG. Counseling and the internet. Couns Hum Dev. 1999;32(1):1.
  33. Nickelson DW. Telehealth and the evolving health care system: Strategic opportunities for professional psychology. Prof Psychol Res Pract. 1998;29(6):527. https://doi.org/10.1037/0735-7028.29.6.527
  34. Stamm BH. Clinical applications of telehealth in mental health care. Prof Psychol Res Pract. 1998;29(6):536. https://doi.org/10.1037/0735-7028.29.6.536
  35. Dopp AR, Wolkowicz NR, Mapes AR, Feldner MT. Implementation of telepsychology services in a university training clinic: Process and lessons learned. Behav Ther. 2017.
  36. McKay JC, Vásquez ÁR, Chipp CL, Ford TJ, Robinson RV, Sharma DKB, et al. Divided by miles, connected by purpose: Telepsychology technology in a graduate training program. Train Educ Prof Psychol. 2013;7(3):166. https://doi.org/10.1037/a0033568
  37. Council for Accreditation of Counseling and Related Educational Programs. CACREP Standards 2015 [Internet]. Available from: https://www.cacrep.org/
  38. Ramalho R, Adiukwu F, Gashi Bytyci D, El Hayek S, Gonzalez-Diaz JM, Larnaout A, et al. Telepsychiatry and healthcare access inequities during the COVID-19 pandemic. Asian J Psychiatr. 2020;53:102234. https://doi.org/10.1016/j.ajp.2020.102234
  39. Hicks-Roof KK, Xu J, Zeglin RJ, Bailey CE, Hamadi HY, Osborne R. Covid-19 impacts on Florida’s healthcare professionals. Hosp Top. 2021:1–12. https://doi.org/10.1080/00185868.2021.1927277
  40. McCord CE, Saenz JJ, Armstrong TW, Elliott TR. Training the next generation of counseling psychologists in the practice of telepsychology. Couns Psychol Q. 2015;28(3):324–44. https://doi.org/10.1080/09515070.2015.1053433
  41. McClellan MJ, Florell D, Palmer J, Kidder C. Clinician telehealth attitudes in a rural community mental health center setting. J Rural Ment Health. 2020;44(1):62. https://doi.org/10.1037/rmh0000127
  42. Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: A 2013 review. Telemed e-Health. 2013;19(6):444–54. https://doi.org/10.1089/tmj.2013.0075
  43. Dinesen B, Nonnecke B, Lindeman D, Toft E, Kidholm K, Jethwani K, et al. Personalized telehealth in the future: A global research agenda. J Med Internet Res. 2016;18(3):e53. https://doi.org/10.2196/jmir.5257
  44. Lokken TG, Blegen RN, Hoff MD, Demaerschalk BM. Overview for implementation of telemedicine services in a large integrated multispecialty health care system. Telemed e-Health. 2020;26(4):382–7. https://doi.org/10.1089/tmj.2019.0079
  45. Hong Y-R, Lawrence J, Williams Jr D, Mainous III A. Population-level interest and telehealth capacity of US hospitals in response to COVID-19: Cross-sectional analysis of Google search and national hospital survey data. JMIR Public Health Surveill. 2020;6(2):e18961. https://doi.org/10.2196/18961
  46. Grundstein MJ, Sandhu HS, Cioppa-Mosca J. Pivoting to telehealth: The HSS experience, value gained, and lessons learned. HSS Journal®. 2020;16(1_Suppl):1–6. https://doi.org/10.1007/s11420-020-09788-y
  47. Truter P, Russell T, Fary R. The validity of physical therapy assessment of low back pain via telerehabilitation in a clinical setting. Telemed e-Health. 2014;20(2):161–7. https://doi.org/10.1089/tmj.2013.0088
  48. Richardson BR, Truter P, Blumke R, Russell TG. Physiotherapy assessment and diagnosis of musculoskeletal disorders of the knee via telerehabilitation. J Telemed Telecare. 2017;23(1):88–95. https://doi.org/10.1177/1357633X15627237
  49. Tanaka MJ, Oh LS, Martin SD, Berkson EM. Telemedicine in the era of COVID-19: The virtual orthopaedic examination. J Bone Joint Surg Am. 2020;102(12):e57. https://doi.org/10.2106/JBJS.20.00609
  50. Spivak S, Spivak A, Cullen B, Meuchel J, Johnston D, Chernow R, et al. Telepsychiatry use in US mental health facilities, 2010–2017. Psychiatr Serv. 2020;71(2):121–7. https://doi.org/10.1176/appi.ps.201900261
  51. Kannarkat JT, Smith NN, McLeod-Bryant SA. Mobilization of telepsychiatry in response to COVID-19—Moving toward 21 st century access to care. Admin Policy Ment Health Ment Health Serv Res. 2020;47(4):489–91. https://doi.org/10.1007/s10488-020-01044-z
  52. Eddy N. Nearly half of physicians using telehealth, up from just 18% in 2018: Healthcare IT News [Internet]. 2020. Available from: https://www.healthcareitnews.com/news/nearly-half-physicians-using-telehealth-just-18-2018
  53. Reed ME, Huang J, Graetz I, Lee C, Muelly E, Kennedy C, et al. Patient characteristics associated with choosing a telemedicine visit vs office visit with the same primary care clinicians. JAMA Network Open. 2020;3(6):e205873-e. https://doi.org/10.1001/jamanetworkopen.2020.5873
  54. Helou S, El Helou E, Abou-Khalil V, Wakim J, El Helou J, Daher A, et al. The effect of the COVID-19 pandemic on physicians’ use and perception of telehealth: The case of Lebanon. Int J Environ Res Public Health. 2020;17(13):4866. https://doi.org/10.3390/ijerph17134866
  55. Connolly SL, Miller CJ, Lindsay JA, Bauer MS. A systematic review of providers’ attitudes toward telemental health via videoconferencing. Clin Psychol Sci Pract. 2020;27(2):e12311. https://doi.org/10.1111/cpsp.12311
  56. Koonin LM, Hoots B, Tsang CA, Leroy Z, Farris K, Jolly B, et al. Trends in the use of telehealth during the emergence of the COVID-19 pandemic—United States, January–March 2020. Morb Mortal Wkly Rep. 2020;69(43):1595. https://doi.org/10.15585/mmwr.mm6943a3