ORIGINAL CLINICAL RESEARCH

Predictive Models to Optimize Resources in Tele-Critical Care in Distributed Hospital Networks

Rafael Palacios, PhD^1,2, Eugenio Sánchez-Úbeda, PhD¹, Ralph Panos, MD³, Peniel Argaw, MS⁴, Malika Shahrawat, MEng⁵, Daniel D. Zhang, BS⁶, Angelina Zhang, BS², Adam Seiver, MD, PhD, MBA⁷, Omar Badawi, PharmD, MPH, FCCM⁸ and Amar Gupta, PhD, MBA, FIEEE²

¹Institute for Research in Technology, Universidad Pontificia Comillas, Madrid, Spain; ²Computer Science & Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA; ³Tele-ICU Program, Cincinnati VA Medical Center, Cincinnati, Ohio, USA; ⁴John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts, USA; ⁵Blue Apron, New York, New York, USA; ⁶Princess Máxima Center for Pediatric Oncology, Utrecht, Netherlands; ⁷Emergency Care and Hospital Respiratory Care, Philips, San Francisco, California, USA; ⁸Independent Consultant, Washington, DC, USA

Keywords: eICU, eICU collaborative research database, electronic ICU, intensive care, length of stay prediction, mortality prediction, resource management, tele-critical care, telemedicine

Abstract

Background: Telemedicine created the opportunity—in pandemic conditions and otherwise—to spread healthcare to regions where intensivists and health services might not be available. In addition, it offers the opportunity to provide better patient care, decrease healthcare costs, and improve population health, overall.

Introduction: The deployment of critical care telemedicine has increased due to its impact on providing care at all times of the day, as well as reaching remote regions of the world. Tele-critical care (Tele-CC) systems can provide concurrent service to several hospitals and can manage available resources more efficiently than traditional intensive care units (ICUs).

Materials and Methods: This study utilizes the Philips eICU system and its collaborative research database (eICU-CRD) to evaluate intensive care operations in the electronic ICU setting, with the objective of analyzing where and how system engineering techniques can be potentially applied to enhance the effectiveness of such environments.

Results: Several metrics are evaluated, including patient outcomes, APACHE (Acute Physiology and Chronic Health Evaluation) score, length of stay (LOcS), and type of unit with regard to the age of the patient. Prediction models based on decision and regression trees are presented to estimate mortality and LOS.

Discussion: Prediction models offer the potential to optimize the Tele-CC environment by helping estimate the number of patients who will remain in the ICU during the following days.

Conclusion: Prediction models accurately estimate mortality and LOS in the ICU. The estimation of the future number of patients can be used to determine the resources needed at each hospital, as well as to provide insight into potential savings when Tele-CC centers provide concurrent services to multiple hospitals.

 

 

Telemedicine includes the use of telecommunication and information technology to provide clinical healthcare services to patients from a distance away from the bedside.¹ In particular, it can improve the quality of healthcare by providing comprehensive clinical support that overcomes geographic barriers at a significantly decreased cost.² The importance of telemedicine is particularly salient during the coronavirus disease 2019 (COVID-19) pandemic, where social distancing measures have forced healthcare providers to reconsider how to treat patients, enhance follow-up care, and improve patient access to these services.^3,4 Telemedicine has been very successful in pediatric consultation⁵ reducing wait times and increasing physician efficiency while retaining high satisfaction and quality of care. The advantages of telemedicine, already identified around 2018, were highlighted during the COVID-19 pandemic.^6,7

The introduction of intensive care unit (ICU) remote monitoring programs has allowed for dynamic task relocation and leveraged a second layer of care—where medical professionals can perform further evaluations, and order and interpret tests, beyond just emergency cases.^8,9 Tele-critical care (Tele-CC) is currently the most common name for systems that use cameras, video monitors, smart alarms, and other technology to connect clinical staff with patient data almost immediately and can be applied to critical care in ICU or nonICU units.¹⁰ This implementation of telemedicine in ICUs provides significant potential to improve patient outcomes.^11,12 By providing care to patients at a distance from medical specialists, Tele-CCs are especially appropriate in situations where hospitals are understaffed or where medical experts may be at risk for contagious diseases such as COVID-19.^13–15

However, to improve the performance of Tele-CC in situations where the Tele-CCs are required to assist a large number of patients concurrently, new performance evaluation models must be developed and analyzed.¹⁶ For evaluating Tele-CCs, one can potentially apply the performance evaluation metrics used for regular ICU, including patient outcomes, risk scores, readmission rates, time of admittance, and time spent with patients.^17,18

The Philips eICU Collaborative Research Database (eICU-CRD) can be used to evaluate the performance of intensive care in the electronic ICU setting. The term eICU stands for Electronic Intensive Care Unit and mimics the notation of e-mail for electronic mail. Over time, the term eICU gradually faded and was replaced by Tele-CC, which today is the term used most commonly. eICU-CRD is the largest easily and freely accessible critical care database licensed by MIT (Massachusetts Institute of Technology) and supported by the NIH (National Institutes of Health) in the United States.^19,20 The evaluation of the performance of eICUs helps determine the advantages and disadvantages of eICU care and can provide insights that can lead to better use of eICU technology and process in critical care. Furthermore, the analysis of these parameters facilitates the development of models to predict mortality and length of stay (LOS) of the patients that provide the key parameters for intelligent resource planning.

While analyzing the performance of medical centers and factors that determine their effectiveness in providing quality care, some studies have focused on specific procedures and tools, while ignoring factors related to the delivery of care and the work processes.²¹ The analysis and the interpretation of the latter category of factors can help provide better quality of care, reduce operation costs, and provide optimal resource allocation.²² By reducing costs and optimizing resources, more budget is available to upgrade medical equipment, renovate facilities, or even reallocate staff or hire additional staff to improve overwhelmed departments. According to the American Telemedicine Association,²³ the benefits of cost efficiencies due to telehealth are especially remarkable for better management of chronic diseases, in addition to reducing travel times and minimizing hospital stays. Among the relatively few studies that have investigated the impact of Tele-CC on hospital outcomes, some have concluded that the addition of Tele-CC leads to a reduction in ICU mortality and ICU LOS.^11,12 A positive effect was also found in overall hospital mortality and hospital LOS when the patient was treated in Tele-CC.²⁴ Another study that compared the performance of an ICU with and without the addition of the eICU found that telemedicine adoption was associated with a small reduction in mortality.²⁵ The effectiveness over multiple hospitals showed wide variations, but 16 case hospitals experienced a statistically significant reduction in mortality with the adoption of telemedicine.

More studies have assessed the effectiveness of traditional ICUs. Typically, the assessment of the performance of a particular ICU involves evaluating its performance against itself over time using various metrics quantified to measure performance.²⁶ Traditional ICU performance metrics include patient outcomes, risk scores, readmission rates, time of admittance, and time spent with patients. The risk of mortality can be in the form of an Acute Physiologic Assessment and Chronic Health Evaluation (APACHE) score, which is a quantitative measure of the severity of the disease of critically ill patients.²⁷ Such scoring systems help monitor the quality of care by standardizing along similar illness severity. Features such as APACHE score, readmission rate, LOS, and mortality rates are used to perform statistical and regressional analyses.^17,18,24,28 The measurement of performance over separate ICU units and over time can lead to interesting results.^29,30 In order to secure a more holistic view, some studies have analyzed one or more hospitals overall.^17,18,25,28 Some studies use both ICU LOS and ICU mortality in addition to hospital LOS and hospital mortality to better understand the effect of the ICU stay.^17,18 The mean LOS was usually found to be skewed toward the left, meaning that a higher proportion of patients were discharged from the ICU early. Furthermore, APACHE score and LOS are correlated with mortality, and a high occupancy rate in the ICU adversely affected the mortality rate. Overall, no consistent major changes were detected in the ICU performance and outcomes based on the standardized mortality ratio and LOS ratio.³⁰ Another study investigated possible differences in clinical outcomes based on days of the week and times of admission since ICUs and Tele-CC are typically staffed differently at different times. Through analysis of variables such as mean LOS, bed turnover, occupancy rate, and turnover interval, it was found that although mortality rates are highest between 10 pm and 2 am, it does not vary significantly across different times of the day or across different days of the week. Admissions during the off hours (non-regular office hours) were also not related to worse clinical outcomes, including differences in mortality or LOS in the ICU.³¹ Yet, another study explored evaluating mortality-based critical care performance by categorizing patients based on their illness severity levels and then performing a logistic regression to predict risk-standardized mortality. Little correlation was found between the performances across risk categories of patients, meaning that hospitals that perform well with high-severity patients do not always perform well in caring for lower-severity patients.³² Other studies have used machine learning methods to predict the LOS and mortality of patients.³³

While the above types of metrics relate heavily to patient outcomes, the impact on the families and friends of the critically ill patient and the hospital and its staff also need more analysis.³⁴ The patient’s post-ICU and hospital discharge quality of health can be revealing to the effect of the stay as well.³⁵ However, it is more difficult to quantify these parameters as the respective data would be required. Moreover, the evaluations conducted using the commonly used metrics should be interpreted in the context of longer-term outcomes.³⁴ By utilizing parameters such as APACHE score, age, unit type of mortality, and LOS of patients, the analysis performed in this research can provide accurate estimates of Tele-CC mortality and Tele-CC LOS. Ultimately, the use of these measures in the evaluation of Tele-CC can determine the resources needed at specific hospitals, as well as optimize resource distributions in hospital networks.

Materials and Methods

The eICU-CRD²⁰ was used to evaluate the performance of intensive care in the eICU and patient outcomes. Specifically, the “patient” and the “APACHE patient result” tables were selected for use in this study. The datasets provided by Philips involved 200,859 eICU unit stays (or events to analyze) in 208 different hospitals. The tables of patients and APACHE results were linked by means of the patient_unit_stay_ID in order to select events in which the APACHE score was registered. There are 166,355 unique hospital stays, in which a patient may have been transferred from one unit to another recording the APACHE score at the beginning of each unit stay. The ethnicity of the patients and the location of the hospitals that participated in the study were not provided in the dataset.

It is important to distinguish between unit stay and hospital stay for the analysis. A patient may be discharged alive from one unit and later expire during the same hospital stay. In addition, the LOS at a unit could be short, but the hospital LOS might be much longer. The main focus of this research was on the analysis of unit mortality and unit LOS rather than hospital mortality or hospital LOS because the prediction algorithms were developed with the purpose of helping to manage eICU resources and to enhance effectiveness.

In analyzing data, the value of the APACHE score is of major importance; it is recorded after admission at the eICU unit. In the database, there are cases in which the APACHE score was not recorded or contains an unknown value.¹ Those unit stays without a valid APACHE score were filtered out for this study.

Results

In total, there were 146,696 unit stays with valid APACHE scores, comprising 139,377 unique hospital stays. Statistics relating to the APACHE score and the LOS derived from the cleansed dataset are summarized in Table 1.

Table 1. APACHE score and length of stay from APACHE-Patient-Result table linked to the patient table

Metric	Median	Mean	Minimum	Maximum
APACHE score	51.0	55.5	0	211
Length of stay (days)	1.8	3.0	0.2	346

Table 2 and Table 3 show the final cohort used from the eICU CRD database, specified by the type of unit and ranges of age, respectively. The final cohort contains 146,687 patient stays and is divided into two tables depending on whether the patient was discharged alive from the unit or the patient expired. Table 2 shows the data corresponding to patients discharged alive, and Table 3 shows the same type of data for patients who expired. In both tables, the main information displayed is the number of patients, the mean APACHE score, and the mean LOS. The total and the details for type of unit are shown in different rows. The columns display different age ranges and also the total for all ages.

Table 2. Summary of features for living patients, sorted by type of unit and age (in years)

ALIVE	Ranges of Age
	[0,10]	[10,20]	[20,30]	[30,40]	[40,50]	[50,60]	[60,70]	[70,80]	[80,90]	[90,100]	Total/Average
Number of Patients
Cardiac ICU		48	317	483	921	1819	2093	1872	1332	353	9238
CCU-CTICU		46	280	456	1039	2253	3021	2859	1707	337	11998
CSICU		7	95	131	280	676	1095	1227	678	109	4298
CTICU		10	116	168	384	992	1422	1215	727	92	5126
Med-Surg lCU	14	742	3729	4671	7233	13584	16281	15481	10989	2736	75460
MICU	1	142	525	722	1120	2197	2673	2399	1757	428	11964
Neurc ICU		105	636	826	1174	2100	2319	2046	1418	303	10927
SICU		112	433	520	845	1783	2076	1933	1184	294	9180
Total	15	1212	6131	7977	12996	25404	30980	29032	19792	4652	138191
Mean APACHE score
Cardiac ICU		39.7	41.1	39.0	41.8	44.8	53.1	58.7	64.4	69.1	52.5
CCU-CTICU		43.8	42 3	38.9	40.6	43.1	51.7	58.2	61.8	63.7	51.7
CSICU		27.6	36.9	40.3	44.4	47.5	57.5	61.7	62.7	66.6	56.3
CTICU		31.2	43.8	46.3	43.5	46.5	54.6	59.0	61.9	67.3	54.0
Med-Surg lCU	38.3	36.7	39.7	39.9	43.9	47.3	55.1	60.1	64.1	68.6	53.6
MICU	49.0	41.1	42.2	43.8	48.5	51.1	60.0	65.0	67.5	69.5	57.7
Neuro ICU		36.1	35.0	32.5	35.4	40.0	47.7	53.8	57.7	60.0	45.7
SICU		44.1	41.6	40.5	43.8	46.2	53.0	58.4	63.3	67.1	52.4
Average	39.0	38.1	39.8	39.6	43.1	46.4	54.4	59.7	63.6	67.7	53.1
Mean Length of Stay (days)
Cardiac ICU		4.4	2.7	3.1	3.1	3.2	3.2	3.1	2.9	2.5	3.1
CCU-CTICU		2.1	3.6	2.7	2.9	2.8	3.0	3.1	2.8	2.6	2.9
CSICU		0.8	2.0	1.8	2.6	2.5	2.7	2.6	2.3	2.0	2.5
CTICU		6.4	3.0	3.5	3.4	3.0	3.2	3.3	3.0	2.3	3.2
Med-Surg lCU	1.5	2.2	2.2	2.4	2.6	2.9	2.9	2.9	2.7	2.4	2.7
MICU	0.8	2.9	2.4	3.1	3.2	3.3	3.5	3.3	2.9	2.2	3.2
Neuro ICU		2.8	3.4	3.4	3.5	3.6	3.4	3.4	2.9	2.3	3.3
SICU		3.1	3.5	3.4	3.7	3.4	3.6	3.5	3.1	2.6	3.4
Average	1.5	2.5	2.5	2.7	2.9	3 0	3.1	3.0	2.8	2.4	2.9
CCU-CTICU: Cardiovascular Intensive Care Unit-Cardiothoracic Intensive Care Unit; CSICU: Cardiac Surgery Intensive Care Unit; CTICU: Cardiothoracic Intensive Care Unit; ICU: Intensive Care Unit; SICU: Surgical Intensive Care Unit.

Table 3. Summary of features for expired patients, sorted by type of unit and age (in years)

Expired	Ranges of Age
	[0,10]	[10,20]	[20,30]	[30,40]	[40,50]	[50,60]	[60,70]	[70,80]	[80,90]	[90,100]	Total/Average
Number of Patients
Cardiac ICU		1	4	15	42	118	184	195	131	30	720
CCU-CTI ICU		1	12	13	30	67	161	173	142	24	623
CSICU			2	4	8	31	66	83	56	9	259
CTICU			1	9	14	24	45	37	31	8	169
Med-Surg lCU		15	87	147	264	688	1033	1179	983	292	4688
MICU		1	15	38	52	159	243	244	217	48	1017
Neurc ICU		5	19	22	44	73	106	137	75	26	507
SICU		5	15	16	36	79	109	109	99	45	513
Total		28	155	264	490	1239	1947	2157	1734	482	8496
Mean APACHE score
Cardiac ICU		106	96	76	95	98	95	102	100	94	98
CCU-CTICU		149	95	104	87	96	90	97	89	89	93
CSICU			97	111	105	102	99	102	93	113	100
CTICU			138	107	77	95	95	87	98	89	93
Med-Surg lCU		99	106	96	95	93	96	96	92	94	95
MICU		135	90	90	101	99	95	98	98	90	97
Neuro ICU		108	98	95	88	77	88	87	87	86	87
SICU		98	105	104	90	89	89	97	92	92	93
Average		104	102	95	94	93	94	97	93	93	95
Mean Length of Stay (days)
Cardiac ICU		8.2	4.8	5.8	6.0	5.9	5.2	4.7	3.5	2.2	4.8
CCU-CTICU		4.5	4.2	5.4	6.6	3.7	5.0	4.3	3.3	3.2	4.3
CSICU			4.2	1.2	3.5	4.8	6.4	3.9	4.0	3.1	4.6
CTICU			0.9	4.4	10.9	6.5	5.8	6.8	4.4	3.8	6.1
Med-Surg lCU		6.6	5.0	3.9	5.1	4.8	4.4	4.0	3.5	2.4	4.1
MICU		1.8	6.7	4.9	4.3	4.7	4.9	4.5	3.6	1.7	4.3
Neuro ICU		1.0	5.2	2.7	4.0	4.7	4.3	3.8	2.8	2.7	3.8
SICU		1.1	4.2	3.6	6.3	7.3	4.8	6.0	4.3	3.1	5.2
Average		4.4	5.0	4.1	5.3	5.0	4.7	4.3	3.5	2.5	4.3
CCU-CTICU: Cardiovascular Intensive Care Unit-Cardiothoracic Intensive Care Unit; CSICU: Cardiac Surgery Intensive Care Unit; CTICU: Cardiothoracic Intensive Care Unit; ICU: Intensive Care Unit; SICU: Surgical Intensive Care Unit.

Most patients are between 50 and 85 years old, with almost no patients younger than 16 years of age. All patients older than 89 years were assigned the value 99 regardless of the actual age. This may create some strange effects in the tail of the age-related graphs, which will be shown later.

Based on the above information regarding the mortality of patients and the LOS in the ICU, statistical analysis was used as the first step for data visualization and understanding of data, as well as for identifying correlation and interaction of variables. Then, several supervised machine learning techniques were used to develop prediction models to estimate the likely outcome for the patients.

Mortality results

A total of 146,696 patient stays were evaluated in this study. From this set, 138,200 (94.2%) patients were alive at ICU discharge, and 8,496 (5.8%) patients had passed away. However, there were 5,065 patients who expired in the hospital after ICU discharge, thereby raising the total mortality rate for patients who were monitored by an eICU to 13,561 (9.2%) as shown in Figure 1.

Fig. 1 Mortality statistics: eICU (left) versus hospital (right). eICU: electronic Intensive Care Unit; ICU: Intensive Care Unit.

In order to evaluate the causes of mortality in greater detail, the database was analyzed from a statistical perspective to identify which parameters are linked with mortality. Next, a prediction algorithm was created to help classify patients at admission time according to their risk condition, therefore providing valuable information for resource planning.

Mortality and APACHE scores

The risk of mortality can be in the form of an APACHE score, a quantitative measure of the severity of disease of critically ill patients standardized along similar illnesses calculated at the beginning of ICU admission. It has been popular in ICU since the introduction of APACHE II in 1985,³⁶ which generates a score from 0 to 71 based on 12 physiologic variables: temperature, mean arterial pressure, heart rate, respiratory rate, oxygenation, arterial pH, serum Na⁺, serum K⁺, serum creatinine (SCr), hematocrit (Hct), white blood cell count (WBC), and Glasgow Coma Scale (GCS). Subsequently, extended and more complex versions were developed: APACHE III was introduced in 1991, and APACHE IV in 2006.²⁷ In addition, other systems were developed to predict patients’ outcomes, such as the Simplified Acute Physiology Score (SAPS II)³⁷ and Mortality Prediction Model (MPM).³⁸ Intensive Care Network publishes a webpage to facilitate calculation of APACHE IV score: https://intensivecarenetwork.com/Calculators/Files/Apache4.html.

For Apache scores in the range of 0 and 150, a positive correlation is observed between APACHE score and mortality, as shown in Figure 2, Left. For most scores above 150, the average mortality rate is 100%.

Fig. 2 Average mortality for each APACHE Score (Left). Percentage change in mortality for each point increase in APACHE Score (Right).

When examining the percentage change in mortality for each point increase in APACHE score (Figure 2, Right), the predictive power of APACHE score is most evident within 150 point score change, with every point increase in APACHE score corresponding to a respective 0.3% increase in mortality. Beyond a difference of 150, a sharp drop is observed, which can be attributed to the unpredictability of cases with large score differences.

Mortality and type of ICU unit

For patients who were discharged alive from ICU but later died in the hospital, the distribution by unit type shows in Figure 3.

Fig. 3 Distribution by unit type for patients who expired in the hospital after discharge from eICU. CCU-CTICU: Cardiovascular Intensive Care Unit-Cardiothoracic Intensive Care Unit; CSICU: Cardiac Surgery Intensive Care Unit; CTICU: Cardiothoracic Intensive Care Unit; ICU: Intensive Care Unit; SICU: Surgical Intensive Care Unit.

It can be seen that Med-Surg ICU (Medical-Surgical ICU) accounts for the largest number of fatalities, while in the case of other types of ICU units, the patients recovered without much complications after being discharged from ICU.

Mortality and patient age

As expected, mortality shows correlation with the age of the patient. The median age of patients who survived is lower than that for patients who expired (Figure 4). Figure 4 also shows that mortality is rare for ages under 40 but reaches nearly 50% for patients who are 90 years of age. Patients older than 90 years old are represented in the database grouped together with the value 100, and surprisingly, the mortality rate for this group is lower than that for the group from 80 to 90 years of age.

Fig. 4 Hospital mortality distribution by age in years.

Length of stay

The LOS in the ICU, stored in variable actual-eICU-LOS, shows a saw shape (see Figure 5) because the values tend to be rounded to complete days, which is consistent with the fact that hospitals decide to discharge or retain patients at fixed times of the day. The most typical value for Tele-CC LOS is 2 days since the area in Figure 5 from 1 to 2 days is larger than from 0 to 1 day.

Fig. 5 Distribution of length of stay at eICU (in days). eICU: electronic Intensive Care Unit.

Length of stay and APACHE scores

When examining the LOS versus APACHE score, the relationship between these two variables for alive patients is strongly positive (Figure 6, Left), meaning that higher APACHE scores lead to longer lengths of stay. Patients with APACHE scores higher than 130 are more likely to expire, so those who survive and were released alive from Tele-CC (Left graph) show lengths of stay are scattered instead of showing a positive correlation.

Fig. 6 Correlation between length of stay and APACHE scores. eICU: electronic Intensive Care Unit.

However, for patients who expired, a strong negative association exists between LOS and the APACHE score. As the APACHE score increases, the LOS may decrease as patients with more severe conditions may expire sooner (Figure 6, Right).

APACHE score represents the severity of the patient’s condition. The results on mortality and LOS are somehow expected. In the previous section, a strong correlation was found to predict mortality; however, the graphs in Figure 6 show that estimating LOS is challenging and cannot be done just using the APACHE score. More sophisticated techniques are needed for LOS prediction and will be presented later.

Length of stay and type of ICU unit

General statistics concerning LOS for each ICU units were evaluated to understand the correlation between outcome and type of ICU unit. As shown in Table 4, the total survival rate is 94.2%, with the lowest survival rates corresponding to MICU (Medical ICU) and Cardiac-ICU (Cardiovascular ICU) and highest corresponding to CTICU (Cardiothoracic ICU). CTICU deals with patients recovering from cardiac bypass surgery or other heart-related procedures.

Table 4. Mean length of stay (in days) of eICU patients compared across each type of unit

Type or Unit	Alive	Expired	Survived	Mean LOS (alive)	Mean LOS (expired)
CCU-CTICU	11,998	623	95.1%	2.92	4.27
CSICU	4,299	259	94.3%	2.50	5.58
CTICU	5,127	169	96.8%	3.16	6.06
Cardiac-ICU	9,238	720	92.8%	3.07	4.82
MICU	11,967	1,017	92.2%	3.19	4.33
Med-Surg-ICU	75,464	4,688	94.2%	2.75	4.07
Neuro-ICU	10,927	507	95.6%	3.34	3.83
SICU	9,180	513	94.7%	3.43	5.21
TOTAL	138,200	8496	94.2%	2.92	4.29
CCU-CTICU: Cardiovascular Intensive Care Unit-Cardiothoracic Intensive Care Unit; CSICU: Cardiac Surgery Intensive Care Unit; CTICU: Cardiothoracic Intensive Care Unit; ICU: Intensive Care Unit.

The overall LOS is longer for those patients who expired in the ICU than for those who recovered and were discharged alive. This is true for all types of units although for Neuro-ICU (Neurology ICU), the LOS does not change significantly with the outcome of the patient. Neuro-ICU shows one of largest mean LOS for patients discharged alive, while the smallest mean LOS (less that 4 days) for patients who expired.

The unit with the largest mean LOS is CTICU, which is also the unit with the highest survival rate. The second largest mean LOS is SICU (Surgical ICU), which deals with surgical operations related to trauma, gastrointestinal, renal, and some cardiac surgeries. The unit with the shortest stay is CSICU (Cardiac Surgery ICU), where patients discharged alive had a mean LOS of 2.5 days.

In the analysis by unit type, CSICU shows a lower median LOS compared with the other units (Figure 7). This is consistent with the lower mortality shown earlier in Figure 3.

Fig. 7 Distribution of length of stay grouped by unit type. CCU-CTICU: Cardiovascular Intensive Care Unit-Cardiothoracic Intensive Care Unit; CSICU: Cardiac Surgery Intensive Care Unit; CTICU: Cardiothoracic Intensive Care Unit; ICU: Intensive Care Unit; SICU: Surgical Intensive Care Unit.

Length of stay and patient age

The LOS in Tele-CC also correlates positively with the age of the patient (Figure 8). For patients older than 80 years, the median LOS decreases, but this is a consequence of higher mortality at an older age that limits the LOS. Additionally, the range of days increases with the age up to 80 years, where LOS ranges from 0 to 7.5 days (after removing outliers).

Fig. 8 Statistical representation of length of stay as distributed by age (in years). EICU: Electronic Intensive Care Unit.

Predictive models

The most important parameter in managing Tele-CC is the rate of mortality. Although APACHE IV²⁷ is better than previous versions, APACHE II³⁶ is still popular because of its simplicity in terms of the number of variables needed. All these methods use ICU Day 1 information to predict the probability of hospital death, and this value is not recalculated during the stay of the patient. APACHE IV uses a multivariate logistic regression procedure based on about 20 variables (some optional) plus information related with the chronic health condition.

While the APACHE score was designed to estimate hospital mortality, it can also be used to estimate Tele-CC mortality. In this paper, decision trees, a well-known supervised machine learning technique,^39–41 were used to analyze the effectiveness of the APACHE score to predict Tele-CC mortality. Researchers have used Gradient Boosting Decision Trees with the MIMIC database to predict mortality, although in a much shorter prediction window (real-time to 24 h).⁴²

A decision tree was developed using the APACHE score and patient age as inputs. After performing the pruning process in the tree, in order to avoid overtraining, it was found that age was not a relevant input for improving predictions. This is reasonable since age is already one of the inputs used to compute the APACHE IV score, so the effect of age in the likelihood of recovery is already considered. As a consequence, the decision tree analysis involving 146,696 patients revealed that a threshold value of 134.5 is the optimal way to estimate Tele-CC mortality, meaning that if the APACHE score is less than 134.5, the patient is likely to survive within the Tele-CC unit in which the patient was admitted. The overall accuracy of this mortality prediction method is 94.4%, and the majority of the errors (5.2%) correspond to patients with APACHE scores less than 134.5 who were expected to survive, but finally expired. Figure 9 shows the test set accuracy results in terms of the confusion matrix.

Fig. 9 Evaluation of the model to predict Tele-CC mortality. Tele-CC: tele-critical care.

The second most important parameter for resource planning in Tele-CC services is the LOS of the patients. Although the LOS is represented as a real number in the database, usually in days and hours, the pre-analysis exercise revealed that most Tele-CC discharges occur around the same time every day (Figure 5). Therefore, it was decided to discretize LOS into an integer number of days, and to group values of 7 days or more into one single category since a prediction window of 1 week would be sufficiently adequate for planning resources. The distribution of LOS in days is shown in Figure 10, with the graph showing that the most common duration of a stay is 2 days.

Fig. 10 Distribution of length of stay in the number of days after discretized. eICU: electronic Intensive Care Unit.

The time evolution of patients accepted to the eICU is shown in Figure 11, which was generated by analyzing the outcome of more than 200,000 patients. After Day 1, most patients stayed at the ICU, some patients were released (home or hospital), and a few expired. The majority of patients (71.8%) leave the ICU within the first 3 days, and the number of patients who need to stay further than day 5 is relatively small (14.3%). The number of new admissions is fairly constant and not difficult to estimate, and it only depends on factors external to the hospital such as weather, disease outbreaks, etc. Therefore, the most important knowledge for an accurate estimation of the size of the service comes from the estimation of evolution of patients as shown in Figure 11, which mostly depends on several factors directly related with the patient condition (age, unit type, and the APACHE score).

Fig. 11 Evolution of patients accepted to eICU (electronic Intensive Care Unit).

In general, the levels of APACHE score are higher on average for patients who experienced longer stays (Figure 12). While APACHE is a good estimator for LOS, there are many outliers that highlight that additional parameters should be used in the analysis.

Fig. 12 Distribution of APACHE scores for patients with different discretized LOS. eICU LOS: electronic Intensive Care Unit-length of stay.

The prediction model developed to estimate LOS uses APACHE, type of unit, and age as inputs. The regression tree, shown in Figure 13, is able to predict the real value of LOS with good accuracy (as explained in the next paragraph). The tree starts by looking at the APACHE score, and then it focuses on patient age and unit type to narrow the results into the predicted LOS.

Fig. 13 Regression tree for a length of stay estimation in eICU. CSICU: Cardiac Surgery Intensive Care Unit; CTICU: Cardiothoracic Intensive Care Unit; eICU: electronic Intensive Care Unit.

The accuracy of this prediction model can be analyzed by computing the residuals, that is, the difference between the actual LOS and the estimated LOS. This difference is negative if the model predicted an LOS longer that the actual one, and the error is positive if the model predicted a shorter stay. The root mean squared error of the model (RMSE) is just 1.8 days for a range of 6 days look forward. This is fairly accurate, and the results can be improved further if a shorter prediction window is used. A histogram of the residuals is shown in Figure 14.

Fig. 14 Regression tree error for prediction windows of 6 days into the future. LOS: length of stay; RMSE: root mean square error. TS: test set.

Discussion

By providing care to patients at a distance from the medical specialists, Tele-CC is especially appropriate in situations where the bedside is understaffed or lacks experience with specific medical interventions and procedures. To ameliorate the performance of the Tele-CC in situations involving a large number of patients on a concurrent basis, new performance and prediction models must be developed and analyzed, instead of relying on the conventional paradigm that considers the level of resources to be linearly dependent on the number of patients.

This study included the creation of prediction models to better estimate Tele-CC mortality and ICU LOS using parameters such as the APACHE score, age, and unit type. The current gold standard parameter to assess the severity of patient conditions (APACHE score) cannot be used to accurately estimate the LOS, while machine learning techniques have the ability to optimally combine different parameters. The prediction models presented in this paper can estimate mortality with 94% accuracy, and LOS with a mean error of 1.8 days for a range of 6 days looking ahead.

By knowing the current number of patients along with a prediction of their respective lengths of stay, it becomes easier to optimize resources for the short-term and medium-term and to explore the option of sharing resources across hospitals and Tele-CC networks. Future work could include evaluating additional parameters such as readmission rates and length of Tele-CC sessions with the patient, depending on the severity of the illnesses, as well as improving the model’s accuracy for different prediction windows. Furthermore, one could consider broadening the prediction models and analysis to include multiple ICUs within the hospital.

The COVID-19 pandemic has resulted in an unprecedented strain on hospital care systems and exponentially increased the demand for ICUs worldwide. As the pandemic continues to escalate, the task of allocating limited healthcare resources such as beds, staff, equipment, etc. has been a major focus for countries around the world. The importance of Tele-CC program improvements in collaboration with improvements in LOS/mortality prediction can better equip hospitals with the necessary tools for resource sharing and demand issues.⁴³ Importantly, the model developed in this study can be applied to future datasets, particularly COVID-19 specific, to create applicable predictions for the LOS and mortality. Research underway has highlighted the importance of predictive models for symptomatic COVID-19 patients.⁴⁴

Contributors

Adam Seiver is an employee of Philips. Omar Badawi was an employee of Philips when this research was performed.

Rafael Palacios, Eugenio Sánchez-Úbeda, Peniel Argaw, Malika Shahrawat, Daniel D. Zhang, Angelina Zhang, and Amar Gupta worked primarily on the technical part of data analysis and models. Ralph Panos, Adam Seiver, and Omar Badawi worked primarily on the medical aspects discussed in this paper.

Acknowledgments

The authors are grateful to many persons at Philips, Massachusetts of Technology, and other organizations for their suggestions and contributions. This includes Brian Gross of Philips and several students from Massachusetts of Technology and Wellesley College.

References

1. Scalvini S, Vitacca M, Paletta L, Giordano A, Balbi B. Telemedicine: a new frontier for effective healthcare services. Monaldi Arch Chest Dis. 2004;61(4). doi: 10.4081/monaldi.2004.686
2. Rheuban KS. The role of telemedicine in fostering health-care innovations to address problems of access, specialty shortages and changing patient care needs. J Telemed Telecare. 2006;12(2_suppl):45–50. doi: 10.1258/135763306778393171
3. Arneson SL, Tucker SJ, Mercier M, Singh J. Answering the call: Impact of Tele-ICU nurses during the COVID-19 pandemic. Crit Care Nurse. 2020;40(4):25–31. doi: 10.4037/ccn2020126
4. Gamus A, Keren E, Kaufman H, Chodick G. Synchronous video telemedicine in lower extremities ulcers treatment: A real-world data study. Int J Med Inform. 2019;124:31–6. doi: 10.1016/j.ijmedinf.2019.01.009
5. Young K, Gupta A, Palacios R. Impact of telemedicine in pediatric postoperative care. Telemedicine and e-Health. 2019;25(11):1083–9. doi: 10.1089/tmj.2018.0246
6. Uscher-Pines L, McCullough C, Dworsky MS, Sousa J, Predmore Z, Ray K, et al. Use of telehealth across pediatric subspecialties before and during the COVID-19 pandemic. JAMA Netw Open. 2022;5(3):e224759. doi: 10.1001/jamanetworkopen.2022.4759
7. Howie F, Kreofsky BL, Ravi A, Lokken T, Hoff MD, Fang JL. Rapid rise of pediatric telehealth during COVID-19 in a large multispecialty health system. Telemed e-Health. 2022;28(1):93–101. doi: 10.1089/tmj.2020.0562
8. Leong JR, Sirio CA, Rotondi AJ. eICU program favorably affects clinical and economic outcomes. Crit Care. 2005;9(5):E22. doi: 10.1186/cc3814
9. Udeh C, Udeh B, Rahman N, Canfield C, Campbell J, Hata JS. Telemedicine/virtual ICU: Where are we and where are we going? Methodist Debakey Cardiovasc J. 2018;14(2):126–33. doi: 10.14797/mdcj-14-2-126
10. Subramanian S, Pamplin JC, Hravnak M, Hielsberg C, Riker R, Rincon F, et al. Tele-critical care: An update from the society of critical care medicine tele-ICU committee. Crit Care Med. 2020;48(4):553–61. doi: 10.1097/CCM.0000000000004190
11. Young LB, Chan PS, Lu X, Nallamothu BK, Sasson C, Cram PM. Impact of telemedicine intensive care unit coverage on patient outcomes. Arch Intern Med. 2011;171(6):498–506. doi: 10.1001/archinternmed.2011.61
12. Sadaka F, Palagiri A, Trottier S, Deibert W, Gudmestad D, Sommer SE, et al. Telemedicine intervention improves ICU outcomes. Crit Care Res Pract. 2013;2013:456389. doi: 10.1155/2013/456389
13. Shirke MM, Shaikh SA, Harky A. Tele-oncology in the COVID-19 era: The way forward? Trends Cancer. 2020;6(7):547–9. doi: 10.1016/j.trecan.2020.05.013
14. Courtney E, Blackburn D, Reuber M. Neurologists’ perceptions of utilising tele-neurology to practice remotely during the COVID-19 pandemic. Patient Educ Couns. 2021;104(3): 452–9. doi: 10.1016/j.pec.2020.12.027
15. Kapoor A, Guha S, Kanti Das M, Goswami KC, Yadav R. Digital healthcare: The only solution for better healthcare during COVID-19 pandemic? Indian Heart J. 2020;72(2):61–4. doi: 10.1016/j.ihj.2020.04.001
16. AlDossary S, Martin-Khan MG, Bradford NK, Smith AC. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Inform. 2017;97:171–94.
17. Hwang S. Reducing ICU length of stay: the effect of Tele-ICU. On-Line J Nurs Inform. 2014;18(3).
18. Kohl BA, Fortino-Mullen M, Praestgaard A, Hanson CW, DiMartino J, Ochroch EA. The effect of ICU telemedicine on mortality and length of stay. J Telemed Telecare. 2012;18(5):282–6. doi: 10.1258/jtt.2012.120208
19. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet. Circulation. 2000;101(23). doi: 10.1161/01.CIR.101.23.e215
20. Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178. doi: 10.1038/sdata.2018.178
21. Berenson RA, Grossman JM, November EA. Does telemonitoring of patients—The eICU—Improve intensive care? Health Aff. 2009;28(5):w937–47. doi: 10.1377/hlthaff.28.5.w937
22. Bauman KA, Hyzy RC. ICU 2020: Five interventions to revolutionize quality of care in the ICU. J Intensive Care Med. 2014;29(1):13–21. doi: 10.1177/0885066611434399
23. The American Telemedicine Association A. Telehealth basics—ATA [Internet]. Available from: https://www.americantelemed.org/resource/why-telemedicine/ [cited 7 February 2023].
24. Lilly CM, Cody S, Zhao H, Landry K, Baker SP, McIlwaine J, et al. Hospital mortality, length of stay, and preventable complications among critically ill patients before and after Tele-ICU reengineering of critical care processes. JAMA. 2011;305(21):2175. doi: 10.1001/jama.2011.697
25. Kahn JM, Le TQ, Barnato AE, Hravnak M, Kuza CC, Pike F, et al. ICU Telemedicine and critical care mortality: a national effectiveness study. Med Care. 2016;54(3):319–25. doi: 10.1097/MLR.0000000000000485
26. Terblanche M, Adhikari NKJ. The evolution of intensive care unit performance assessment. J Crit Care. 2006;21(1):19–22. doi: 10.1016/j.jcrc.2005.12.003
27. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today’s critically ill patients*. Crit Care Med. 2006;34(5):1297–310. doi: 10.1097/01.CCM.0000215112.84523.F0
28. Collins TA, Robertson MP, Sicoutris CP, Pisa MA, Holena DN, Reilly PM, et al. Telemedicine coverage for post-operative ICU patients. J Telemed Telecare. 2017;23(2):360–4. doi: 10.1177/1357633X16631846
29. Lam KW, Lai KY. Evaluation of outcome and performance of an intensive care unit in Hong Kong by APACHE IV model: 2007–2014. J Emerg Crit Care Med. 2017;1(8):16. doi: 10.21037/jeccm.2017.07.02
30. Afessa B, Keegan MT, Hubmayr RD, Naessens JM, Gajic O, Long KH, et al. Evaluating the performance of an institution using an intensive care unit benchmark. Mayo Clin Proc. 2005;80(2):174–80. doi: 10.4065/80.2.174
31. Anushiravani A, Masoompour SM. Assessing the performance of a medical intensive care unit: A 5-year single-center experience. Indian J Crit Care Med. 2017;21(3):163–6. doi: 10.4103/ijccm.IJCCM_420_16
32. Barbash IJ, Le TQ, Pike F, Barnato AE, Angus DC, Kahn JM. The effect of intensive care unit admission patterns on mortality-based critical care performance measures. Ann Am Thorac Soc. 2016;13(6):877–86. doi: 10.1513/AnnalsATS.201509-645OC
33. Awad A, Bader-El-Den M, McNicholas J. Patient length of stay and mortality prediction: A survey. Health Serv Manag Res. 2017;30(2):105–20. doi: 10.1177/0951484817696212
34. Garland A. Improving the ICU. Chest. 2005;127(6):2151–64. doi: 10.1378/chest.127.6.2151
35. Timmers TK, Verhofstad MH, Moons KG, Leenen LP. Intensive care performance: How should we monitor performance in the future? World J Crit Care Med. 2014;3(4):74. doi: 10.5492/wjccm.v3.i4.74
36. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13(10):818–29. doi: 10.1097/00003246-198510000-00009
37. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270(24):2957–63. doi: 10.1001/jama.1993.03510240069035
38. Lemeshow S, Teres D, Pastides H, Avrunin JS, Steingrub JS. A method for predicting survival and mortality of ICU patients using objectively derived weights. Crit Care Med. 1985;13(7):519–25. doi: 10.1097/00003246-198507000-00001
39. James G, Witten D, Hastie T, Tibshirani R. An introduction to statistical learning [Internet]. New York, NY: Springer New York, 2021; 607 p. (Springer Texts in Statistics). Available from: https://link.springer.com/book/10.1007/978-1-0716-1418-1 [cited 9 January 2023].
40. Hastie T, Tibshirani R, Friedman J. The elements of statistical learning. data mining, inference, and prediction [Internet]. New York, NY: Springer New York; 2009. (Springer Series in Statistics). Available from: http://link.springer.com/10.1007/978-0-387-84858-7 [cited 9 January 2023].
41. Breiman L. Classification and regression trees [Internet]. Routledge; 2017. eBook. Available from: https://www.taylorfrancis.com/books/9781315139470 [cited 9 January 2023].
42. Johnson AEW, Mark RG. Real-time mortality prediction in the Intensive Care Unit. AMIA—Annu Symp Proceed. 2017;2017:994–1003. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29854167 [cited 9 January 2023].
43. Kaur D, Panos RJ, Badawi O, Bapat SS, Wang L, Gupta A. Evaluation of clinician interaction with alerts to enhance performance of the tele-critical care medical environment. Int J Med Inform. 2020;139:104165. doi: 10.1016/j.ijmedinf.2020.104165
44. Wollenstein-Betech S, Cassandras CG, Paschalidis IC. Personalized predictive models for symptomatic COVID-19 patients using basic preconditions: Hospitalizations, mortality, and the need for an ICU or ventilator. medRxiv. 2020;2020.05.03.20089813. doi: 10.1101/2020.05.03.20089813

Copyright Ownership: This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, adapt, enhance this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0.