Objective To explore a possible solution in clinical practice of fluid therapy for patients with sepsis by reinforcement learning method. Methods A total of 11 913 patients with sepsis were screened by using the Medical Information Mark for Intensive Care (MIMIC) Ⅲ Database, and randomly divided into a training set and a test set according to the ratio of 8:2. Twenty-six features were used in modeling, including 24 state features of patients (bloc, vital signs, laboratory tests, blood gas analysis index and basic information), 1 action feature (liquid inflow and outflow difference) and 1 outcome feature (outcome in ICU). Data rules of SARSA model learning training set were used to get the relationship between return rewards and mortality, so as to evaluate whether the return rewards were reasonably set. Deep Q-learning (DQN), a deep learning model based on Q-learning,models the relationship between the state and behavior of the test set, predicts the patients' fluid balance, and compares the results of reinforcement learning and the actual outcomes of patients, which further proved the different effects of predicted liquid therapy and actual therapy on prognosis. Results According to the behavior category distribution, the differences of liquid inflow and outflow were divided into 5 intervals (–3000 to –239.40 ml, –239.39 to –1.94 ml, –1.93 to 160.00 ml, 160.01 to 363.58 ml, and 363.59 to 3000 ml). The SARSA model calculated the training data set, results showed that the higher the Q (s, a) return, the lower the mortality rate. The DQN model suggested that both too high and too low of the difference between the liquid input and output volume may increase the case mortality, and the mortality of patients is higher in low difference of inflow and outflow than in high difference of inflow and outflow volume. Using Doubly robust estimator to evaluate the DQN model average expected return of the test set showed the stability of the model (Q-learning iteration number >20 000). The use of validation set hinted that the mortality was obvious lower in the subgroups predicted dehydration consistent with the reality than in the other three subgroups, indicating that the model can be used in actual clinical verification. Conclusion A predictive model for possibly guiding the fluid therapy on patients with sepsis is proposed using the reinforcement learning method, which can accurately predict the direction of liquid therapy,patients got a better prognosis by using the model predicted dehydration treatment and dehydration was actually carried out.
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |