Adolescent idiopathic scoliosis (AIS) is an abnormal curvature with three-dimensional (3D) deformation of the human spine that occurs in adolescents without any known reason. Bracing, as the most common nonsurgical treatment, is prescribed to halt or mitigate curvature progression and avoid surgery. Three common types of braces, including full-time rigid, night-time rigid, and soft braces, have been developed to date [1-7]. One of the most important challenges in the AIS brace treatment is to fit the brace on the patient's trunk and adjust the in-brace correction pressure, which has a direct impact on the brace's effectiveness. The brace is currently adjusted passively by regulating the tightness of the brace's strap using follow-up and X-ray checks with trial and error on the patient. Although only a few robotic braces that use typical motion control and force control strategies, for example, PID, have been developed [8-11], the advantages of active closed-loop control methods have not been fully exploited in the AIS brace treatment yet. Impedance control (IC) has been widely implemented in robotic rehabilitation [12-16], but IC has not been implemented into the AIS brace treatment yet. The IC is a better solution than typical motion or force control for human–robot interaction (HRI) control, where the robot has direct contact with the human subject as the environment, because it regulates the dynamic relationship between motion and interaction force to provide smooth physical HRI in contrast to typical motion control and force control strategies that control only motion and force independently. In our previous works [17, 18], a robotic brace was developed, and novel IC strategies were proposed to control the biomechanical interaction in the AIS brace treatment. However, the major issue with the typical IC is how to define the desired impedance parameters. What is the desired impedance model for the AIS brace treatment?

On the other hand, reinforcement learning (RL) is a widely studied topic in learning and control communities. The concept of RL policy, that is, taking the states of the environment (observations) and generating actions to perform a task in an optimal manner, is similar to the role of a controller in a control system. In RL, the first step is to determine the environment, which can be a simulated environment or the physical one. The environment in RL includes the plant, for example, the robot, its surrounding workplace, for example, the human body, and any other elements outside the RL agent. The observations for the RL agent are defined using sensory measurements. The action variables can be the control input, for example, the actuator torques. The action can also be any other variables, for example, the reference position that is used as the desired input to the low-level controller. The next step is to define the reward function. Since the RL algorithm is trying to take the best actions for performing the task in an optimal manner by maximizing the reward function, the reward function should be defined such that it will be increased if the task is performing well, and the desired goal is approaching. After that, a policy should be chosen to enable the agent to map the observations into the optimal actions by maximizing the reward function. Finally, a training algorithm is defined to train the policy and find the optimal solutions. Sutton and Barto [19] presented a good introduction to RL, and the specific RL problems in robotics were described [20-22]. Chatzilygeroudis et al. [23] presented recent advancements in RL, with a focus on data-efficient algorithms, and deep learning advanced solutions for RL problems were reviewed by Arulkumaran et al. [24]. In brief, RL is an interesting topic for control engineers because a typical control problem is mapped to an RL problem without any need for concerns about designing the control law and tuning the controller gains. Since there is no research work concerning learning-based active control in the AIS brace treatment, the RL-based IC is an open research topic for interaction control of the AIS brace treatment so that the robotic brace can learn the desired impedance parameters required to provide the desired physical HRI for the AIS brace treatment. The RL agent learns from the observations obtained from trial and error by applying good or bad actions to the environment. However, in the AIS brace treatment, training the RL agent using trial and error on a human subject may cause serious challenges, e.g., injuries in the human torso and new spinal deformities because of the wrong actions that the RL agent might take. Besides, the long-time process and harmful radiation exposure due to repeated radiographic examinations are the other disadvantages of trial and error with human subjects. Therefore, computational modeling can be adopted to develop a simulated environment for training the RL agent without trial and error on human subjects. However, the output of the computational model might be different from its physical counterpart because of the model uncertainty. Therefore, the significant issue to be addressed is how to improve the computational model and increase the model's reliability to mirror the biomechanical behavior of the AIS brace treatment. On the other hand, digital twin (DT) is an emerging technology that entered our lives in the field of production and engineering, and it is also going to lead to a revolution in healthcare [25]. A DT is a digital replica of a physical system (PS) that is updated using data from the PS and allows simulation and prediction of the state of the system in a virtual environment. The concept of DT was first implemented in the NASA Apollo 13 program in the 1960s [26, 27]. The first documented definition of DT was provided by Michael Grieves in his presentation in the context of product life cycle management in 2003 and later in a white paper [28]. A comprehensive classification of the research articles in the field of DT was presented in a recently published two-part review paper [29, 30]. It is indicated that the DT has been implemented in different application domains, including manufacturing, mechanics, civil, structure, aerospace, energy, and healthcare. Although the review paper [29] shows a leading role of manufacturing in the DT field, healthcare has the least number of published papers in DT. An automatic gait data control system for fully actuated lower limb exoskeleton digital twining was proposed by improving the integration of DT with the medical rehabilitation field and analyzing the patient's gait data through a simulation experiment [31]. A robot-assisted telerehabilitation system with integrated industrial internet of things (IIOT) and DT was developed for post-stroke patients with upper limb dysfunction [32]. Although some attempts were made to develop DT in robotic rehabilitation, there are no research articles focusing on creating a DT for the AIS brace treatment. Therefore, developing a DT of the AIS brace treatment will overcome the above-mentioned limitations and it can be used for training the RL agent in a virtual environment.

In brief, various DT models, including 3D and 5D, have been proposed in the past decade [28, 29, 33]. Grieves [28] proposed a 3D DT composed of a PS, a digital system (DS), and the connections between them. A 5D DT that consists of PS, DS, a service component, DT data, and connections was proposed by Tao et al. [33]. A new 5D DT of a lithium-ion cell composed of PS, DS, physical to virtual (P2V), virtual to physical (V2P), and optimization was proposed in the two-part review paper [29, 30] and this is adopted to develop a 5D three-layer DT of the AIS brace treatment in this paper. Besides, a RL-based position-based impedance control (RLPIC) algorithm is proposed for active closed-loop interaction control of the AIS brace treatment. The RL agent is trained using the 5D three-layer DT. The P2V data flow is established by estimating the unknown parameters of the 5D three-layer DT using a Neural Network (NN)-based regressor according to the data set collected from real-time experiments. Using the NN-based regression model, the time needed for manually tuning the contact parameters is reduced. Furthermore, to establish a V2P data flow, the desired impedance parameters for a given desired displacement are predicted using the RLPIC trained by the 5D three-layer DT. The RLPIC can improve the safety and compliance of the patient by learning the desired HRI for AIS treatment in the simulated environment (DT). The developed DT can also potentially improve the brace adjustment process without trial and error on the human subject. Note that, in this paper, the real-time experiments refer to the practical tests that are performed on the PS. Figure 1 presents the main contribution of this paper.

The rest of the paper is organized as follows: The 5D three-layer DT is created in Section 2. The proposed RLPIC algorithm is formulated in Section 3. Section 4 presents the numerical simulations and real-time experiments. Section 5 concludes the paper.

A DT of a PS mimics the behavior of its physical counterpart in a virtual environment. In this paper, the DT technology is exploited to improve and optimize the brace treatment of AIS by modeling and predicting its biomechanical behavior in advance. First, an overview of our preliminary works [17, 18] on the modeling of the AIS brace treatment is explained in Section 2.1. Second, the 5D three-layer DT model of the AIS brace treatment is presented in Section 2.2. It consists of five dimensions, including the PS, its three-layer digital model, P2V data flow, V2P data flow, and optimization. The physical model consists of the robotic brace developed in our previous work [17], the human torso, and the physical HRI. The three-layer digital model is created by adding the solid body model of the human torso to the multi body (MB) model of the robotic brace and modeling the HRI as a mass–spring–damper system. To develop a P2V data flow, the unknown parameters of the HRI model are identified by training anNN-based regressor using data sets from real-time experiments. The identified values of the unknown parameters are optimized to develop the fifth dimension. The V2P data flow is established by training the RLPIC algorithm proposed in Section 3 using the 5D three-layer DT to predict the desired impedance model required for AIS treatment. The NN-based regression model used for identifying the unknown parameters of the digital model is finally presented in Section 2.3.

First, an overview of our previous works [17, 18] on the HRI control of the AIS brace treatment is presented. Two novel controllers from the IC family, including model reference adaptive impedance control (MRAIC) [17] and novel position-based impedance control (NPIC) [18], were proposed. In MRAIC, an adaptive law was designed to regulate the impedance parameters such that the actual impedance model follows the reference impedance model. In NPIC, the actual impedance parameters are estimated using interaction force–motion measurements and the impedance parameters' errors were then used to regulate the measured interaction force, which is the primary input of the PIC, such that the desired impedance model is achieved.

In this section, an RLPIC algorithm is proposed to improve the performance of the typical PIC in terms of pose tracking, velocity tracking, and HRI control. The schematic diagram of the RLPIC is shown in Figure 3. First, the typical PIC algorithm is described. In PIC, the virtual pose of the robot is computed using the desired impedance model and the measured interaction force. An internal motion control loop is then utilized to provide the virtual pose so that the desired pose is achieved in the presence of the physical HRI. The desired impedance model is considered as follows:where , , and are diagonal matrices and denote the desired mass, damping, and stiffness parameters, respectively. Note that the desired impedance parameters of each DOF are defined independently and the correlations between the impedance parameters of different DOFs are assumed to be zero [34]. This is why the desired impedance matrices , , and have a diagonal form with the size of ., , and are vectors representing the pose error, velocity error, and acceleration error, respectively. and denote the desired pose and actual pose vectors of the robotic brace, respectively. The actual pose of the robotic brace is defined as , where denote the position vector of the moving ring with respect to the world frame. Besides, represents the orientation of the moving ring expressed in the screw representation form with respect to the world frame. It should also be mentioned that the -, -, and -directions are defined along the normal vector to the coronal, sagittal, and transverse planes, respectively. Note that the interaction force vector includes both contact forces and contact torques. The virtual pose of the robotic brace required to achieve the desired pose in the presence of the measured interaction force is computed by rewriting Equation (4) as follows:where and represent the desired velocity and acceleration of the robotic brace. and also denote the virtual velocity and acceleration of the robotic brace, respectively. The inverse dynamic control (IDC) is then utilized for the pose control of the robotic brace, while the virtual pose is applied as a reference input to the IDC. Therefore, the control law of the IDC is formulated as follows:where and are diagonal matrices and denote the proportional and derivative gains of the IDC. , , and denote the mass matrix, the Coriolis and the centrifugal matrix, and the gravity vector of the robotic brace, respectively [17, 18]. represents the actual velocity of the robotic brace and is also an vector representing the projection of the actuator forces in the task space.

The main issue in implementing IC into the AIS brace treatment lies in finding a way to determine the desired impedance parameters. In other words, finding the desired impedance parameters required for providing the desired HRI for the AIS brace treatment is challenging. Although there is no research article focusing on implementing IC into the AIS brace treatment, the importance of the in-brace correction pressure/force (interaction force ) has been widely studied in the literature [35-38]. In the AIS brace treatment, it is crucial for orthotists to know how much interaction force is exerted on the patient's torso by the brace, because the interaction force has a direct impact on the AIS curvature correction and the effectiveness of the AIS brace treatment. The interaction force reported by the above-mentioned in vivo studies has an extensive range, and it is still unclear [35-38]. On the other hand, the impedance parameters directly impact the interaction force and motion variables. This is why determining the desired impedance parameters required for providing the desired interaction force and motion for the AIS brace treatment is challenging. In this section, an RL-based approach is proposed to predict the desired impedance parameters of the PIC ( and ) according to the interaction force and motion measurements. For this purpose, the environment, actions, observations, the reward function, the RL algorithm, and the architecture of the NN used for training the RL algorithm are defined. The robotic brace, human torso, and the PIC approach presented in Equations (4)–(6) are assumed to be the environment for the RL agent, as shown in Figure 3. Note that the 5D three-layer DT is used to train the RL agent because the RL agent learns from both good and bad actions and training the RL algorithm using the human subject may cause unexpected motion/forces and consequently injuries on the patient's torso. Therefore, instead of using the PS as the environment, the 5D three-layer DT along with the PIC is assumed to be the environment for the RL agent. The desired stiffness and damping are chosen as the action variables to allow the RL agent to control the physical HRI. Predicting the desired impedance parameters by the RL agent while following the desired pose will allow the robot to provide the desired HRI. The observation space consists of the robot's pose , desired pose , velocity , desired velocity , and the interaction force .

The design of the reward function plays a vital role in learning a task using an RL agent. The main goal of the reward function is to quantify the goodness of a selected action, and it is used for updating the current policy to take the best action in the next time step. Design of the reward function depends on the type of the task and it should be defined such that the robot gets rewards to incentivize such a behavior. In this paper, the reward function is defined such that the robotic brace follows the desired trajectory , keeps tracking a reference velocity in the task space , and avoids exerting a large interaction force on the patient's torso. The reward function is defined as follows: [39]where , , and are column vectors denoting the pose reward, the interaction force reward, and the velocity reward in 18 DOFs, respectively. The weights , , and are row vectors representing the weights of the pose, the interaction force, and velocity rewards in 18 DOFs, respectively. The weights are tuned manually according to the importance of the corresponding reward. The reward terms for DOF are defined as follows:where , , and denote the error coefficients of the corresponding reward term. represents the upper boundary for the interaction force to ensure safe physical HRI. The error coefficients should be chosen such that each error has reasonable sensitivity [39].

The proximal policy optimization (PPO) algorithm is chosen as the RL algorithm [40, 41]. It is based on an actor–critic structure, combined with the trust region method. The architecture of the actor's NN consists of two fully connected layers of neurons with as the activation function, followed by a fully connected layer with a size of one neuron and as the activation function, which is in parallel with one fully connected layer with one neuron. The architecture of the NN for the critic in the PPO algorithm consists of two fully connected layers with 256 neurons. The activation function for each layer is . A third fully connected layer is also used to map the output of the previous layer into proper dimensions.

In Section 4.1, the 5D three-layer DT is validated through numerical simulations and real-time experiments. The proposed RLPIC is trained and verified using the 5D three-layer DT in Section 4.2. The workflow of this section is shown in Figure 4. First, real-time experiments are conducted, and the interaction force–displacement measurements are collected. The same displacement is then applied as the primary input to the 5D three-layer DT. The NN regression model proposed in Section 2.3 is trained using data collected from real-time experiments and the unknown parameters of the “Spatial Contact Force“ block are estimated. The estimated unknown parameters are finally optimized using the “Parameter Estimator” App from the Simulink Design Optimization Toolbox such that the output interaction force obtained from numerical simulations fits the interaction force collected through real-time experiments. Second, the RL agent is trained using the 5D three-layer DT and the RLPIC is verified in terms of position tracking, velocity tracking, and HRI control.

AIS brace treatment has not been integrated with DT technology and learning-based HRI control approaches. This paper proposed an RLPIC algorithm that enables the robotic brace to learn the desired HRI required for AIS rehabilitation. Besides, a 5D three-layer DT of the AIS brace treatment is developed to be used for training the RLPIC algorithm to avoid trial an error on human subjects. The 5D three-layer DT includes the PS, a digital model, bidirectional data flow (P2V and V2P), and optimization. The three-layer digital model is created by adding the MB model of the robotic brace to the solid body model of the human torso, while the physical HRI between the robotic brace and the torso is modeled as a mass–spring–damper system using the “Spatial Contact Force” block. Real-time experiments and numerical simulations are carried out to update the unknown parameters of the HRI model, establish the P2V data flow, and validate the DT. An NN-based regression model is proposed to estimate the unknown stiffness and damping parameters of the physical HRI model using the interaction force–motion data collected from the real-time experiments. The estimated parameters are optimized using the ‘Parameter Estimator’ App from the Simulink Design Optimization Toolbox such that the interaction force obtained from the real-time experiments fits the interaction force obtained from the 5D three-layer DT. Finally, the RLPIC algorithm trained using the DT model is utilized to predict the desired HRI and establish the V2P data flow such that the robotic brace follows a desired trajectory, maintains the interaction force to less than a maximum value, and tracks the desired velocity.

Since the RLPIC algorithm is trained using the DT model, the more accurate the digital model, the more reliable the RLPIC algorithm is in the clinical rehabilitation therapy. On the other hand, the ML algorithms used for identifying the unknown parameters of the DT model improve the digital model accuracy and twin the digital model with its physical counterpart. In other words, ML algorithms improve the reliability of the DT model and consequently can increase the reliability and effectiveness of the RLPIC algorithm in the clinical rehabilitation therapy. Besides, ML algorithms can be useful in updating the unknown parameters of the DT model such that the DT model adapts to the biomechanical changes in the patient's torso that occur during the period of growth spurt.

Although an RLPIC approach is proposed and a 5D three-layer DT is created for training the RL agent, further improvements are needed. For example, more real-time experiments with different types of motion need to be performed to collect a huge source of the data set that can be used for unknown parameter identification, DT validation, and training the RL agent for different variants of the environment. In addition, a more complete version of mass–spring–damper models, for example, nonlinear mass–spring–damper models, can also be used for HRI modeling as an interesting subject for future work. However, this represents preliminary research work in implementing RLPIC and DT modeling into the AIS brace treatment and no other research work has focused on implementing RLPIC and DT modeling into the AIS brace treatment. This is why the simplest mass–spring–damper model, that is, the linear model with constant coefficients, is utilized in this paper for HRI modeling. Besides, the DT model can also improve the brace design process and brace adjustments by predicting the biomechanical behavior of the brace performance in advance without any need for trial and error on human subjects. It can also play an important role in actual AIS rehabilitation by training the RLPIC algorithm, which has the potential to be used in the actual AIS rehabilitation to provide the desired HRI required for AIS treatment. Furthermore, improving the RL algorithm used in the RLPIC algorithm by adopting advanced RL-based approaches currently used in robotic applications will be an interesting topic for future work. Note that the trained RLPIC algorithm has not been validated through real-time experiments yet because of the lack of human subjects and limited time. As ongoing work, we will conduct more real-time experiments to collect more data and validate the trained RLPIC algorithm.