Haptic Sensing and Feedback Techniques toward Virtual Reality

Haptic Sensing and Feedback Techniques toward Virtual Reality

PDF

Yuxiang Shi¹^,², Guozhen Shen¹^,²^,^*

Research. Vol 7 Article ID 0333

Less

Research. Vol 7 Article ID 0333

• Review Article •

Haptic Sensing and Feedback Techniques toward Virtual Reality

Full

Yuxiang Shi¹^,², Guozhen Shen¹^,²^,^*

Affiliations

¹School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing 100081, China.

²Institute of Flexible Electronics, Beijing Institute of Technology, Beijing 102488, China.

Published: 2024-03-23 doi: 10.34133/research.0333

Outline

Abstract

Less

Haptic interactions between human and machines are essential for information acquisition and object manipulation. In virtual reality (VR) system, the haptic sensing device can gather information to construct virtual elements, while the haptic feedback part can transfer feedbacks to human with virtual tactile sensation. Therefore, exploring high-performance haptic sensing and feedback interface imparts closed-loop haptic interaction to VR system. This review summarizes state-of-the-art VR-related haptic sensing and feedback techniques based on the hardware parts. For the haptic sensor, we focus on mechanism scope (piezoresistive, capacitive, piezoelectric, and triboelectric) and introduce force sensor, gesture translation, and touch identification in the functional view. In terms of the haptic feedbacks, methodologies including mechanical, electrical, and elastic actuators are surveyed. In addition, the interactive application of virtual control, immersive entertainment, and medical rehabilitation is also summarized. The challenges of virtual haptic interactions are given including the accuracy, durability, and technical conflicts of the sensing devices, bottlenecks of various feedbacks, as well as the closed-loop interaction system. Besides, the prospects are outlined in artificial intelligence of things, wise information technology of medicine, and multimedia VR areas.

Cite this Article

Yuxiang Shi, Guozhen Shen. Haptic Sensing and Feedback Techniques toward Virtual Reality[J]. Research, 2024 , 7 (3) : 0333 . DOI: 10.34133/research.0333

Full Text

Less

Introduction

Less

Virtual reality (VR) is a novel technique that can combine input/output data and instructions, including visual, auditory, olfactory, haptic, and gustatory information, into an organic whole. Its real-time responding to outside input from operator and simultaneously adjusting inner output to act on operator make it a closed-loop human machine interaction (HMI) [1]. Haptic interaction plays an important role in VR system for its realistic interactions between operator and the virtual object [2,3], which contributes to more immersive virtual experiences, and is also worthy of applications such as virtual training and virtual controlling.

Haptic sensation is defined as acquiring information in the process of contact or operation via skin tactile system, which is supposed to be the most complex sensation because it relies on the tactile receptors distributed on the whole body. The density of tactile cells under epidermis is about 100/cm² [4], which can detect various kinds of outside stimuli. This complete sensing system is also the perfect to be imitated by artificial tactile interfaces in HMI area.

From technical aspect, haptic interaction mainly contains tactile sensing and haptic feedback. Tactile sensing devices are capable of acquiring touch characteristics via proximity or contact between human hand and object. Tactile sensor has two prime functions of sensing and recognition. The first is to detect the contact conditions including contact or not, touch area, pressure value, pressure distribution, and even multidimensional force [5,6]. Other information, such as the physical characters of object (e.g., smoothness, hardness, texture, and shape) and operation status (e.g., the contact, friction, and slippage), can also be collected by tactile sensors [7–9]. This function can real-time monitor the interaction process between robot hand and operand, which is beneficial for the efficient virtual control or robot teleoperation [10–12]. The second function of tactile sensor is smart identification containing gesture capturing and texture identifications [13–15]. For example, the gesture recognition by tactile sensor array has been applied in extracting hand movement as input instructions in virtual control area, and the captured gesture information are essential data to model virtual object in motion-sensing entertainment. In addition, the texture, stiffness, and material identification are also emerging technologies that have improved the versatility of tactile sensors in teleoperation and VR system.

Haptic feedback, a reverse process of tactile sensing, is to stimulate skin to evoke the tactile receptor and generate tactile sensations via feedback devices just like touching a real object [16–18]. It is an interface aiming to enable bilateral signal communications between human and computer [19,20], and is indispensable for enhancing immersion, interaction, and imagination of VR system. Haptic feedback device has been applied in virtual training, virtual surgery, and dynamic braille, and it is considered as the most important component than virtual vision and audition, because the real-time control and feedback is necessarily dependent on these tasks.

In this review, to address the importance of tactile sensor and haptic feedback shown in Fig. 1, we primarily elaborate their advanced techniques and versatile applications toward HMI in VR areas. We first introduce the sensing mechanisms of human haptics from anatomic aspect with details of various tactile receptors (in the “Fundamentals of Human Haptics” section), which is the guidelines for imitating the advanced skin tactile systems to design multifunctional haptic sensors and gives details of haptic for the feedback device to selected appropriate parameters (e.g., the blocking force, frequency, and displacement) to stimulate skin. Then, in the “Tactile Sensing Devices” section, tactile sensors in VR system are presented with force sensor, gesture recognition, and touch identification. The haptic feedback techniques are recommended in the “Haptic Feedback Techniques” section, covering specific feedback types of mechanical vibration, electrotactile (ET) display, and dielectric elastomer actuator (DEA). Furthermore, the applications of these devices, such as virtual input, force mapping, immersive entertainment, and feedback prosthesis, are summarized in the “Applications” section. We finally review the challenges to develop tactile interfaces in VR system, and the potential opportunities are also expected in the end.

Fundamentals of Human Haptics

Less

Tactile sensation is a synergetic effect of various neuro-system responding to press, temperature, and joint position. It can sense the microcosmic/macroscopic and spatial characteristics of the stimulus. The microcosmic features include material, roughness, and viscosity, while macroscopic characters cover shape, size, and structure and the space information relates to the special position of the stimuli. When object of different characters contacts with skin, skin can generate various encoded tactile information with various receptors under skin.

Tactility is mainly realized by human skin. Skin is not only the first line of defense of human body but also the important organ for brain to sense outside stimuli and make fast response. It is the most direct and fundamental approach for human communicating with the outside world. The structures of skin are shown in Fig. 2 [21]. Human skin consists of epidermis, dermis, and subcutaneous tissue. The fine hair locates across the epidermis and dermis, and roots in the follicles at dermis. Quantities of nerve endings distributed around the fine hair and those with tactile sensing abilities are mechanoreceptors. The mechanoreceptor can generate neuro-signals when stimulating the epidermis and transmit signals to brain neuro-systems. These mechanoreceptors are mainly classified into four categories: Merkel cells, Meissner corpuscles, Ruffini endings, and Pacinian corpuscles [22–25]. Each receptor only shows exclusive response to specific mechanical stimulus, and these responding processes are heavily depended on the mechanical character of the local skin. From the adaptation rate aspect, these mechanoreceptors can be categorized into two units: slow adapting (SA) receptors and fast adapting (FA) receptors. FA receptors can generate frequent output responding to dynamic stimulus, while SA receptors respond to static stimulus with time-invariant signals. Furthermore, in terms of the area of receptive field and density, each unit covers type I and type II receptors, among which the type I receptors distribute closely to the epidermis and have small receptive fields, while type II receptors are at the deeper dermis and have wide receptive fields. The Merkel cell belongs to SA-I mechanoreceptor, is located approximately 1.5 mm from the epidermis, and is very sensitive to static pressure and the stimuli of object's surface structure. Ruffini corpuscle, an AS-II mechanoreceptor at deep dermis, is responsible for dynamic mechanical stimulus and is sensitive to skin stretch and slips especially at fingertips. The FA-I type receptor, Meissner corpuscle, at the 0.5 to 0.7 mm close to the epidermis, shows the highest sensitivity to low frequent vibration of 10 to 50 Hz. It can detect light contact (e.g., tapping) and is in charge of the grip control and texture identification. The last one Pacinian corpuscle is FA-II type receptor at depth 1.5 to 2 mm in skin, which is responsible for sensing vibrations at frequency from 200 to 300 Hz, which is very crucial to percept surface textures. Details of the four receptors are summarized in Table 1. The synergistic operation of the four receptors in various modes can impart human hand with abilities to finish tasks more precisely and delicately than any advanced robotic system. Analyzing the mechanoreceptor in skin is beneficial for precisely understanding the generation of tactile sensation and is worthy of mimicking tactile sensors and designing haptic feedback devices.

Tactile Sensing Devices

Less

Tactile sensors in skin can detect interactive information generated from physical contact between skin and environment. Simulating this function of human skin, artificial tactile sensors are invented to gather haptic information, such as physical contact and mechanical sliding, and contact-based details including material, roughness, and texture. In VR system, tactile sensor is the interface between human and machine, monitoring the contact station between the two sides. With the development of main sensing mechanism including piezoresistive [26–28], piezoelectric [29,30], capacitive [31,32], and triboelectric [33–35] (Fig. 3A to D), tactile sensor has advanced from force sensor [36,37] to gesture recognition [38,39] and even intelligent identification [40,41].

Piezoresistive haptic sensor relies on the changed resistance with mechanical stimuli to realize mechanoelectrical transduction, and the main mechanism is summarized by the equation of resistance

R = ρ L A

where ρ is the resistive, and L and A are the length and cross-sectional area of the resistor device, respectively. The signal originates from the changed macroscopic geometrical parameter (L and A), where L increases and A decreases because of the Poisson effect upon stretching state [42], and also the changed ρ resulting from the dynamic percolation or quantum tunneling [43] of the conductive filler in dielectric layer [such as silver nanowires (Ag NWs)] or varied energy band structure of semiconductors (such as carbon nanotubes [44], graphene [45], or MXene [46]). Piezoresistive sensors are popular for high sensitivity, simple readout circuits, and low signal-to-noise ratio, but sometimes suffer from unfortunate frequency response due to hysteresis.

Capacitive sensors are always sandwiched structure with a dielectric layer between two electrode layers, and the mechanoelectrical signal is determined by

C = ε 0 ε r A d

where ε₀ and ε_r are the permittivity of free space and the relative permittivity of the dielectric layer, respectively, and A and d represent the effective overlapping area and distance of the two electrodes, respectively. The sensitivity of capacitive sensor depends on the changes of A and d to external mechanical stimuli, where A is sensitivity to shear force and d responds to the normal pressure. The sensitivity of capacitive sensor can be improved by embedded fillers (e.g., ionic liquid [47] and Ag NWs [48]) and specific microstructures (such as micropyramid [49] and micropillar [50]). Although capacitive sensors are sensitive to electromagnetic stimuli, they merit attention for the high sensitivity, low power consumption, and temperature interferences.

Piezoelectric sensor originates from the piezoelectric effect that the dipole separation of piezoelectric materials is able to generate electric field due to the deformation under mechanical stimuli. The first piezoelectric effect comes from the shift of anions' and cations' center in noncentrosymmetric crystal structure materials, for example, zinc oxide (ZnO) [51], lead zirconate titanate (PZT) [52], aluminum nitride (AlN) [53], and cadmium sulfide (CdS) [54]. Another mechanism is the re-alignment of permanent dipole moment in material, such as the poled polyvinylidene fluoride (PVDF) [55]. Piezoelectric constant, d₃₃, is induced to quantify the capability of materials to transduce mechanical signal into piezoelectric potential. Inorganic materials, PZT, ZnO, and CdS, are sensitive to mechanical loads because of high d₃₃, but limited in high Young's modulus that leads to poor performance in haptic sensing. PVDF is naturally flexible, but its d₃₃ is low. Thus, the combination of polymer matrix and inorganic piezoelectric material has been the effective approach to trade off the dielectric and mechanical properties [56]. The instantaneous electrical signal of piezoelectric sensor responding to high frequency load makes it an ideal candidate for measuring vibration.

Triboelectric sensors work on triboelectric effect and electrostatic induction, which can convert irregular low-frequency mechanical stimuli into electrical signal. As shown in Fig. 3D, the triboelectric effect brings positive and negative electrostatic charges on the two tribomaterials, while electrostatic induction causes current flow between the external circuit linking the two back electrodes. Triboelectric sensors possess a wide range selection of materials, such as polytetrafluoroethylene (PTFE), fluorinated ethylene propylene (FEP), polyimide (PI), and polyvinyl chloride (PVC) as positive triboelectric materials and polyamide (PA), polyurethane (PU) Cu, cotton, indium tin oxide, and human skin as negative triboelectric materials [57]. The electrostatic charges can be generated from friction [58], corona polarization [59], and iron irradiation [60]. Although poor signal-to-noise ratio causes difficulties, the diverse working mode [33], high sensitivity under low frequency, and self-powered naturality make it promising in haptic HMI areas.

Force sensor

Pressure measurement is the most basic function of tactile sensors by converting force input into electrical signals. The sensing signal can be simply recorded as “0” or “1” to represent contact or contactless station [61,62], and the force quantity can also be characterized with correction by standard curve [63,64]. Choi et al. [65] presented a transparent and linear capacitive pressure sensor with high linearity (R² = 0.995) over a wide pressure range (5 to 100 kPa). Serving as a transparent cover on phone screen, it could directly convert mechanical touch into electrical signal to input contact signal and even record the sliding trajectory, showing promising application as input unit in a VR system. Bai et al. [66] reported an iontronic pressure sensor with ultra-broad-range (0.008 Pa to 360 kPa) high sensitivity. They induced graded intrafillable architecture strategy that boosted the sensitivity and simultaneously widened the pressure sensitivity. With the correction of standard curve, it could respond to a broad pressure range of 360 kPa with a sensitivity over 200 kPa⁻¹. The integrated microsensor array also presented high resolution under submillimeter of 100 μm with negligible noise, which merited tactile sensors with high sensitivity under wide pressure range.

For force sensor, the high resolution under linear response is always the target for researcher in area of tactile sensing. So far, various processing strategies have been explored to fabricate the active layer in force sensors, for example, laser microprocessing [67], photolithography [68], 3D printing [69], and textile technology [70]. Huang et al. [71] developed a paper-based force sensor with embedded Ag NW micro-probe arrays. Through double-sided laser printing approach, Ag NWs were fabricated on paper substrate with 2.5-mm unit size, and the sensor array could sensitively detect spatial distribution of touch pressure. As shown in Fig. 4A, Zhang et al. [72] induced photolithography strategy to design a flexible pressure sensor array with ultralow spatial cross-talk. Silver nanofibers (Ag NFs) and polydimethylsiloxane (PDMS) were etched with micropatterns and integrated with a photo-reticulated strain localization film (prslPDMS) to form a sensor array with micro-cage structure. Thus, this sensor array exhibited reduced pixel deformation overflow by 90.3%, sufficient pressure resolution to detect 1 g (~150 Pa) even in bend condition, and simulated pixel resolution over 4,000 ppi. Assembled on phalanges of the palm, it could monitor the pressure on fingertip when grasping an object (Fig. 4B), showing its ability in HMI. Zhang et al. [73] produced force sensors via a fast photocurable, solid-state conductive ionoelastomer (SCIE) by high-resolution three-dimensional (3D) printing. The printed SCIE-based building blocks had high-resolution architectures (about 50-μm overhanging lattices), high Young's modulus, good stretchability, and sustained conductivity in a broad temperature range. It could be shaped into 3D flexible tactile sensors with enhanced performance, such as the printed gyroid-based piezoresistive sensor exhibiting sensitivities of 3.7-fold higher than its bulky counterparts. Qi et al. [74] reported a kind of sensing yarn based on core-shell fibers of stretchable electrode and piezoresistive elastic wrap. The textile structure endowed the sensor with high sensitivity (12.3 N⁻¹) with wide sensing range (0.001 to 5 N), relatively larger contact area multiple contact sites, and larger deformation space for multimodal mechanical stimuli, which offered tactile sensors potentialities in wearable VR components, and artificial intelligence applications. Huang et al. [75] proposed a flexible capacitive sensor based on PDMS microfoam and laser-induced graphene electrode. The sandwiched structure of plate–foam–plate provided high sensitivity of about 0.026 kPa⁻¹ in 15 to 40 kPa with low hysteresis of about 9.762%. Accordingly, this sensor could be integrated on gloves to detect joint movements and was possible to be applied in gesture recognition. The iontronic sensor reported by Luo et al. [76] was based on slant hierarchical microstructure inspired by gecko. This hierarchical layer, acting as electrode contacting with an ionic gel layer, could eliminate the pressure resistance and increase the functional interface expansion, which improved the relative capacitance change in both the low- and high-pressure region and realized the sensitivity of 36,000 kPa⁻¹ and effective measurement range up to 300 kPa.

With the demand for detecting complex mechanical stimuli, tactile sensors are gradually optimized to respond to multidimensional forces such as pressure [77], shear force [78], and twisting force [79]. Chen et al. [80] proposed a skin-inspired multidimensional sensor capable of detecting 3D stimuli in three orthogonal axes. The sensor was rationally integrated into three subsensors of highly anisotropic, which realized distinguishing various mechanical stimuli including in-plane tension, normal pressure, and shearing. Ren et al. [81] reported a fully elastic and metal-free tactile sensor that could detect both normal and tangential forces. The elastic deformation was realized by tiny burr arrays on the active layer. Thus, the sensitivity of the sensor responding to normal pressure was improved to 51.43 kPa V⁻¹, and a wide detection limit of tangential force from 0.3 to 40 N with rough sensitivity of 0.83 N V⁻¹ (0.5 to 3 N) and 2.50 N V⁻¹ (3 to 40 N) was achieved. As shown in Fig. 4C, Yan et al. [82] demonstrated a soft tactile sensor with self-decoupling and super-resolution abilities based on a sinusoidally magnetized flexible film. Its deformation could be characterized by a Hall sensor based on the varied magnetic flux densities under external forces. Accordingly, this sensor could accurately measure normal and shear force with a single unit and realize a 60-fold super-resolved accuracy enhanced by deep learning. The well performance of this sensor is shown in Fig. 4D, assisted by the sensor, and the robotic gripper could accomplish challenging tasks such as stably grasping fragile objects, which was beneficial for VR-based HMIs.

Besides abovementioned force sensors, plantar pressure monitoring is also an important aspect in VR system [83–87]. For example, as shown in Fig. 4E, Zhang et al. [88] produced a kind of intelligent socks based on triboelectric effects for monitoring foot-based activities. Frustum structure of millimeter scale was patterned on silicone rubber aided by 3D-printed mold to design the active layer, and the integrated socks could harvest waste energy from low-frequency body motions to transmit wireless sensory data. It could help to gait analysis realizing a 93.54% identification accuracy of 13 participants and recognize five human activities with high accuracy (96.67%). In this case, a digital human system was established (Fig. 4F) for applications in healthcare monitoring, personal identification, and future smart home. Shi and colleagues [89] reported a smart floor monitoring system based on triboelectric sensor. It is fabricated with unique “identity” electrode patterns using scalable screen printing technique and enabled a parallel connection to reduce system complexity as well as the deep-learning computational cost. The developed smart floor technology exhibited its potentiality in establishing virtual building/home system in near future.

Gesture recognition

Gesture can convert physical motions into digital language and is capable of delivering ideas, emotions, and decisions [90]. In VR system, gesture recognition enables human to realize gesture navigation and virtual control close to real world, resulting in a more direct HMI approach, which is also necessary for building a more immersive virtual real world [91,92]. Hence, it means to precisely reproduce hand or body gesture in a computer-created virtual space. The first key segment of this process is sensing techniques [91–95]. Tactile sensors are able to respond to stretching and bending movement of hand or body motions and have been applied in gesture recognition in recent years [96–99].

Wang et al. [100] proposed bioinspired data fusion architecture based on skin-like flexible strain sensor made of single-walled carbon nanotubes. The somatosensory data from sensor integrated with visual data were processed by the architecture and performed the gesture recognition. They finally built an auto-recognition and feedback system, which realized gesture-controlled quadruped robot, showing the potential application in virtual control. As shown in Fig. 5A and B, Zhou et al. [101] designed a wearable sign-to-speech translation system-based triboelectric yarn sensor array. Owing to unique structure and induced soft materials, the yarns are self-adaptive to fingers no matter under releasing or stretching condition. Mechanical motions of fingers could be converted into electrical signals via the sensor arrays, and assisted by deep learning, electric signals were processed to realize sign-to-speech translation. Figure 5C exhibits the wireless real-time translation system, and a total of 660 sign language hand gestures (American Sign Language) were successfully analyzed with a high recognition rate of 98.63% and a short recognition time of less than 1 s.

Tan et al. [102] reported a gesture recognition wristband that could achieve virtual keyboard input (Fig. 5D). As the inset in Fig. 4D shows, this wristband was based on physiological anatomy to capture movements of the muscle belly or tendon in wrist that was related to hand activities. Relying on the hybrid signal of piezoelectric and triboelectric sensor (Fig. 5E), this wristband could precisely recognize mechanical information regarding hand gesture without consuming electricity (Fig. 5F), and a maximum accuracy of 92.6% in recognizing 26 letters is achieved. Thus, it was considered to be applied in decoding gesture command and sign language translation, which could broaden the input modalities of VR systems. Moin et al. [103] proposed a wearable biosensing system with in-sensor adaptive machine learning for hand gesture. The main part of this system was a screen-printed, conformal electrode array that could collect electromyography signals, and it was characterized with in-sensor adaptive machine learning abilities. The reported experiments proved its performance that hand gesture classification accuracy of 97.12% was obtained when classifying 13 hand gestures using only a 4-s window of training data per gesture.

Figure 5 (G to K) shows a deep-learned skin sensor developed by Kim et al. [104] that could decode the epicentral human motions. This sensor was fabricated by consecutive laser serpentine patterning electrode allowing to conformably attach to epidermis (Fig. 5H and I), and high sensitivity was achieved with the laser-induced nanoscale cracking. The sensor system was capable of collecting data from arbitrary part of the wrist (Fig. 5J) and automatically training the model in real-time demonstration with a virtual 3D hand that reproduced real hand motions (Fig. 5K). Besides, it was also available for pelvis to capture dynamic gait movements in real time and would facilitate an in-direct remote measurement of human movements that contributed to VR applications.

Zhu et al. [105] designed an exoskeleton manipulator based on bidirectional triboelectric sensors that could recognize body motions. As shown in Fig. 6A, this exoskeleton could capture movements of arm and fingers and then project these into robotic arm or virtual space. Because of the structural consistency between the exoskeleton and the human body, further kinetic analysis offered additional physical parameters without introducing other types of sensors (Fig. 6B). Figure 6C demonstrates its application in controlling a virtual figure for immersive physical training. Guo et al. [106] reported a wearable multidimensional motion sensor toward VR sports based on detecting both vertical and planar movements. This sensor could be integrated with a belt to identify low degree-of-freedom motions. Assembled on ankle position, it was able to achieve differentiated kicking force and direction with an accuracy of 97.5%. Thus, the virtual game (e.g., fitness game and shooting game) was successfully demonstrated. Luo et al. [26] fabricated a kind of conformal tactile textile to integrate a tactile learning platform. Figure 6D depicts the fabrication method of the piezoresistive fibers, and the textile sensor was able to be assembled on various parts of clothing (Fig. 6E). This platform could estimate whole-body gestures according to the distribution of the plantar pressure (Fig. 6F and G) contributing to the promising wearable VR systems.

Touch identification

Aiming to establish more immersive VR systems, functions of tactile sensors are not limited in force and motion recognitions, but extended to intelligent identifications [107–109], such as pressure mapping [110,111], texture[112] , and object recognition [113].

As shown in Fig. 7A, Wang et al. [114] designed a pressure-sensitive triboelectric sensor matrix and realized real-time tactile mapping. They treated PDMS with dry etching, which generated a micro/nanostructure surface (Fig. 7B) that could increase the sensing performance of the sensor. Figure 7C shows the flexible matrix of 16 × 16 pixel with a resolution of 5 dpi. Accordingly, patterned pressure (e.g., mold in a shape of “6” in Fig. 7D) imposed on the matrix could be readily imaged though the simultaneous use of multiple pixels (Fig. 7E), which would offer opportunities for application in VR-based HMIs. Based on biomimetic mechanoreceptor and stress field sensing, Shang et al. [115] reported an electronic skin for modular multi-parameter perception. It could decode complex tactile information into field information. By reconstructing and analyzing the stress field, distribution of 3D forces could be resolved with 1.8° polar angle resolution and 3.5° azimuthal angle resolution, and the hardness of object could also be detected.

With tactile sensors of multi-mode, the surface texture can also be recognized. For example, Li et al. [116] proposed a multifunctional biomimetic tactile system via a stick-slip sensing strategy, which was a universal approach for monitoring slippage and estimate the surface characters of objects by sliding. As is depicted in Fig. 7F, the sensor consisted of an epidermis-inspired double-helix top layer and a spinosum-inspired pyramidal bottom layer, which were designed for transmitting vibrations obtained from stick-slip phenomena and sensibilization, respectively. Assisted by deep learning, this system realized a high recognition rate of 100.0% for both static and sliding states, and obtained the ability to distinguish six types of materials (93.3%) and six different roughnesses (92.8%) (Fig. 7G).

Shan et al. [117] designed a photoelectric tactile sensing system for high-accuracy texture recognition. This system realized the simulation of human tactile information, and the recognition of 16 kinds of fabrics was distinguished with an accuracy of 94.1%. Chun et al. [118] reported a tactile sensor with a single sensor architecture made of single-layer graphene. Because of the local deformation of a specific area of the single-layer graphene, resistance variations could be reflected in the resistance of the entire sensor and the sensor was able to detect a vertical pressure as low as 24 Pa with a fast response of ~2 ms for deformation and ~3 ms for restoration. Thus, by introducing microstructures inspired by human fingerprints, surface texture was successfully defined through fast Fourier transform analysis, which provided a simple method that, using a single sensor, realized surface texture recognition at the level of human sensation. Chun et al. [119] reported an artificial neural tactile sensing system using particle-based polymer composite sensors (piezoresistive and piezoelectric) and a signal-converting system (Fig. 7H). The sensor could specially respond to pressure and vibration, and its output signals were similar to those of slow-adapt and fast-adapt mechanoreceptors. The system integrated with sensing signals and deep learning was able to classify fine and complex textures (Fig. 7I), and could also be used to predict unknown textures on the basis of the trained model.

Distinguishing object is also another intelligent application of tactile sensors [120]. Li et al. [121] integrated quadruple tactile sensors with a robot hand and realized precise object recognition via grasping. The quadruple sensor was fabricated with skin-inspired multilayer microstructure and worked as thermoreceptor and pressure sensor to perceive thermal conductivity of a material and the contact pressure, respectively. Combining data acquisition and processing, the intelligent robot hand was able to precisely recognize objects of various shapes, sizes, and materials, which was beneficial for VR system to realize complex recognition. In Fig 8A and B, Sundaram et al. [122] created a scalable tactile glove to help learn the signatures of human grasp. The glove was fabricated with piezoresistive pressure sensors to analyze the grasp patterns of human hand based on pressure distribution (Fig. 8C). As shown in Fig. 8D, they recorded a large-scale tactile map (a dataset of 135,000 frames) when grasping various objects with a single hand. Deep learning process revealed that the spatial correlations and correspondence between finger regions that emerged from the dataset represented the tactile signatures of the human grasping strategy. Thus, these tactile data were very useful for duplicating operations of virtual hand in a VR system.

Machine learning in intelligent haptic sensing

In haptic sensing area, data acquisition and processing are gradually becoming the most important part since multimode and real-time sensing bring a bulk of data. As a main branch of artificial intelligence, machine learning possesses the ability to analyze, extract, and decode these complex data, providing opportunities to efficiently handle the data process. The four main tasks of machine learning are classification, regression, clustering, and dimension reduction based on related algorithms. Machine learning applied in above haptic sensing devices is summarized in Table 2.

Haptic Feedback Techniques

Less

For a closed-loop tactile interaction in VR system, haptic feedback is a reverse process compared with sensing devices. In the development of VR technology, visual, auditory, and olfactory have been maturely applied, while virtual tactile is still a weak link, because it needs feedback interfaces conformal with skin and the flexible, high resolution, and scalable characters are necessary. To date, efforts in haptic feedback exist, including techniques such as mechanical vibration [123], ET [124,125], DEA [126], and so on [127–129], which have enriched VR applied in immersive games, virtual communication, and teleoperated robotics.

Mechanical vibration feedback

Mechanical vibrations, aiming to evoke the vibration-sensitive skin receptors, are able to generate virtual haptic sensations on skin. Vibration stimuli can be realized by pneumatic and magnetic actuators. Pneumatic actuators are always facilitated with soft membrane that can expand or contract relaying on air inflation or deflation, leading to the vibration stimulus [130,131] or fingertip force feedback [132]. For example, Qi et al. [133] reported a pneumatic glove shown in Fig. 9A. Two main modules were low-pressure (<60 kPa) actuated PneuClutch and PneuIndenter, which allowed this glove with untethered lightweight characters (283 g) and enabled users to sense kinesthetic and cutaneous feedback that realized touching, pressing, grasping, squeezing, and pulling virtual objects with immersive haptic sensation. The authors also developed a VR environment for using in object grasping and complex archery game with the glove, showing its potentiality in medical training, industrial training, entertainment, and social interaction. As shown in Fig. 9B, Song et al. [134] developed a pneumatic actuator to generate fingertip feedback. The integrated glove (Fig. 9C) used electrostatic force to generate inner air pressure instead of the bulky compressor, which makes it a light weight (0.57 g) and miniaturization system. Thus, the glove was capable of providing realistic feedback when holding a virtual object (Fig. 9D), contributing effective VR experiences conjugated with other virtual devices (Fig. 9E). Sonar et al. [135] provided a soft pneumatic actuator skin with soft strain sensors where high-frequency sensing and actuation were realized. The liquid metal sensor was embedded in the upper layer of the vesicular actuator to form a sensor and actuator laminate skin that could provide controllable actuator shape in real time up to 100 Hz at output forces up to 1 N. This bifunctional actuator skin offered a promising direction for intuitive and comprehensive haptic interactions. These pneumatic actuators induced controlled air flow to inflate or deflate the deformable bladder realizing skin stimulation. Although they require externalized hardware for pumping, the actuator merits its flexible structure to be conformal with human skin, and the soft membrane coordinated with appropriate air pumping can provide flexible and controllable virtual tactile sensation.

Magnetic actuator is another mechanical tactile approach that is commonly used because of fast response and controlled flexibility. It relies on the vibration of miniature motors to generate sensations like object moving in a direction perpendicular to skin, which is able to simulate the most realistic haptic sensations. As shown in Fig. 10A, Yu et al. [136] reported a haptic interface with skin conformability, wireless, and battery-free characters, which could simulate skin by localized mechanical vibrations to create virtual haptic information of programmable patterns. The actuator is made of magnet actuators embedded in silicone-based substrate with controller of near-field communication (NFC) antenna. It could be directly laminated onto curved skin owing to the flexible encapsulation, and the actuator was capable of generating vibration force of about 135 mN with a fast-responding frequency (about 300 Hz) that was sufficient to simulate the skin. Figure 10 (B and C) demonstrates its applications in social media interactions and prosthetic feedback, and it was expected to offer opportunities in personalized rehabilitation, surgical training, and multimedia entertainment experiences. Li et al. [137] miniaturized this kind of actuator and developed a finger-integrated haptic interface (Fig. 10D and E). The actuators were designed in diameter of 5 mm and thickness of 1.45 mm to integrate a 2 × 3 array as dynamic Braille on fingertip (Fig. 10F). Figure 10G summarizes the correction of a subject group (five testers) in 10 groups of blind tests (totally 50 sets of data) when testing Braille recognition of patterns in Fig. 10H. For these seven letters, the average recognition accuracy was 85.4%, which was sufficient for Braille interaction. In magnetic actuator, the magnet can freely vibrate in an encapsulated interface so that the vibration intensities and high frequency are available controlled under current magnitudes, pulsation character, and oscillation frequencies.

Controllable frequency in a wide range is the advantage of mechanical vibration feedback, but also brings the nonconformal contact with the skin. There are developed adhesive interfaces to tackle this interface issue. For example, Kim et al. [138] reported a bio-inspired skin-attached haptic interface with interconnect structures that could resist sweating and vibration. This interface was designed with microchannels coated with a soft gel inspired by free frog and interconnected architectures inspired by snail, which provided with an attachment weighing about 300 g on rough sweaty skin and the reversible adhesion that could tolerate vibration-induced fatigue on both dry and sweaty skin surface. Integrated with a vibrational haptic actuator (10 mm shaftless vibration motor), the developed haptic interface could implement virtual physical interaction by recognizing human motion and vibrohaptic feedback. Hwang et al. [139] proposed adhesive interface for adaptive haptic interaction. Inspired by diving beetle-like small dense hairs possessing concave cavities, this interface obtained high adaptability on various nonflat surfaces with robust adhesion in dry (≈16 kPa) and wet (≈27 kPa) conditions. Integrating electronics with PVC gel actuator, the adhesive platform was able to transmit vibration to the skin surface. They encoded various vibration information to simulate the real surface texture of objects, and provided touching and feeling representative of a gecko lizard in a VR environment.

ET feedback

Imparting appropriate electric current on skin is a straightforward approach to realize proprioceptive haptic feedback, since the skin receptors depend on bioelectricity to transmit tactile sensations [140–142]. The ET feedback devices are always fabricated with electrode pairs of anode and cathode to form the current flow under skin, and are advantageous for small size as well as high spatial resolution. Sato reported an ET display for the distribution of force vectors [143]. Since the selective stimulation of mechanoreceptors represented various distributions of force vectors, anodic and cathodic stimuli were induced to trigger FA receptors and SA receptors, respectively. The distributed force vectors were reproduced by selecting different stimulating point. Thus, this strategy indicated that the representation of the distribution of force vectors was feasible via ET stimulations. Komurasaki et al. [144] reported an electrovibration and electrical stimuli integrated haptic display based on micro-fabrication process. The frequency-dependent relationship of tactile perception toward electrovibration and electrical stimuli was revealed. Accordingly, virtual tactile sensations of vibrational friction, pressure, and vibration could be provided.

For ET feedback, the appropriate current and voltage is the most concerning factor because of the drift skin impendence and variable individual properties. Tezuka et al. [145] improved the ET electrodes using a micro-needle electrode array (Fig. 11A). They arranged the needle electrodes on the fingertip and the ground electrode on the fingernail (Fig. 11B). The micro-needle electrode penetrated the part of stratum corneum but not reached the pain point. In this case, the high impendence of stratum corneum was overcame and the threshold voltage on fingertip was reduced under 10 V, which was smaller than that of plat electrode (over 70 V). The patterned ET stimulation was realized by the electrode array with a resolution of 2 mm (Fig. 11C). As shown in Fig. 11D and E, Lin et al. [146] proposed an ET rendering system realizing ET stimuli with both high spatial resolution (76 dots/cm²) and rapid refresh rates (4 kHz) based on a current-steering super-resolution stimulation technique. They used a high-frequency modulation approach to reduce voltage under 13 V, and achieved Braille display as well as digital virtual experiences.

To reduce current flow, as shown in Fig. 12A, Shi et al. [147] designed a self-powered ET system based on triboelectric nanogenerator and flexible ET interface. The ET interface was fabricated with ball-shape electrode (Fig. 12B), and the electrostatic discharge between electrode and skin could induce sensitive ET stimulation (Fig. 12C). They used the controlled distance to regulate the current induced into skin, resulting in an ultralow current of 25 μA, which was remarkably lower than the safety current threshold current (10 mA) on skin [148]. Via the effective ET stimulation with the skin-conformable interface (Fig. 12D and E), the virtual tactile interaction was realized in Fig. 12F, where the patterned hand input could be converted into electrostatic discharge on skin, and then sensed with the subject with right feedback. This ET system proposed a different ET approach than traditional ET devices and could be utilized in dynamic Braille, virtual tactile interaction, and augmented virtual protection. These strategies of micro-needle electrode and stimulation of electrostatic discharge solved the weakness of ET devices suffering from shifted impendence of human skin. The ET stimulation has advantages of simplicity and immediate ability with stretchable patterned electrode that facilitate integration with human skin to develop wearable virtual tactile devices.

DEA feedback

DEA consisted of electroactive elastomer sandwiched between two compliant electrodes. When an appropriate voltage is applied, as shown in Fig. 13A, the generated attractive coulombic force will induce compressive stress on the elastomer, resulting in the deformation of the actuator, and the moderate deformed actuator is able to squeeze the local skin leading to tactile sensations. Koo et al. [149] developed a haptic display device that could offer stimulation on skin without any additional electromechanical transmission. The display was fabricated with electroactive polymer and flexible encapsulation, which could be directly integrated on fingertip. Mun et al. [150] presented a soft actuator based on multi-layered accumulation strategy. The layer-by-layer structure of electroactive polymer and electrodes provided the maximum vertical protrusion of 350 μm, and the output force was up to 255 mN, which was sufficient to stimulate skin. This large deformation also offered opportunities for realizing hierarchical ET stimulation, and five-level stimuli on forearm could be sensed by the subject. As shown in Fig. 13B, Ji et al. [151], using an 18-μm-thick DEA of multilayer structure, realized untethered feel-through haptic sensation. They integrated DEA, electronics, and battery into a haptic device of 1.3 g that is beneficial for wearing during everyday activities for extended periods. The DEA (with three electroactive layers) could provide stretches and compresses at frequencies from 1 to 500 Hz, and deform the skin in normal direction by over 6 μm (Fig. 13D). The demonstration in Fig. 13E and F was a tactile reading scenario, where the blindfolded subject could read the black letters using a photodiode to trigger DEA in the black area (Fig. 13F). Thus, this feel-through DEA device was able to be combined with visual and auditory device to realize more immersive VR experiences.

Electrohydraulic actuators are soft liquid-filled shells that deform due to electrostatic forces, which is indeed a variation of DEA [152–154]. Leroy et al. [155] reported a submillimeter-thick flexible hydraulically amplified electrostatic actuator that was able to deform out of plane and in plane, which could induce normal and shear forces to fingertip, hand, and arm. As shown in Fig. 14A, the actuator consisted of an oil-filled cavity with a metalized polyester boundary and a central elastomer region. When applying voltage to the annular electrodes, the dielectric oil could be squeezed into to the central stretchable region (Fig. 14B), generating a raised bump. Via segmented electrodes, the central bump could be pushed up, and also shifted north/south and east/west, forming rotation-like mechanostimulation (Fig. 14C). A 90-mg actuator in 6 mm × 6 mm could generate 300-mN force and displacements of 500 μm in a low-profile geometry (60% vertical strain). Owning to its thin structure and skin compliance, this actuator was promising for integrating a glove interface used in multimedia VR systems. Frediani et al. [156] used electroactive elastomer to control the liquid-filled bubble interface and realized tactile stimuli. The active dielectric elastomer would expend the bubble to release the liquid pressed on finger, while the maximum pressure occurred when voltage was off. The DEA was set to avoid any direct contact between skin, so that could provide electrical safety. Grasso et al. [157] developed a fully 3D-printed flexible haptic interface shown in Fig. 14D. The actuator was printed in six layers with dielectric oil filled in the cavity, and it depicted sufficient flexibility that could be directly attached to skin (Fig. 14E) as well as other curved surfaces (Fig. 14F). Even under stretch over 50%, the actuator could also give cutaneous stimuli at a wide frequency range (from DC to 1 kHz). By integrating a 2 × 2 array on fingers of human subject, the accuracy to recognize quadrant could reach 86%. Park et al. [158] proposed a soft haptic actuator based on knitted PVC gel fabric. Meriting from the intertwined structure, the electric field-induced deformation and electrostatic force are combined to maximize the vibration force and response time. Thus, the effects generated vibrations that were sufficiently strong for human perception. For these kinds of DEA actuators, they could directly convert electric signal into mechanical vibration without moving unit, avoiding bulk and mechanical complexity than mechanical vibration feedback. Although high input voltage is indispensable, its integration with dielectric liquid also provided shifted force stimulation, which impart DEA actuators multi-functions in haptic VR interactions.

As summarized in Table 3, examples of haptic feedbacks mentioned above are compared according to the mechanism, material section, and input and output properties. Subcategory of the main approaches is also listed in terms of the power mechanism, such as the pneumatic and magnetic actuators arranged clearly. The fabrication dimensions are noted with the shape of the device: dimeter (D), length (L), thickness (T), and weight (W). For mechanical vibration and DEA, the output parameters mainly concentrate on the none-load displacement (D), blocking force (F), and frequency (f), while ET devices merit for the output current (I) and high resolutions, which originate from their power source and point electrodes. Performance of pneumatic actuators depends heavily on the input air pressure, while that of magnetic actuator and DEA are decided by the input current signal covering voltage and frequency. Besides, typical applications of listed examples are also described. In summary, mechanical vibration are advantageous for the high safety and available frequency range, and ET devices with the point electrode and light weight are favorable for wearable designing and high resolutions. Although DEA needs high voltage to trigger virtual tactility, its flexibility and skin-compliant property are worthy of more attentions.

Other feedbacks

In addition to mentioned approaches, other novel feedbacks have been investigated for expanding applications of haptic feedback in VR systems, such as thermal feedback [159,160], ultrasound feedback [161,162], and piezoelectric actuator [163]. Temperature sensing is also considered as the main function of human tactile receptor, and temperature sensation originates from heat flux occurring when contacting with object of different thermal conductivities. Various thermoreceptors under skin are able to detect independent temperature zones with a wide temperature range of 4 to 52 °C and resolution of 0.02 °C [164]. There are Joule heating [165], electrocaloric effect [166], magnetocaloric effect [167], and thermoelectric effect [168] that are induced to realize virtual thermal sensation, and functional materials, such as fluidic heat transfer [169], mechanocaloric materials [170], and phase-charge materials [171], can also generate varied heat exchange with skin. Oh et al. [172] developed a thermal feedback device based on patterned liquid metal (Fig. 15A) and integrated glove (Fig. 15B). The printed liquid metal electrode was encapsulated in silicone layers and has improved heating performance from 25 to 85 °C even strained to 50%. They finally proposed a multimodal sensing and feedback system that enabled immersive VR experiences (Fig. 15C and D). Lee et al. [168] reported a skin-like wearable thermo-haptic device that could actively cool down and heat up deformable skin surfaces even under maximum stretching over 230%. The unique highlight of this haptic interface was the capability to rapidly return to the original temperature after reaching the target temperature within a few seconds by changing the direction of current, which was beneficial for the fast thermal sensation without subsequent unintended parasitic thermal residue that enabled to realize various virtual situations with higher accuracy. Li et al. [173] integrated a thermal and vibrational feedback glove based on miniature vibrators and semiconductor refrigerators. It could allow temperature feedback change from 10 to 47 °C, and realized the VR experience to dilute sulfuric acid in a virtual environment. Thermal haptic feedbacks as an approach to realize virtual haptic over mechanical force and vibration will enhance the degree of artificial immersion and render VR/AR in a more authentic manner. Ultrasound feedbacks, relying on mid-air interface, are also promising approaches because they allowed users to avoid being tethered to any holdable or wearable devices. For example, Howard et al. [174] reported a pan-tilt ultrasound interface for larger interaction workspace in VR. It could provide delivering sensations from multiple different directions because of two degrees of freedom, enlarged usable workspace in a 14-fold increase, focal point repositioning speeds over 0.85 m/s, and positional accuracy below 18 mm. Liu et al. [161] used a piezoelectric mircromachined ultrasonic transducer (pMUT) to achieve ultrasonic haptic feedback. This pMUT is able to provide resonance frequency of 32.9 kHz and unit acoustic pressure of 0.227 Pa. The integrated array of 251 × 251 pMUTs enabled sufficient acoustic pressure to create haptic sensations, and this proposed demonstration has potential to be applied in ultrasound haptic feedbacks.

Applications

Less

The development of haptic sensing and feedback devices has promoted the innovation of VR techniques and also provided more immersive experiences than systems only with visual and auditory interactions. Haptic sensor acts as an input approach, and feedback offers available output modality, which have enhanced the multi-functionality of VR systems [175–177].

Haptic sensors are mostly used for collecting touch information including contact or not, pressure quantity, and force distribution, which are important data for modelling in virtual world. For example, Kim et al. [178] reported a soft transparent touch panel based on ionic hydrogel, and applied it as an epidermal input panel that realized virtual input (Fig. 16A). As shown in Fig. 16B, Li et al. [179] introduced a flexible supercapacitive nanotexture that achieved pressure monitoring such as facial and gripping pressure mapping, which could facilitate the signal transforming modality for next-generation VR systems. Basically, to capture hand gesture is the first step to reconstruct hand-based task and give instructions in VR environment. For example, Sun et al. [7] used their ring sensor (intergrade on finger in Fig. 16C) to collect the data knuckle movements and processed the data to reproduce the hand instruction for playing a virtual piano. Also, capturing body motion is the advanced application of hand gesture recognition. Gong et al. [180] designed a piezoresistive artificial bionic skin that could adhere compliantly on human body to record the body motion, and demonstrated its application to capture motion gesture and wirelessly control a robot in real time.

Haptic feedback as the reverse process of sensing enables users to understand touch information such as materials, texture, and roughness, and proved immersive HMI experiences. Simultaneously, corresponding instructions made by users are able to enhance the control and collaborative ability in HMI, contributing to closed-loop HMI. Therefore, haptic feedback devices have been applied in immersive entertainment, feedback prosthesis, and tactile interactions. The wireless haptic interface, reported by Yu et al. [136], could be attached on opisthenar, arm, and back (Fig. 16D) and generated mechanical vibrations to simulate the impacts when playing combat game. Jung et al. [181] reported skin-integrated haptic actuators to facilitate the robotic prosthetics. As shown in Fig. 16E, the amputee could indirectly sense tactile experience of contact according to the vibration intensities and patterns and transmit instructions to the prosthetics. Thus, the eggshell could be grasped without broken shell. The application of prosthetic is similar as controlling virtual robotic arm in a VR environment. Fan et al. [182] designed a Digital Twin-driven mixed reality framework for immersive teleoperation with haptic rendering based on soft pneumatic haptic feedback, and realized immersive teleoperating in both real and realistic environment. The ET interface shown in Fig. 16F could transmit safe current to skin to generate ET sensations and could translate the trajectory of virtual mouse into tactile patterns to achieve tactile communications from virtual environment to real world [183]. Thus, haptic feedback process in VR system is more of immersion than that only with vision and auditory environments. For example, the vibration of cell phone would increase shock sensation when playing phone games. Accordingly, such skin-integrated vibration feedback can provide mimic impact during fighting game, and to be vagarious, an overdosage ET stimulation is able to reproduce pain sensation when playing shooting games. This haptic feedback can be extended to be applied in virtual military training to increase the immersive level.

In addition, the requirement of artificial intelligence of things (AIoT), wise information technology of medicine (WITMED), and smart home and cities will create promising paths forward for haptic sensing and feedback. The virtual control of teleoperated robot is an example that haptic sensing and feedback are necessary. The human hand instructions need to be translated into control code by sensing devices that are also in charge of monitoring the operation status between robotic hand and object. The feedback section is also important, because it makes the operator in real time aware of the operation status and gives following instructions. The integrated sensing and feedback are especially a key point for special task with fragile object. Surgery simulation and training also need virtual haptic interactions to allow doctors to perform in virtual environment. Haptic feedback can provide sensations similar with the real process to accomplish the training purpose. This is also applied for the interactive education in which students are capable of understand the learning content and experimental subject, which can enhance the teaching in an immersive way. In entertainment area, virtual vision and audition have been realized with VR glasses and audio devices, but virtual haptic is missed. Inducing body sensing and feedback parts, for example, the body gesture capturing and force feedback, will greatly improve the levels of immersion. To expect that if a haptic feedback servicer covering data of hand shaking and hugging is established, the multi-media social communication maybe come true. In this case, haptic greeting is not impossible to realize without the special and temporal limits. The greeting can be sent to ourselves in tens of years future, which will be an obsessed experience.

Conclusions and Prospects

Less

The past decade has seen the rapid development of haptic sensing and feedback devices along with efforts in emerging materials, miniaturized electronics, and data processing, and their applications range from HMI to medical monitoring, interactive entertainment, immersive learning, and remote surgery. However, challenges in this area still remain. For haptic sensing devices, first, the sensitivity and accuracy need to be improved. Most of the time, the wide liner responsibility and high sensitivity cannot be realized simultaneously because they are heavily dependent on the modulus character of active materials. Next, stability and durability of the wearable devices should be guaranteed during exercising, sweating, or other operating circumstances that are extremely cold or hot. Finally, technical conflicts between data processing and real-time response exist, because the time to finish an operation (e.g., hand gesture) is necessary, and so is it to collect the data for recording the operation. Thus, the data processing is always delayed, resulting in an asynchronous HMI process. Besides, the most developed haptic sensors are still under laboratorial level and disjointed with realistic applications. The integration between wearable sensor and VR system is still in a low degree due to the complex data acquisition, transformation, and hardware analysis.

For the mentioned kinds of haptic feedback approaches, they all suffer from technical bottlenecks. Mechanical vibrant feedback heavily relies on magnet of millimeter level, failing to integrate thin-film devices, and the high-resolution skin stimulations are always sacrificed. ET method, though proud of high resolution, is stuck in the appropriate current and voltage to adapt to the variable skin impedance across the body and between individuals. In terms of DEA, depending on the hundreds or thousands of voltages to power millimeters of deformations, the safety and sensitive skin stimulation cannot be guaranteed. In addition, the functions of these feedbacks are limited only in pressure feedback, and virtual haptics, such as slip sensation, thermal feedback, and cold sensation, are difficult to be realized. The high-resolution haptic stimulation needs to further increase the density of stimulators, while sophisticated wiring and signal transmitting are necessary. Most importantly, for the cooperation of haptic sensing and feedback, devices can hardly realize closed-loop virtual haptic interactions, which are supposed to be a future mainstream in developing VR systems. The commercializing development of haptic sensing and feedback devices is also a considerable challenge, because most devices in laboratory do not keep price in consideration, the cumbersome wired power source, data acquire meter, and processing hardware are necessary, the integrations of these devices with others of different mode (e.g., the visual part) are difficult, and the wearability in daily life is also under development.

However, the developments in haptic interaction are still expected and promising. Following are several specific trends and developments of haptic interaction in VR system.

a. From materials aspect, burgeoning graphene, 2D materials, liquid metal, and ionic gel will boost the transparent, flexible, self-healing, self-adopt, biocompatible characteristics of sensing and feedback interfaces.

b. Advanced workmanship, such as 3D printing, direct laser writing, electron beam lithography, and microelectronic packaging, provides opportunities to fabricate flexible electrode of various patterns with high conductivity, micro-structured active layers with increased sensitivity, miniaturized multi-mode array with multi-functions, and miniaturized wearable devices that are portable, self-powered, and skin-integrated.

c. Emerging computer technologies, for example, artificial intelligence and deep learning, are accelerating the data acquisition and processing segment. For example, this would decrease the difficulty to extract characteristic signal from interminable data, which will facilitate the gesture identification and virtual model establishment.

d. Multi-mode haptic feedback, not only in force feedback, but also containing slip sensation, thermal feedback, and cold sensation, will increase the diversified multi-media interaction between human and virtual environment.

e. Closed-loop haptic interaction process is the ultimate goal. Real-time haptic sensing and simultaneously feedback will be finally applied in virtual control, smart protheses, virtual surgery, and so on.

Challenges and opportunities of haptic sensing and feedback will coexist in the near future. Although haptic perception is a subjective and complex process, and haptic interaction is a multidisciplinary and interdisciplinary area, co-efforts have been dedicated to materials, electronics, mechanics, and computer science. We believe that the bottleneck in this area will be overcome, and haptic interaction in VR systems will benefit applications including but not limited to medical, education, rehabilitation, internet of things, robotics, entertainment, and so on.

Funding

Less

National Natural Science Foundation of China (61888102)
Beijing Natural Science Foundation(L223006)
China National Postdoctoral Program for Innovative Talents(BX20230466)

References

Less

Liu

, Yiu

, Song

, Huang

, Yao

, Wong

, Zhou

, Zhao

, Huang

, Nejad

, et al. Electronic skin as wireless human-machine interfaces for robotic VR. Sci Adv. 2022;8(2): Article eabl6700.

A touch of virtual reality. Nat Mach Intell. 2023;5(6):557.

Kim

, Wang

, Lee

, Yokota

, Someya

. Skin electronics: Next-generation device platform for virtual and augmented reality. Adv Funct Mater. 2021;31(39):2009602.

Biswas

, Visell

. Haptic perception, mechanics, and material technologies for virtual reality. Adv Funct Mater. 2021;31(39):2008186.

Fang

, Guo

, Wu

. Wearable triboelectric devices for haptic perception and VR/AR applications. Nano Energy. 2022;96: Article 107112.

Wang

, Xu

, Zhang

, Yin

, Sun

, Yang

, Hu

, Liu

. Field effect transistor-based tactile sensors: From sensor configurations to advanced applications. InfoMat. 2022;5(1): Article e12376.

Sun

, Zhu

, Shan

, Lee

. Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions. Nat Commun. 2022;13(1):5224.

Pyo

, Lee

, Bae

, Sim

, Kim

. Recent progress in flexible tactile sensors for human-interactive systems: From sensors to advanced applications. Adv Mater. 2021;33(47): Article e2005902.

Senthil Kumar

, Chen

P-Y

, Ren

. A review of printable flexible and stretchable tactile sensors. Research. 2019;2019:3018568.

10.

Pirozzi

. Tactile sensors for robotic applications. Sensors. 2020;20(24):7009.

11.

Bayer

. MEMS-based tactile sensors: Materials, processes and applications in robotics. Micromachines. 2022;13(12):2051.

12.

Ozioko

, Dahiya

. Smart tactile gloves for haptic interaction, communication, and rehabilitation. Adv Intell Syst. 2021;4(2):2100091.

13.

Wen

, Zhang

, He

, Lee

. AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nat Commun. 2021;12(1):5378.

14.

Jin

, Sun

, Li

, Zhang

, Zhu

, Zhang

, Yuan

, Chen

, Tian

, Hou

, et al. Triboelectric nanogenerator sensors for soft robotics aiming at digital twin applications. Nat Commun. 2020;11(1):5381.

15.

Duan

, Shi

, Hong

, Zhu

, Lin

, Li

, Lei

, Lee

, Wu

. Water-modulated biomimetic hyper-attribute-gel electronic skin for robotics and skin-attachable wearables. ACS Nano. 2023;17(2):1355–1371.

16.

Yin

, Hinchet

, Shea

, Majidi

. Wearable soft technologies for haptic sensing and feedback. Adv Funct Mater. 2020;31(39):2007428.

17.

Pacchierotti

, Sinclair

, Solazzi

, Frisoli

, Hayward

, Prattichizzo

. Wearable haptic systems for the fingertip and the hand: Taxonomy, review, and perspectives. IEEE Trans Haptics. 2017;10(4):580–600.

18.

Tong

, Wei

, Zhang

, Xiao

, Wang

. Survey on hand-based haptic interaction for virtual reality. IEEE Trans Haptics. 2023;16(2):154–170.

19.

Wang

, Guo

, Liu

, Zhang

, Xu

, Xiao

. Haptic display for virtual reality: Progress and challenges. Virt Real Intell Hardw. 2019;1(2):136–162.

20.

Yang

, Kim

, Jin

, Gil

, Koo

, Kim

. Recent advances and opportunities of active materials for haptic technologies in virtual and augmented reality. Adv Funct Mater. 2021;31(39):2008831.

21.

Park

, Bok

, Ahn

, Kim

. Recent advances in tactile sensing technology. Micromachines. 2018;9(7):321.

22.

Girão

, Ramos

PMP

, Postolache

, Miguel Dias Pereira

. Tactile sensors for robotic applications. Measurement. 2013;46(3):1257–1271.

23.

Hammock

, Chortos

, Tee

BCK

, Tok

JBH

, Bao

. 25^th anniversary article: The evolution of electronic skin (E-skin): A brief history, design considerations, and recent progress. Adv Mater. 2013;25(42):5997–6038.

24.

Zimmerman

, Bai

, Ginty

. The gentle touch receptors of mammalian skin. Science. 2014;346(6212):950–954.

25.

Chortos

, Liu

, Bao

. Pursuing prosthetic electronic skin. Nat Mater. 2016;15(9):937–950.

26.

Luo

, Li

, Sharma

, Shou

, Wu

, Foshey

, Li

, Palacios

, Torralba

, Matusik

. Learning human–environment interactions using conformal tactile textiles. Nat Electron. 2021;4(3):193–201.

27.

, Xiang

, Shao

, Zhang

, Gu

. Multifunctional wearable silver nanowire decorated leather nanocomposites for joule heating, electromagnetic interference shielding and piezoresistive sensing. Angew Chem Int Ed Engl. 2022;61(15): Article e202200705.

28.

Xia

, Huang

, Lin

, Wu

, Li

, Gao

, Zhong

. The piezoresistive pressure sensors based on ITO nanocrystalline-plant fiber composite. Sci China Mater. 2023;66(10):3922–3930.

29.

Chorsi

, Curry

, Chorsi

, Das

, Baroody

, Purohit

, Ilies

, Nguyen

. Piezoelectric biomaterials for sensors and actuators. Adv Mater. 2018;31(1):e1802084.

30.

Mahapatra

, Mohapatra

, Aria

, Christie

, Mishra

, Hofmann

, Thakur

. Piezoelectric materials for energy harvesting and sensing applications: Roadmap for future smart materials. Adv Sci. 2021;8(17):e2100864.

31.

, Heo

, Ji

, Bien

, Park

J-U

. Transparent and flexible fingerprint sensor array with multiplexed detection of tactile pressure and skin temperature. Nat Commun. 2018;9(1):2458.

32.

Ruth

SRA

, Beker

, Tran

, Feig

, Matsuhisa

, Bao

. Rational design of capacitive pressure sensors based on pyramidal microstructures for specialized monitoring of biosignals. Adv Funct Mater. 2019;30(29):1903100.

33.

Wang

. Nanogenerators and piezotronics: From scientific discoveries to technology breakthroughs. MRS Bull. 2023;48:1014–1025.

34.

Yang

, Shi

, Li

, Tao

, Liu

, Wang

, Chen

. Monitoring the degree of comfort of shoes in-motion using triboelectric pressure sensors with an ultrawide detection range. ACS Nano. 2022;16(3):4654–4665.

35.

Zhao

, Bu

, Zhang

, Pang

, Li

, Zhang

, Liu

, Wang

, Zhang

. Intrinsically stretchable organic-tribotronic-transistor for tactile sensing. Research. 2020;2020:1398903.

36.

Liao

, Song

, Zhang

, Zhan

, Liu

, Wang

, Zheng

. Hetero-contact microstructure to program discerning tactile interactions for virtual reality. Nano Energy. 2019;60:127–136.

37.

Choi

, Kim

, Oh

, Jung

S-H

, Jung

, Sung

, Lee

. Highly stretchable, hysteresis-free ionic liquid-based strain sensor for precise human motion monitoring. ACS Appl Mater Interfaces. 2017;9(2):1770–1780.

38.

Zhou

, Huang

, Xiao

, Zhang

, Li

, Hu

, Feng

, Wu

, Zhu

, Mao

. Deep-learning-assisted noncontact gesture-recognition system for touchless human-machine interfaces. Adv Funct Mater. 2022;32(49):2208271.

39.

Wang

, Fei

, Wu

, Ye

, Qi

, Wang

, Zhao

, Zhang

, Qin

, et al. Wearable bending wireless sensing with autonomous wake-up by piezoelectric and triboelectric hybrid nanogenerator. Nano Energy. 2023;112: Article 108504.

40.

Qiu

, Sun

, Wang

, Shi

, Wang

, Ma

, Zhang

, Bao

, Tian

, Zhang

, et al. Nondestructive identification of softness via bioinspired multisensory electronic skins integrated on a robotic hand. npj Flex Electron. 2022;6(1):45.

41.

, Kong

, Yang

, Wang

, Pang

, Luo

, Yang

, Xu

. Machine learning-enabled tactile sensor design for dynamic touch decoding. Adv Sci. 2023;10(32):2303949.

42.

Yong-Lae

, Bor-Rong

, Wood

. Design and fabrication of soft artificial skin using embedded microchannels and liquid conductors. IEEE Sensors J. 2012;12(8):2711–2718.

43.

Stassi

, Canavese

, Cosiansi

, Gazia

, Fallauto

, Corbellini

, Pirola

, Cocuzza

. Smart piezoresistive tunnelling composite for flexible robotic sensing skin. Smart Mater Struct. 2013;22(12): Article 125039.

44.

Tombler

, Zhou

, Alexseyev

, Kong

, Dai

, Liu

, Jayanthi

, Tang

, Wu

S-Y

. Reversible electromechanical characteristics of carbon nanotubes underlocal-probe manipulation. Nature. 2000;405(6788):769–772.

45.

Bae

S-H

, Lee

, Sharma

, Lee

H-J

, Kim

J-H

, Ahn

J-H

. Graphene-based transparent strain sensor. Carbon. 2013;51:236–242.

46.

, Wei

, Tang

, Xiong

, Qin

, Luo

, Fang

, Wang

. Recent progress in Ti₃C₂T_x MXene-based flexible pressure sensors. ACS Nano. 2021;15(12):18880–18894.

47.

Niu

, Li

, Gao

, Li

, Wei

, Chen

, Yue

, Zhou

, Shen

. Perception-to-cognition tactile sensing based on artificial-intelligence-motivated human full-skin bionic electronic skin. Adv Mater. 2022;34(31): Article e2202622.

48.

Wang

, Jiu

, Nogi

, Sugahara

, Nagao

, Koga

, He

, Suganuma

. A highly sensitive and flexible pressure sensor with electrodes and elastomeric interlayer containing silver nanowires. Nanoscale. 2015;7(7):2926–2932.

49.

Bai

, Xue

, Chen

, Shi

, Zhang

, Hou

, Cheng

, Huang

, Wang

, et al. A robotic sensory system with high spatiotemporal resolution for texture recognition. Nat Commun. 2023;14(1):7121.

50.

Liu

, Wo

, Huang

, Huo

, Xu

, Zhan

, Li

, Zeng

, Jin

, Zhang

, et al. A flexible capacitive 3D tactile sensor with cross-shaped capacitor plate pair and composite structure dielectric. IEEE Sensors J. 2021;21(2):1378–1385.

51.

Zhang

, Lu

, Chen

, Liu

, Yao

. Flexible self-powered tactile sensors based on hydrothermally grown ZnO nanorods. IEEE Sensors J. 2022;22(13):12613–12621.

52.

Huang

, Qin

, Wang

, Xiang

, Zheng

, Lu

, Lv

, Wu

, Yan

, Wang

, et al. Piezoelectric nanogenerator for highly sensitive and synchronous multi-stimuli sensing. ACS Nano. 2021;15(12):19783–19792.

53.

Carluccio

, Fachechi

, Mastronardi

, Natta

, Algieri

, de Marzo

, Rizzi

, De Vittorio

. Fabrication, characterization, and signal processing optimization of flexible and wearable piezoelectric tactile sensors. IEEE Sensors J. 2023;23(10):10959–10969.

54.

Wang

, Peng

, Zhao

, Bao

, Li

, Tian

, Dong

, Shen

, Pan

. CdS@SiO₂ core-shell electroluminescent nanorod arrays based on a metal-insulator-semiconductor structure. Small. 2016;12(41):5734–5740.

55.

Hari

, Karumuthil

, Varghese

, Rajan

. Performance enhancement of flexible and self-powered PVDF-ZnO based tactile sensors. IEEE Sensors J. 2022;22(10):9336–9343.

56.

Liu

, Zheng

, Zhao

, Liu

, Yao

, Li

, Yiu

, Gao

, Avila

, Chirarattananon

, et al. Electronic skin from high-throughput fabrication of intrinsically stretchable lead zirconate titanate elastomer. Research. 2020;2020:1085417.

57.

Zou

, Zhang

, Guo

, Wang

, He

, Dai

, Zheng

, Chen

, Wang

, Xu

, et al. Quantifying the triboelectric series. Nat Commun. 2019;10(1):1427.

58.

Choi

, Lee

, Lin

, Cho

, Kim

, Ao

, Soh

, Sohn

, Jeong

, Lee

, et al. Recent advances in triboelectric nanogenerators: From technological progress to commercial applications. ACS Nano. 2023;17(12):11087–11219.

59.

Shi

, Yang

, Lei

, Liu

, Dong

, Tao

, Chu

, Wang

, Chen

. Eye tracking and eye expression decoding based on transparent, flexible and ultra-persistent electrostatic interface. Nat Commun. 2023;14(1):3315.

60.

, Fan

, Chen

, Nie

, Liang

, Tao

, Zhang

, Chen

, Fu

, Wang

. Manipulating the triboelectric surface charge density of polymers by low-energy helium ion irradiation/implantation. Energ Environ Sci. 2020;13(3):896–907.

61.

Tao

, Chen

, Yu

, Zeng

, Wu

, Jia

, Li

, Fu

, Chang

, et al. Ultra-sensitive, deformable, and transparent triboelectric tactile sensor based on micro-pyramid patterned ionic hydrogel for interactive human-machine interfaces. Adv Sci. 2022;9(10): Article e2104168.

62.

Huang

, Wang

, Li

J-a

, Zhang

, Li

, Ma

, Xin

, Yan

, Cheng

, He

, et al. High-performance flexible capacitive proximity and pressure sensors with spiral electrodes for continuous human–machine interaction. ACS Materials Lett. 2022;4(11):2261–2272.

63.

Yang

, Dutta

, Li

, Tiwari

, Zhang

, Niu

, Gao

, Erdely

, Xin

, Li

, et al. Iontronic pressure sensor with high sensitivity over ultra-broad linear range enabled by laser-induced gradient micro-pyramids. Nat Commun. 2023;14(1):2907.

64.

Zhang

, Wen

, Lei

, Gao

, Chen

, Zhang

, Liu

, Xie

, Sun

. Surface-microengineering for high-performance triboelectric tactile sensor via dynamically assembled ferrofluid template. Nano Energy. 2021;87: Article 106215.

65.

Choi

, Oh

, Kim

, Pyatykh

, Chang Yang

, Ryu

, Park

. Transparent pressure sensor with high linearity over a wide pressure range for 3D touch screen applications. ACS Appl Mater Interfaces. 2020;12(14):16691–16699.

66.

Bai

, Wang

, Deng

, Wang

, Lu

, Huang

, Li

, Zhang

, Yang

, et al. Graded intrafillable architecture-based iontronic pressure sensor with ultra-broad-range high sensitivity. Nat Commun. 2020;11(1):209.

67.

Yan

, Wang

, Xia

, Qiu

, Liu

, Wu

, Zhu

, Jia

, Zhu

, et al. Flexible high-resolution triboelectric sensor array based on patterned laser-induced graphene for self-powered real-time tactile sensing. Adv Funct Mater. 2021;31(23):2100709.

68.

Zhao

, Tang

, Yuan

, Li

, Dai

, Yao

, Zhang

, Ding

, Li

, Zhang

, et al. Large-scale integrated flexible tactile sensor array for sensitive smart robotic touch. ACS Nano. 2022;16(10):16784–16795.

69.

Chen

, Liu

, Ren

, Yang

, Shang

, Ma

, Zhang

, Jiang

, Sun

. Conformable core-shell fiber tactile sensor by continuous tubular deposition modeling with water-based sacrificial coaxial writing. Mater Design. 2020;190: Article 108567.

70.

Zhong

, Ming

, Jiang

, Ke

, Ding

, Li

, Jia

, Wang

. Full-textile human motion detection systems integrated by facile weaving with hierarchical core–shell piezoresistive yarns. ACS Appl Mater Interfaces. 2021;13(44):52901–52911.

71.

Huang

G-W

, Li

, Xiao

H-M

, Feng

Q-P

, Fu

S-Y

. A paper-based touch sensor with an embedded micro-probe array fabricated by double-sided laser printing. Nanoscale. 2017;9(27):9598–9605.

72.

Zhang

, Lu

, He

, Huo

, Zhou

, Han

, Jia

, Pan

, Wang

, Zhai

. Localizing strain via micro-cage structure for stretchable pressure sensor arrays with ultralow spatial crosstalk. Nat Commun. 2023;14(1):1252.

73.

Zhang

, Zheng

, Sun

, Zhou

, Xu

, Dai

, Mo

, Wang

. 3D printed, solid-state conductive ionoelastomer as a generic building block for tactile applications. Adv Mater. 2021;34(2):e2105996.

74.

, Zhou

, Ou

, Dai

, You

, Wang

, He

, Qin

, Wang

. Weavable and stretchable piezoresistive carbon nanotubes-embedded nanofiber sensing yarns for highly sensitive and multimodal wearable textile sensor. Carbon. 2020;170:464–476.

75.

Huang

, Wang

, Zhan

, Fang

. Flexible capacitive pressure sensor based on laser–induced graphene and polydimethylsiloxane foam. IEEE Sensors J. 2021;21(10):12048–12056.

76.

Luo

, Chen

, Tian

, Li

, Lu

, Liu

, Shao

. Gecko-inspired slant hierarchical microstructure-based ultrasensitive iontronic pressure sensor for intelligent interaction. Research. 2022;2022:9852138.

77.

Liu

, Wen

, Lei

, Gao

, Sun

. A liquid-solid interface-based triboelectric tactile sensor with ultrahigh sensitivity of 21.48 kPa⁻¹. Nanomicro Lett. 2022;14(1):88.

78.

, Zhou

, Chen

, Dai

, Li

, Xu

, Gao

, Wen

, Zhou

. In situ assembly of a wearable capacitive sensor with a spine-shaped dielectric for shear-pressure monitoring. J Mater Chem C. 2020;8(44):15634–15645.

79.

Park

, Lee

, Hong

, Lee

, Ha

, Jung

, Lim

, Kim

, Ko

. Tactile-direction-sensitive and stretchable electronic skins based on human-skin-inspired interlocked microstructures. ACS Nano. 2014;8(12):12020–12029.

80.

Chen

, Jing

, Lee

J-H

, Liu

, Kim

, Chen

, Huang

, Shen

, Zheng

, Yang

, et al. Human skin-inspired integrated multidimensional sensors based on highly anisotropic structures. Mater Horiz. 2020;7(9):2378–2389.

81.

Ren

, Nie

, Shao

, Lai

, Wang

, Chen

, Wang

. Fully elastic and metal-free tactile sensors for detecting both normal and tangential forces based on triboelectric nanogenerators. Adv Funct Mater. 2018;28(31):1802989.

82.

Yan

, Hu

, Yang

, Yuan

, Song

, Pan

, Shen

. Soft magnetic skin for super-resolution tactile sensing with force self-decoupling. Sci Robot. 2021;6(51):eabc8801.

83.

Shi

, Zhang

, Yang

, Shan

, Salam

, Lee

. Artificial intelligence of things (AIoT) enabled floor monitoring system for smart home applications. ACS Nano. 2021;15(11):18312–18326.

84.

Zhu

, Shi

, He

, Yi

, Ma

, Yang

, Chen

, Lee

. Self-powered and self-functional cotton sock using piezoelectric and triboelectric hybrid mechanism for healthcare and sports monitoring. ACS Nano. 2019;13(2):1940–1952.

85.

Nayak

, Li

, Tay

, Zamburg

, Singh

, Lee

, Koh

SJA

, Chia

, Thean

AV-Y

. Liquid-metal-elastomer foam for moldable multi-functional triboelectric energy harvesting and force sensing. Nano Energy. 2019;64: Article 103912.

86.

Zhu

, Sun

, Jia

, Huang

, Wang

, Li

, Chou

, Mao

. A 3D printing triboelectric sensor for gait analysis and virtual control based on human–computer interaction and the internet of things. Sustain For. 2022;14(17):10875.

87.

Zou

, Libanori

, Xu

, Nashalian

, Chen

. Triboelectric nanogenerator enabled smart shoes for wearable electricity generation. Research. 2020;2020:7158953.

88.

Zhang

, He

, Zhu

, Sun

, Shi

, Zhu

, Dong

, Yuce

, Lee

. Deep learning-enabled triboelectric smart socks for IoT-based gait analysis and VR applications. Npj Flex Electron. 2020;4(1):29.

89.

Shi

, Zhang

, He

, Sun

, Wang

, Feng

, Shan

, Salam

, Lee

. Deep learning enabled smart mats as a scalable floor monitoring system. Nat Commun. 2020;11(1):4609.

90.

Clough

, Duff

. The role of gesture in communication and cognition:Implications for understanding and treating neurogenic communication disorders. Front Hum Neurosci. 2020;14:323.

91.

, Huang

, Tian

, Wang

H-A

, Dai

G-Z

. Gesture interaction in virtual reality. Virtual Real Intell Hardw. 2019;1(1):84–112.

92.

, Hou

, Cui

, Shi

, He

, Sun

, Wang

, Chou

. Flexible PDMS-based triboelectric nanogenerator for instantaneous force sensing and human joint movement monitoring. Sci China Mater. 2019;62(10):1423–1432.

93.

Liang

, Li

, Wang

, Liu

, Ghannam

, Fioranelli

, Heidari

. Fusion of wearable and contactless sensors for intelligent gesture recognition. Adv Intell Syst. 2019;1(7):1900088.

94.

, Chen

, Li

, Pei

, Guo

. Flexible strain sensors for wearable hand gesture recognition: From devices to systems. Adv Intell Syst. 2022;4(2):2100046.

95.

Liu

, Zhang

, Zhu

. A wearable motion capture device able to detect dynamic motion of human limbs. Nat Commun. 2020;11(1):5615.

96.

Yang

, Liu

, Meng

, Xu

, Liu

, Jia

, Chen

, Qin

, Han

, Li

. Self-powered tactile sensor for gesture recognition using deep learning algorithms. ACS Appl Mater Interfaces. 2022;14(22):25629–25637.

97.

Yan

, Mi

, Lu

, Qin

, Wang

, Meng

, Liu

, Wang

, Cao

. Weaved piezoresistive triboelectric nanogenerator for human motion monitoring and gesture recognition. Nano Energy. 2022;96: Article 107135.

98.

Wang

, Zhao

, Wang

. A flexible iontronic capacitive sensing array for hand gesture recognition using deep convolutional neural networks. Soft Robot. 2023;10(3):443–453.

99.

Duan

, Lin

, Wang

, Tang

, Li

, Zhu

, Wu

, Tao

, Choi

C-H

, Sun

, et al. Conductive porous MXene for bionic, wearable, and precise gesture motion sensors. Research. 2021;2021:9861467.

100.

Wang

, Yan

, Wang

, Cai

, Gao

, Zeng

, Wan

, Wang

, Pan

, Yu

, et al. Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat Electron. 2020;3(9):563–570.

101.

Zhou

, Chen

, Li

, Zhang

, Wu

, Zhou

, Meng

, Sun

, He

, Fan

, et al. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays. Nat Electron. 2020;3(9):571–578.

102.

Tan

, Han

, Zou

, Qu

, Xue

, Li

, Wang

, Luo

, Cui

, Xi

, et al. Self-powered gesture recognition wristband enabled by machine learning for full keyboard and multicommand input. Adv Mater. 2022;34(21): Article e2200793.

103.

Moin

, Zhou

, Rahimi

, Menon

, Benatti

, Alexandrov

, Tamakloe

, Ting

, Yamamoto

, Khan

, et al. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat Electron. 2020;4(1):54–63.

104.

Kim

, Ha

, Kim

, Choi

, Won

, Jo

, Ko

. A deep-learned skin sensor decoding the epicentral human motions. Nat Commun. 2020;11(1):2149.

105.

Zhu

, Sun

, Chen

, Lee

. Low cost exoskeleton manipulator using bidirectional triboelectric sensors enhanced multiple degree of freedom sensory system. Nat Commun. 2021;12(1):2692.

106.

Guo

, Zhang

, An

, He

, Sun

, Pu

, Lee

. A wearable multidimensional motion sensor for AI-enhanced VR sports. Research. 2023;6:0154.

107.

Liu

, Zhang

, Wang

, Qin

, Yang

, Sun

, Hao

, Shu

, Liu

, Chen

, et al. A star-nose-like tactile-olfactory bionic sensing array for robust object recognition in non-visual environments. Nat Commun. 2022;13(1):79.

108.

Tang

, Liu

, Pan

, Zhang

, Xu

, Yao

, Zhang

, Tong

. Optical micro/nanofiber-enabled compact tactile sensor for hardness discrimination. ACS Appl Mater Interfaces. 2021;13(3):4560–4566.

109.

Chen

, Li

, Peng

, Huang

, Zhao

, Tang

, Guo

. Silver nanowire/polymer composite soft conductive film fabricated by large-area compatible coating for flexible pressure sensor array. J Semicond. 2018;39(1): Article 013001.

110.

Wang

, Que

, Chen

, Han

, Li

, Pan

, Wang

. Full dynamic-range pressure sensor matrix based on optical and electrical dual-mode sensing. Adv Mater. 2017;29(15):1605817.

111.

Wang

, Zhang

, Huo

, Li

, Que

, Peng

, Wang

, Pan

. A highly stretchable transparent self-powered triboelectric tactile sensor with metallized nanofibers for wearable electronics. Adv Mater. 2018;30(12): Article e1706738.

112.

, Liu

, Tan

, Wang

, Liu

, Feng

, Luo

, Li

, Wang

. Artificial tactile perception smart finger for material identification based on triboelectric sensing. Sci Adv. 2022;8(31): Article eabq2521.

113.

Wei

, Wang

, Wu

, Wang

. An open-environment tactile sensing system: Toward simple and efficient material identification. Adv Mater. 2022;34(29): Article e2203073.

114.

Wang

, Zhang

, Dong

, Han

, Du

, Zhai

, Pan

, Wang

. Self-powered high-resolution and pressure-sensitive triboelectric sensor matrix for real-time tactile mapping. Adv Mater. 2016;28(15):2896–2903.

115.

Shang

, Xu

, Liang

, Zhang

, Li

, Peng

. Multi-parameter e-skin based on biomimetic mechanoreceptors and stress field sensing. npj Flex Electron. 2023;7(1):19.

116.

, Zhao

, Yan

, He

, Wang

, Xiong

, Wang

, Bai

, Sun

, Lu

, et al. Multifunctional biomimetic tactile system via a stick-slip sensing strategy for human–machine interactions. npj Flex Electron. 2022;6(1):46.

117.

Shan

, Zeng

, Liu

, Zhang

, Li

, Yu

, Hu

, Guo

, Chen

. Artificial tactile sensing system with photoelectric output for high accuracy haptic texture recognition and parallel information processing. Nano Lett. 2022;22(17):7275–7283.

118.

Chun

, Choi

, Suh

, Bae

, Hyun

, Park

. A tactile sensor using single layer graphene for surface texture recognition. Nanoscale. 2017;9(29):10248–10255.

119.

Chun

, Kim

J-S

, Yoo

, Choi

, Jung

, Jang

, Lee

, Song

K-I

, Nam

, Youn

, et al. An artificial neural tactile sensing system. Nat Electron. 2021;4(6):429–438.

120.

Liu

, Wu

, Sun

, Guo

. Recent progress on tactile object recognition. Int J Adv Robot Syst. 2017;14(4):1–12.

121.

, Liu

, Wang

, Zhu

. Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition. Sci Robot. 2020;5(49): Article eabc8134.

122.

Sundaram

, Kellnhofer

, Li

, Zhu

, Torralba

, Matusik

. Learning the signatures of the human grasp using a scalable tactile glove. Nature. 2019;569(7758):698–702.

123.

Tremblay

, Le Franc

, Fleury

, Cogne

, Butet

, Barillot

, Lecuyer

, Bonan

. Influence of virtual reality visual feedback on the illusion of movement induced by tendon vibration of wrist in healthy participants. PLOS ONE. 2020;15(11): Article e0242416.

124.

Zhou

, Wang

, Yang

, Zeng

, Liu

. Mathematical model of fingertip skin under constant-current electrotactile stimulation. IEEE Trans Haptics. 2023;16(1):3–12.

125.

Kourtesis

, Argelaguet

, Vizcay

, Marchal

, Pacchierotti

. Electrotactile feedback applications for hand and arm interactions: A systematic review, meta-analysis, and future directions. IEEE Trans Haptics. 2022;15(3):479–496.

126.

Ankit

HTYK

, Nirmal

, Kulkarni

, Accoto

, Mathews

. Soft actuator materials for electrically driven haptic interfaces. Adv Intell Syst. 2021;4(2):2100061.

127.

Mazzotta

, Carlotti

, Mattoli

. Conformable on-skin devices for thermo-electro-tactile stimulation: Materials, design, and fabrication. Mater Adv. 2021;2(6):1787–1820.

128.

Wang

, Ohnishi

, Xu

. Multimodal haptic display for virtual reality: A survey. IEEE Trans Ind Electron. 2020;67(1):610–623.

129.

de Tinguy X, Howard T, Pacchierotti C, Marchal M, Lécuyer A. WeATaViX: WEarable Actuated TAngibles for VIrtual Reality eXperiences. In: Nisky I, Hartcher-O'Brien J, Wiertlewski M, Smeets J, editors. Haptics: Science, Technology, Applications. EuroHaptics 2020. Lecture Notes in Computer Science. Cham: Springer; 2020.

130.

Sonar

, Paik

. Soft pneumatic actuator skin with piezoelectric sensors for vibrotactile feedback. Front Robot AI. 2016;2:38.

131.

Wang

, Ding

, Lee

, Han

. Engineering liquid-vapor phase transition for refreshable haptic interfaces. Research. 2022;2022:9839815.

132.

Frediani

, Carpi

. Tactile display of softness on fingertip. Sci Rep. 2020;10(1):20491.

133.

, Gao

, Sun

, Yeo

, Lim

. Haptglove—Untethered pneumatic glove for multimode haptic feedback in reality–virtuality continuum. Adv Sci. 2023;10(25): Article e2301044.

134.

Song

, Kim

, Jin

, Kim

, Lee

, Kim

, Park

, Cha

. Pneumatic actuator and flexible piezoelectric sensor for soft virtual reality glove system. Sci Rep. 2019;9(1):8988.

135.

Sonar

, Gerratt

, Lacour

, Paik

. Closed-loop haptic feedback control using a self-sensing soft pneumatic actuator skin. Soft Robot. 2020;7(1):22–29.

136.

, Xie

, Yu

, Lee

, Vazquez-Guardado

, Luan

, Ruban

, Ning

, Akhtar

, Li

, et al. Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature. 2019;575(7783):473–479.

137.

, He

, Song

, Yao

, Wu

, Fu

, Liu

, Gao

, Zhou

, Wei

, et al. Miniaturization of mechanical actuators in skin-integrated electronics for haptic interfaces. Microsyst Nanoeng. 2021;7:85.

138.

Kim

, Hwang

, Song

, Lim

, Kim

, Choi

, Lee

, Kim

, Yang

, Pang

. A reversible, versatile skin-attached haptic interface platform with bioinspired interconnection architectures capable of resisting sweat and vibration. Adv Funct Mater. 2023; Article 2311167.

139.

Hwang

, Jeon

, Song

, Kim

, Lee

, Kim

, Jo

, Park

, Kim

, et al. A spatially selective electroactive-actuating adhesive electronics for multi-object manipulation and adaptive haptic interaction. Adv Funct Mater. 2023;34(6): Article 2308747.

140.

Folgheraiter

, Gini

, Vercesi

. A multi-modal haptic interface for virtual reality and robotics. J Intell Robot Syst. 2008;52(3–4):465–488.

141.

Kajimoto

, Kawakami

, Tachi

, Inami

. Smarttouch: Electric skin to touch the untouchable. IEEE Comput Graph Appl. 2004;24(1):36–43.

142.

Abbass

, Saleh

, Dosen

, Valle

. Embedded electrotactile feedback system for hand prostheses using matrix electrode and electronic skin. IEEE T Biomed Circ S. 2021;15(5):912–925.

143.

Sato K, Susumu T. Design of electrotactile stimulation to represent distribution of force vectors. Paper presented at: 2010 IEEE Haptics Symposium; 2010 Mar 25–26; Waltham, MA, USA.

144.

Komurasaki

, Kajimoto

, Ishizuka

. Fundamental perceptual characterization of an integrated tactile display with electrovibration and electrical stimuli. Micromachines. 2019;10(5):301.

145.

Tezuka

, Kitamura

, Tanaka

, Miki

. Presentation of various tactile sensations using micro-needle electrotactile display. PLOS ONE. 2016;11(2): Article e0148410.

146.

Lin

, Zhang

, Lee

, Li

, Hong

, Pan

, Zhang

, Peng

, Tan

, Zhang

, et al. Super-resolution wearable electrotactile rendering system. Sci Adv. 2022;8(36): Article eabp8738.

147.

Shi

, Wang

, Tian

, Li

, Fu

, Nie

, Lei

, Ding

, Chen

, Wang

. Self-powered electro-tactile system for virtual tactile experiences. Sci Adv. 2021;7(6): Article eabe2943.

148.

Kono

, Takahashi

, Nakamura

, Miyaki

, Rekimoto

. Design guideline for developing safe systems that apply electricity to the human body. ACM Trans Comput Hum Interact. 2018;25(3):1–36.

149.

Koo IM, Jung K, Koo JC, Nam J-D, Lee YK, Choi HR. Development of soft-actuator-based wearable tactile display. IEEE Trans Robot. 2008;24(3):549–558.

150.

Mun

, Yun

, Nam

, Park

, Lim

, Kyung

K-U

. Electro-active polymer based soft tactile interface for wearable devices. IEEE Trans Haptics. 2018;11(1):15–21.

151.

, Liu

, Cacucciolo

, Civet

, El Haitami

, Cantin

, Perriard

, Shea

. Untethered feel-through haptics using 18-μm thick dielectric elastomer actuators. Adv Funct Mater. 2020;31(39):2006639.

152.

Rothemund

, Kirkman

, Keplinger

. Dynamics of electrohydraulic soft actuators. Proc Natl Acad Sci USA. 2020;117(28):16207–16213.

153.

Chen

, Chen

, Yang

, Han

, Yao

. Skin-integrated stretchable actuators toward skin-compatible haptic feedback and closed-loop human-machine interactions. npj Flex Electron. 2023;7(1):1.

154.

Firouzeh

, Mizutani

, Groten

, Zirkl

, Shea

. Poptouch: A submillimeter thick dynamically reconfigured haptic interface with pressable buttons. Adv Mater. 2023;2307636.

155.

Leroy

, Hinchet

, Shea

. Multimode hydraulically amplified electrostatic actuators for wearable haptics. Adv Mater. 2020;32(36):2002564.

156.

Frediani

, Boys

, Ghilardi

, Poslad

, Busfield

JJC

, Carpi

. A soft touch: Wearable tactile display of softness made of electroactive elastomers. Adv Mater Technol. 2021;6(6):2100016.

157.

Grasso

, Rosset

, Shea

. Fully 3D-printed, stretchable, and conformable haptic interfaces. Adv Funct Mater. 2023;33(20):2213821.

158.

Park

W-H

, Shin

E-J

, Yoo

, Choi

, Kim

S-Y

. Soft haptic actuator based on knitted PVC gel fabric. IEEE Trans Ind Electron. 2020;67(1):677–685.

159.

Jimenez MC, Fishel JA. Evaluation of force, vibration and thermal tactile feedback in prosthetic limbs. Paper presented at: 2014 IEEE Haptics Symposium; 2014 Feb 23–26; Houston, TX, USA.

160.

Yang GH, Kyung KU, Srinivasan MA, Kwon DS. Quantitative tactile display device with pin-array type tactile feedback and thermal feedback. Paper presented at: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006; 2006 May 15–19; Orlando, FL, USA.

161.

Liu

Y-H

, Su

H-Y

, Lin

H-C

, Li

C-Y

, Wang

Y-H

, Huang

C-H

. Investigation of achieving ultrasonic haptic feedback using piezoelectric micromachined ultrasonic transducer. Electronics. 2022;11(14):2131.

162.

Reardon

, Dandu

, Shao

, Visell

. Shear shock waves mediate haptic holography via focused ultrasound. Sci Adv. 2023;9(9): Article eadf2037.

163.

Choi

D-S

, Kim

S-Y

. Transparent film-type vibrotactile actuator array and its haptic rendering using beat phenomenon. Sensors. 2019;19(16):3490.

164.

Lee

, Kim

, Sul

, Ko

. Thermo-haptic materials and devices for wearable virtual and augmented reality. Adv Funct Mater. 2020;31(39):2007376.

165.

Hong

, Lee

, Kwon

, Han

, Suh

, Cho

, Shin

, Yeo

, Ko

. Highly stretchable and transparent metal nanowire heater for wearable electronics applications. Adv Mater. 2015;27(32):4744–4751.

166.

Zhang

, Zhang

, Huang

, Wang

, Li

, Chen

, Wang

. Toward wearable cooling devices: Highly flexible electrocaloric Ba_0.67Sr_0.33TiO₃ nanowire arrays. Adv Mater. 2016;28(24):4811–4816.

167.

Shen

, Sun

, Hu

, Zhang

, Cheng

. Recent progress in exploring magnetocaloric materials. Adv Mater. 2009;21(45):4545–4564.

168.

Lee

, Sul

, Lee

, Pyun

, Ha

, Kim

, Park

, Eom

, Yoon

, Jung

, et al. Stretchable skin-like cooling/heating device for reconstruction of artificial thermal sensation in virtual reality. Adv Funct Mater. 2020;30(29):1909171.

169.

Cai S, Ke P, Narumi T, Zhu K. ThermAirGlove: A pneumatic glove for thermal perception and material identification in virtual reality. Paper presented at: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR); 2020 Mar 22–26; Atlanta, GA, USA.

170.

Chen

, Xiao

, Liang

, Li

, Jin

, Fukuda

. Giant elastocaloric effect with wide temperature window in an Al-doped nanocrystalline Ti–Ni–Cu shape memory alloy. Acta Mater. 2019;177:169–177.

171.

Zhang

, Yuan

, Cao

, Du

, Zhang

, Gui

. Latent heat thermal energy storage systems with solid–liquid phase change materials: A review. Adv Eng Mater. 2018;20(6):1700753.

172.

, Kim

, Lee

, Jeong

, Ko

, Bae

. A liquid metal based multimodal sensor and haptic feedback device for thermal and tactile sensation generation in virtual reality. Adv Function Mater. 2020;31(39).

173.

, Tong

, Huang

, Wu

, Yang

, Pan

. A thermal and vibrational feedback glove based on the tactile characteristics of human hand skin. IEEE Access. 2020;8:226671–226684.

174.

Howard

, Marchal

, Lecuyer

, Pacchierotti

. PUMAH: Pan-tilt ultrasound mid-air haptics for larger interaction workspace in virtual reality. IEEE Trans Haptics. 2020;13(1):38–44.

175.

Salazar

, Pacchierotti

, de Tinguy

, Maciel

, Marchal

. Altering the stiffness, friction, and shape perception of tangible objects in virtual reality using wearable haptics. IEEE Trans Haptics. 2020;13(1):167–174.

176.

Choi

, Yoon

, Lee

, Kim

, Lee

, Pang

. Conductive hierarchical hairy fibers for highly sensitive, stretchable, and water-resistant multimodal gesture-distinguishable sensor, VR applications. Adv Funct Mater. 2019;29(50):1905808.

177.

Zhang

, Wen

, Sun

, Guo

, He

, Lee

. Artificial intelligence-enabled sensing technologies in the 5G/internet of things era: From virtual reality/augmented reality to the digital twin. Adv Intell Syst. 2022;4(7):2100228.

178.

Kim

C-C

, Lee

H-H

, Oh

, Sun

J-Y

. Highly stretchable, transparent ionic touch panel. Science. 2016;353(6300):682–687.

179.

, Si

, Zhu

, Guo

, Zhang

, Pan

, Sun

, Pan

. Supercapacitive iontronic nanofabric sensing. Adv Mater. 2017;29(36):201700253.

180.

Gong

, Zhang

, Fang

, Sun

, Niu

, Lai

W-Y.

Wireless human–machine interface based on artificial bionic skin with damage reconfiguration and multisensing capabilities. ACS Appl Mater Interfaces. 2022;14(41):47300–47309.

181.

Jung

, Yoo

J-Y

, Vázquez-Guardado

, Kim

J-H

, Kim

J-T

, Luan

, Park

, Lim

, Shin

H-S

, Su

C-J

, et al. A wireless haptic interface for programmable patterns of touch across large areas of the skin. Nat Electron. 2022;5:374–385.

182.

Fan

, Guo

, Feng

, Lin

, Wang

, Liang

, Garrad

, Rossiter

, Zhang

, Lepora

, et al. Digital twin-driven mixed reality framework for immersive teleoperation with haptic rendering. IEEE Robot Autom Lett. 2023;8(12):8494–8501.

183.

Yao

, Zhou

, Huang

, Wu

, Yiu

, Li

, Huang

, Li

, Su

, Hou

, et al. Encoding of tactile information in hand via skin-integrated wireless haptic interface. Nat Mach Intell. 2022;893–903.

184.

Chi

, Sun

, Xue

, Li

, Liu

. Recent progress in technologies for tactile sensors. Sensors. 2018;18(4):948.

185.

Jung

, Kim

, Rogers

. Skin-integrated vibrohaptic interfaces for virtual and augmented reality. Adv Funct Mater. 2020;31(39):2008805.

Appendix

Less

Year 2024 volume 7 Issue 3

PDF

203

111

Cite this Article

BibTeX

Article Info

doi: 10.34133/research.0333

Receive Date：2023-12-02
Online Date：2025-07-24
Published：2024-03-23

Article Data

Affiliations

History

Received：2023-12-02
Accepted：2024-02-10

Funding

National Natural Science Foundation of China (61888102)

Beijing Natural Science Foundation(L223006)

China National Postdoctoral Program for Innovative Talents(BX20230466)

Affiliations

¹School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing 100081, China.

²Institute of Flexible Electronics, Beijing Institute of Technology, Beijing 102488, China.

Corresponding:

^* Address correspondence to: gzshen@bit.edu.cn

References

Share

https://castjournals.cast.org.cn/joweb/research/EN/10.34133/research.0333

Share to

Scan QR to access full text

Cite this article

BibTeX

Citations

表12种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT

Table 1. Characteristics of the four main mechanoreceptors in human skin. Reproduced with permission from [21,184]. Copyright, 2018, MDPI.

	Meissner corpuscle	Pacinian corpuscle	Merkel cell	Ruffini endings
Category	FA-I	FA-II	SA-I	SA-II
Adaptation rate	Fast	Fast	Slow	Slow
Location	Shallow	Deep	Shallow	Deep
Stimuli frequency (Hz)	10–200	70–100	0.4–100	0.4–100
Density (units/cm²)	140	30	70	10
Spatial resolution (mm)	3–4	>10	0.5	>7
Functional range	Object slip, light contact, texture	High-frequency, vibrations	Static forces with high resolutions	Tension deep in the skin and fascia
Receptive field (RF)	Small and sharp, 3–5 mm	Very large and diffuse, >20 mm	Small and sharp, 2–3 mm	Large and diffuse, 10–15 mm

Table 2. Summaries of machine learning in haptic sensing device

Category	Reference	Algorithms	Task	Application	Accuracy
Piezoresistive	[26]	CNN	Classification and regression	Body motion recognition	63.76%
	[68]	MLP	Classification	Hand trajectory	98.8%
	[93]	SVM	Classification	Hand gesture identification	92.67%
	[94]	RBFNN	Classification	Hand gesture identification	93.3%
	[100]	CNN	Classification, clustering, and dimension reduction	Gesture recognition	96.7%
	[104]	RSL	Classification and regression	Human action recognition	96.2%
	[107]	CNN	Classification, cluster, and dimension reduction	Object identification	96.9%
	[116]	XGB	Classification	Material recognition	93.3%
	[122]	CNN	Classification, regression, clustering, and dimension reduction	Object identification	89.4%
Piezoelectric	[119]	-	Classification	Texture recognition	99.1%
Triboelectric	[88]	1D-CNN	Classification	Body motion recognition	96.67%
	[101]	SVM	Classification and feature extraction	Hand gesture identification	98.63%
	[83]	1D-CNN	Classification	Body movement trajectory	85.67%
	[86]	ANN	Classification	Gait signal identification	-
	[89]	CNN	Classification	Human action recognition	91.47%
	[106]	1D CNN	Classification	Body motion recognition	97.5%
	[112]	LDA	Classification	Object identification	96.8%
	[113]	VGG	Classification	Material recognition	96.62%
Hybrid (piezoelectric and triboelectric)	[96]	CNN	Classification	Hand gesture identification	94.16%
	[102]	LDA	Classification	Human action recognition	92.6%
Magnetic	[82]	-	Classification, clustering, and dimension reduction	Human action recognition	-
Electromyography	[103]	AM	Dimension reduction	Hand gesture identification	97.12%
Photoelectric	[117]	ANN	Classification	Texture recognition	94.1%
Thermal conductive	[121]	MLP	Classification	Object recognition	94%

Table 3. Examples of haptic feedback devices summarized with their mechanism, material properties, design features, and input and output characters. Reproduced with permission from [183,185]. Copyright, John Wiley and Sons and Springer Nature.

Approach	Key mechanism	Reference	Materials and characters	Dimensions (mm)/weight	Array	Position	Output parameters (voltage/current)(displacement/force/frequency)	Resolution	Input (electric/mechanical signal)	Demonstration/application
Mechanical vibration	Pneumatic actuator	[130]	Air chamber created with silicone layers	D = 15T = 1.4–1.5	3 channels	Fingertip	F = 0.3 Nf = 5–100 Hz	10 mm	Pressure, 70 kPa	-
	Pneumatic actuator	[132]	PDMS membrane	L = 22T = 8	-	Fingertip	D = 6 mmF = 1 Nf = 0.1–10 Hz	-	Pressure, 20 kPa	Virtual control
	Pneumatic actuator	[133]	PDMS/Ecoflex	D = 9T = 1.5	-	Fingertip	F = 3.23 Nf = 10–160 Hz	-	Pressure, 100 kPa	Virtual control
	Pneumatic actuator powered by electrostatic force	[134]	Ecoflex	D = 15T = 5W = 0.57 g	-	Fingertip	D = 0.15 mmf = 0.2, 0.3, 1.0 Hz	-	Square wave, 0–6 kV	Virtual control
	Pneumatic actuator	[135]	Silicone elastomer	D = 10	-	Fingertip	D = 5 mmf = 0–100 Hz	-	Pressure, 30 kPa	-
	Flexible voice-coil magnetic actuator	[136]	Cu coil, NdFeB magnet with PDMS encapsulation	D = 18T = 4.0W = 1.4 g	32 channels	Arm, bac, opisthenar	D = ≈191–300 μmF = 135 mNf = 100–300 Hz	18 mm	Pulse, 5 V	Social media communication, feedback prosthetic, immersive games
	Magnetic actuator	[137]	Cu coil, magnet with PDMS encapsulation	D = 5T = ≈1.8	3 × 3	Fingertip, wrist	D = ≈1.55 mmf = 200 Hz	~5 mm	Sine wave, 0.01–0.5 V, 1.12–56.18 mA, 70–740 Hz, 1.4 mW	Dynamic Braille
Electrotactile	Electrical stimuli	[141]	Steel electrodes	D = 1	4 × 4	Fingertip	Pulse, 0.2 ms,I = 1.0–3.0 mAf = 30 Hz	2–4 mm	Pulse, 0.2 ms, 100–300 V, 1.0–3.0 mA, 30 Hz	-
	Electrical stimuli	[143]	Pin electrode (Cu)	D = 1	31 channels	Fingertip	Pulse, 20 μsI = 0.0–5.0 mAf = 210 Hz	2 mm	Pulse, 20 μs0.0–5.0 mAf = 210 Hz	Force vector identification
	Electrovibration and electrical stimuli	[144]	Cr electrode	18 × 0.9	-	Fingertip	I = 0–5 mA,f = 5–320 Hz	2 mm	0–600 V, 0–5 mA, 5–320 Hz	-
	Electrical stimuli	[145]	Microneedle electrode (Ti)	D = 0.3	1 × 5	Fingertip	f = 10–100 Hz	2 mm	10–100 V, 10–100 Hz	-
	Electrical stimuli	[146]	Ag/Cu electrode	D = 1	5 × 510 × 10	Fingertip, palm	f = 10 kHz	4 mm	Square wave,15–20 V, 10 kHz	Font Braille, electrotactile rendering
	Electrostatic discharge stimuli	[147]	Tin ball-shaped electrode	D = 0.5	21 channels	Fingertip,forearm	I = 25 μA	3 mm	Kinetic energy, body movement	Social media communication, dynamic Braille
Dielectric elastomer actuator (DEA)	DEA	[149]	8-layer silicone layer separated with flexible carbon electrode	D = 2T = 2	4 × 5	Fingertip	D = ≈471 μmF = 14 mN	3 mm	Square wave, 3,500 V, 0–100 Hz	-
	DEA	[150]	6-layer silicone layer separated with flexible Ag NW electrode	D = 16T = 0.7	2 × 3	Fingertip, forearm	D = ≈300–650 μmF = 50–250 mN	-	Sinusoidal wave, 3–4 kV, 0–500 Hz	-
	DEA	[151]	PDMS membrane with PMMA holder, carbon nanotube	T = 18 μmW = 1.3 g	-	Fingertip	f = 1–500 Hz	-	480 V	Haptic reader
	Electrohydraulic actuator	[153]	Dielectric oil, silicone membrane, dielectric shell and Ag electrode	D = 2	7 channels	Fingertip, palm	D = 500 μmP = 1 MPa	3 mm	0–4 kV	Feedback recognition of surface texture and object shape
	Electrohydraulic actuator	[154]	Dielectric oil, stretchable membrane, ITO electrode on PET holder	T = 500 μmL = 22	16 channels	Fingertip	D = 1.5 mmF = 4 Nf = 2 Hz	-	1–1.5 kV, 0–30 Hz, 6 mW	Feedback touchscreen
	Electrohydraulic actuator	[155]	Dielectric oil, PDMS membrane and aluminum electrode	L = 10	5 × 5	Fingertip, arm	D = 500 μmF = 300 mN	10 mm	Square bipolar signal, 0–1,400 v, 200 Hz	Normal and shear force feedback
	Electrohydraulic actuator	[156]	VHB dielectric film, silicone/carbon black composite electrode and insulating silicone grease	D = 12.5T = 5.5	-	Fingertip	F = 1 N	-	Square wave, 0.5–4 kV, 0.1–3 Hz	Various feedback identification
	Electrohydraulic actuator	[157]	Dielectric oil, carbon black electrode, PDMS encapsulation and PET substrate	T = 350 μmW = 1 g	2 × 3	Fingertip	D = 200 μmF = 40 mN	-	Sine waves, 4 kV, 1–1,000 Hz	Static haptic stimulation

Fig. 1. Haptic sensing and feedback techniques. Reproduced with permission from [88,101,114,136,147,155]. Copyright, Springer Nature, John Wiley and Sons, and the American Association of the Advancement of Science.

Fig. 2. Mechanoreceptors in human skin. Reproduced with permission from [21]. Copyright, 2018, MDPI.

Fig. 3. Mechanisms of four classic haptic sensing devices. (A) Piezoresistive. (B) Capacitive. (C) Piezoelectric. (D) Triboelectric.

Fig. 4. Tactile sensing devices of force sensors for VR applications. (A and B) Schematic illustration of the structure of pressure sensor arrays with micro-cage structures (A) and its application process (B). (I) Photographs of sensor arrays attached on palm. (II) Multiple sensors attached to the front of the finger to detect the different grasping postures. The voltage signal of the sensors at fingertip with a half hold apple (III) and fully grasped (IV). Reproduced with permission from [72]. Copyright 2023, Springer Nature. (C and D) Illustration of skin-inspired soft magnetic tactile sensor with working principle (C) and its application for an egg-grasping task (D). Reproduced with permission from [82]. Copyright 2021, the American Association for the Advancement of Science. (E and F) Schematic illustration of the intelligent sock sensor for monitoring plantar pressure (E) and the established digital human system with the sock senor (F). Reproduced with permission from [88]. Copyright 2020, Springer Nature.

Fig. 5. Tactile sensing devices of gesture recognitions for VR applications. (A to C) Schematic illustration of the triboelectric yarn sensor array (A), working principle (B), and the wearable sign-to-speech translation system (C). Reproduced with permission from [101]. Copyright 2020, Springer Nature. (D to F) Photograph of the gesture recognition wristband sensor (D), working principle (E), and gesture data processing (F). Reproduced with permission from [102]. Copyright 2022, John Wiley and Sons. (G to K) Schematic depicting the possible flow of body information with the epicentral sensor (G), illustration of measuring the epicentral motions of fingers (H) with the insert scanning electron microscopy (SEM) image of the cracked sensor (scale bar, 40 μm), photograph of the sensor conformably on skin (I), depiction of skin deformations for different finger bending motions (J), and photograph of the system for gesture duplication (K). Reproduced with permission from [104]. Copyright 2020, Springer Nature.

Fig. 6. Tactile sensing devices of body movements recognition for VR applications. (A to C) Schematic illustration of the exoskeleton sensory system for realizing the manipulation in virtual space and robotics (A), body kinetic analysis (B), and its application in controlling a virtual figure for immersive physical training (C). Reproduced with permission from [105]. Copyright 2021, Springer Nature. (D to G) Schematic manufacturing method of the scalable piezoresistive fibers (D), the fabricated textile sensor (E), the readouts of plantar pressure from the sock sensor (F), and various poses recorded by the sensor system (G). Reproduced with permission from [26]. Copyright 2021, Springer Nature.

Fig. 7. Tactile sensing devices of intelligent identification for VR applications. (A to E) Schematic illustration of a 16 × 16 pressure-sensitive triboelectric sensor matrix (A), SEM image of the etched PDMS surface microstructure (B), photograph of the sensor matrix (C), demonstration of the mapping output voltage of the sensor matrix under patterned pressure (D), and simulation results for the potential distribution (E). Reproduced with permission from [114]. Copyright 2016, John Wiley and Sons. (F and G) Schematic illustration of the biomimetic tactile system via a stick-slip sensing strategy (F) and the real-time object recognition system (G). Reproduced with permission from [116]. Copyright 2022, Springer Nature. (H and I) Schematic illustration of the artificial neural tactile sensing system (H) and the sensor responding to textured object (I). Reproduced with permission from [119]. Copyright 2021, Springer Nature.

Fig. 8. Tactile sensing devices of object identification for VR applications. (A to D) Schematic illustration of the scalable tactile glove for learning the signatures of human grasp (A), the individual locations of the sensor unit (B), the set of decomposed object-related pressure frames (C), and the circular plot of the relative correspondences between different parts of the hand (D). Reproduced with permission from [122]. Copyright 2019, Springer Nature.

Fig. 9. Haptic feedback devices of pneumatic actuator for VR applications. (A) Schematic illustration of the pneumatic glove (ii) consisting of PneuClutch (i) and PneuIndenter (iii) and its various applications. Reproduced with permission from [133]. Copyright 2023, John Wiley and Sons. (B to E) Working principle (B), photograph (C), operation image (D), and the promising VR system of the pneumatic actuator with fingertip feedback. Reproduced with permission from [134]. Copyright 2019, Springer Nature.

Fig. 10. Haptic feedback devices of magnetic actuator for VR applications. (A to C) Exploded-view schematic illustration of the wireless haptic interface (A) and its applications demonstrated in social media interactions (B) and prosthetic feedback (C). Reproduced with permission from [136]. Copyright 2019, Springer Nature. (D to H) Exploded view of the fingertip interface (D), the exploded actuator (E), photograph of the interface integrated on finger (F), and the correction (G) of blind test to recognize Braille patterns (H). Reproduced with permission from [137]. Copyright 2021, Springer Nature.

Fig. 11. Haptic feedback devices of ET interface for VR applications. (A to C) ET interface made of micro-needle electrodes (A) and the arrangement of the electrode (B) and test image of the ET interface on fingertip (C). Reproduced with permission from [145]. Copyright 2016, PLOS. (D and E) Photograph of the ET electrode of the rendering system (D) and ET stimulation mechanisms with negative and positive current (E). Reproduced with permission from [146]. Copyright 2022, the American Association for the Advancement of Science.

Fig. 12. Haptic feedback devices of ET interface based on electrostatic discharge. (A to F) Schematic illustration of the ET system to transmit virtual spatial pattern (A), exploded view of the electrode array (B), ET sensation based on electrostatic discharge (C), photograph of the flexible ET interface (D) integrated on arm (E), and the application of virtual tactile interaction (F). Reproduced with permission from [147]. Copyright 2021, the American Association for the Advancement of Science.

Fig. 13. Haptic feedback devices of DEA for VR applications. (A) Principle of DEA. (B to F) Photograph of the feel-through DEA film in fingertip (B), working principle (C), photograph of the haptic reading system with photodiode as a swatch (D), reading principle (E), and tactile reading demonstration (F). Reproduced with permission from [151]. Copyright 2020, John Wiley and Sons.

Fig. 14. Haptic feedback devices of electrohydraulic actuator for VR applications. (A to C) The principle of the submillimeter-thick electrostatic actuator (A), photograph of off/on state (B), and illustration when powered by segmented electrode to generate in-plane forces (C). Reproduced with permission from [155]. Copyright 2020, John Wiley and Sons. (D to F) Schematic of the 3D-printed flexible haptic interface (A), photograph attached on finger (E), and curved surfaces (F). Reproduced with permission from [157]. Copyright 2023, John Wiley and Sons.

Fig. 15. Thermal feedback devices for VR applications. (A and B) An exploded view of the thermal haptic interface (A) and integrated haptic glove (B). (C and D) Virtual scenarios when grabbing (C) and picking up (D) a metal ball and real-time temperature images by infrared camera. Reproduced with permission from [172]. Copyright 2020, John Wiley and Sons.

Fig. 16. VR applications of haptic sensors and feedback devices. (A to E) Application examples of haptic sensors in virtual input (A), force mapping (B), and gesture capturing (C), as well as haptic feedbacks in immersive entertainment (D), feedback prosthesis (E), and tactile interaction (F). Reproduced with permission from [7,136,178,179,181,183]. Copyright, the American Association for the Advancement of Science, John Wiley and Sons, and Springer Nature.

Articles: Latest Articles; Most Read; Collections

Updates: Events; News; Multimedia

About: About Us

Contact

No. 86 Xueyuan South Road, Haidian District, Beijing

100081

010-62199257

qkjq@cast.org.cn

Copyright © 2025 China Association for Science and Technology. All rights reserved. For all open access content, the relevant licensing terms apply.
Sponsored by the Office of the Leading Group for Cybersecurity and Informatization of CAST, and supported by Science and Technology Review Publishing House