Braden McGratha, Angus Rupertb (2019). Tactile Displays: From the Cockpit to the Clinic. In: Advances in Cognitive Engineering and Neuroergonomics. AHFE Open Access, vol. 7. https://doi.org/10.54941/ahfe100210
There is a great deal in common between a pilot maintaining controlled flight in aviation and a person maintaining balance while standing upright. Both activities require sufficient and accurate sensory information provided in a format permitting the central nervous system to respond quickly with appropriate corrective motor actions to prevent a fall or a mishap. Loss of control in either environment is a frequent occurrence when sensory information is compromised such as in degraded visual environments (DVE), or when a patient experiences loss of sensation. We developed an alternative sensory cueing device using the sense of touch, which can prevent loss of control in both the aviation and terrestrial environments. A single belt with tactile transducers provides touch cues concerning direction, velocity, and extent of movement. This continuous, intuitive, orientation information to pilots and patients reduces the likelihood of a loss of aircraft control or a fall. The algorithms providing tactile cues are similar for a pilot hovering a helicopter in DVE and a sensory-compromised patient performing balance tasks. The rapidity of learning correct responses to tactile cues reflects the intuitive nature of tactile cueing.
For all mammals, the act of maintaining upright posture requires coordinating the inputs from multiple sources of sensory information with a complex set of reflexes developed over millions of years of evolution. Normally the sensory information for simple standing is provided by three systems: the somatosensory information from skin, muscle, and joint receptors; the gravity/acceleration sensing information from the vestibular organ; and vision. There is some redundancy in these systems, since vision and the gravity sensing organs can both provide the direction of “down.” Although it is possible to stand and maintain upright posture with the eyes closed or in the absence of a vestibular organ, standing and walking is not possible without the presence of a functioning skin muscle and joint sensory system. The system of balance is normally extremely reliable and robust since the system has been honed through the evolutionary process to perform flawlessly in the terrestrial environment to which it has been exposed for millions of years. Furthermore, the balance and locomotion system is an automated system operating at a very low level of attention, permitting the animal to devote full attention to survival-related tasks. The problem occurs when we leave “terra firma” and enter the aerospace environment or if our balance and locomotion system is compromised by disease, injury, and/or aging.
The largest contributor to aviation mishaps from the beginning of aviation to the present has been pilot error. Spatial disorientation (SD) and loss of situation awareness (SA) account for the greatest number of fatalities (McGrath et al., 2002). Why has it been so difficult for pilots to maintain orientation and stability (i.e. balance) of their aviation platform when they rarely become disoriented or lose balance on earth? This is because maneuvering in the aerospace environment causes our two most basic sensory orientation systems to provide false but corroborating information as to the direction “down.” The vestibular and somatosensory (skin-muscle-joint) sensors provide accurate, reliable information in the terrestrial environment where they are not subjected to unusual or prolonged accelerations. These two systems are collectively referred to as our proprioceptive sensory system and are colloquially called the “seat of the pants” sensation by pilots and astronauts. The proprioceptive system provides accurate orientation information continuously concerning the direction down and this information rarely comes to our attention. Although we are unaware of this information, it is used by the brain automatically to adjust postural, locomotive, and vegetative reflexes associated with orientation. However, in the aerospace environment, the proprioceptive senses frequently provide false information as to the direction down and cannot detect constant velocity motion without the assistance of visual information. An example of the latter is shown in the diagram below.
Figure 1. Pilot misperceiving orientation in degraded visual environment. (Adapted from Benson, 1999b).
The aircraft is in a constant rate of turn at a fixed altitude. The aircraft and pilot are subjected to two forces – gravity directed towards the center of the earth and a centrifugal force associated with the turn. The two “seat-of-the-pants” senses (vestibular and skin-muscle-joint) use the resultant force to determine the direction “down,” which does not match the direction of gravity. Unless the pilot can see the horizon or consciously direct attention to the attitude indicator instrument, the pilot will perceive that he/she is in straight and level flight when in reality the pilot and aircraft are banked in a turn.
Pilots must frequently (every few seconds) scan their instruments or refer to the outside horizon to update their attitude especially under dynamic flight conditions. When visibility becomes degraded and pilots are distracted from the instruments, the visual system fails to upgrade the true attitude of the aircraft while the brain automatically continues to compute orientation with the only information available -- the inner ear and skin-muscle-joint systems -- which are providing false information as to the direction “down.” It only takes a short period of time to drift into an attitude from which the pilot cannot recover in time or the aircraft impacts the ground before the pilot becomes aware of his unusual unsafe attitude.
These and related types of unrecognized SD mishaps have plagued pilots from early aviation to the present and continue to occur to well-trained pilots at all levels of experience and currency. Virtually all SD mishap reports contain the same comment – “The pilot(s) failed to maintain an adequate cross-check of the instruments.”
In contrast to the vestibular and skin-muscle-joint senses, the sense of vision is discontinuous in the flight environment especially under DVE conditions. It only provides orientating information when our eyes are open and there are vertical or horizontal referenced structures (e.g cockpit attitude indicator) present in the visual fields to define the direction “down”. In isolation, vision generally fails to provide adequate orientation information when we are directing our attention to a specific object, or when we are distracted in high workload environments of the cockpit.
The solution by early human factors engineers to solve the problem of inaccurate information from the vestibular and somatosensory systems was not to provide corrective orientation information via one or more of these systems, but to provide accurate flight visual information of pitch and roll attitude, visual information as to heading, visual information of altitude, visual information of vertical velocity, and other visual instruments. Pilots learn to develop an instrument scan whereby they quickly look at each instrument just long enough to gather the information from that instrument before moving on to the next instrument and then mentally putting together an image of their orientation and SA. This collection of instruments must be scanned frequently and, in highly dynamic conditions, can easily overwhelm one or both pilots especially when there are other mission related cockpit tasks that require attention.
When these visual displays were demonstrated in the late 1920s, it was believed that SD mishaps would come to an end since all of the information required to fly and land safely was now available on several cockpit instruments. When SD mishaps continued, the human factors engineers developed Heads Up Displays (HUD) with all of the key orientation information available on the HUD. This was followed by Helmet Mounted Displays (HMD) so that orientation information would be always available in the field of view. Unfortunately SD mishaps continued despite the evolutionary development of visual displays. HUDs and HMDs, which present a wide range of information in addition to attitude information, have not proven to be the solution for SD as was once envisioned.
What are the deficiencies of visual displays? Visual information is discontinuous and only available when the pilot consciously directs attention to the display, which means that he/she is not attending to other tasks. More importantly, abstract, symbolic visual information requires cognitive effort to interpret the information as opposed to the “seat-of-the-pants” senses (somatosensory and vestibular), which are processed in an automated fashion requiring little to no cognitive workload. Just as in the late 1920s, modern sophisticated displays provide no orientation information when the pilots’ attention is directed to other visual tasks including weapons systems and navigation. The result is that pilot workload to maintain orientation has actually increased (Gillingham, 1992). During conditions of high workload, the visual scan often breaks down resulting in SD. Alternatively, disorientation may occur during long periods of routine flight when pilots become bored, allowing the instrument scan to deteriorate.
The other human factors engineering solution to SD and loss of SA is to remove the pilot from the loop using automation to control the aircraft. This solution also creates some problems, especially when the pilot is unexpectedly required to assume manual control since there is a period of time required to reacquire the knowledge of aircraft flight parameters including attitude. Pilots routinely train for this situation with practice of a maneuver known as “recovery from unusual attitudes” in which the check pilot places the aircraft in an unusual attitude while the pilot in training has his/her eyes closed until instructed to recover the aircraft. U.S. Air Force studies have shown that the average pilot requires approximately 26 sec to regain control of the aircraft when the aircraft is placed into an unusual attitude (Krause, 1959).
HUMAN SYSTEMS INTEGRATION DEVICE TO ENHANCE BALANCE PERFPRMANCE Aviation Solution to “Balance” of Helicopters
Over the past 20 years, military and emergency medical services (EMS) helicopters have experienced high rates of mishaps associated with the inability to hover safely in DVE. Hovering a helicopter is a complex control task, because similar to the human balance condition, helicopters are inherently unstable in hover. A system is said to be stable if it can recover from small changes to its position. Most fixed-wing aircraft are inherently stable, so that if you take your hands off the controls, they will continue flying, at least for a short time. However, this is not the case for helicopters, which are unstable in hover. To maintain a hover, it is critical that the pilot can detect small changes in the aircraft’s altitude or attitude during all environmental and operational conditions and then make constant input on the controls. In DVE, rescue operations, and other high workload situations, occasionally the pilot is unable to detect these small changes and mishaps can eventuate.
To solve this “balance” problem, especially when the pilot cannot visually detect small changes in the aircraft’s altitude or attitude via outside visual cues or cockpit displays, we developed an alternative hover capability using tactile cueing to provide drift velocity information that is continuous, intuitive, and concordant with the visual displays (McGrath, 2000; Rupert, 2000). These tactile cueing systems have been demonstrated in both simulator and in-flight tests. (McGrath, 1998; Rupert, 2000; Jennings et al., 2004; Schultz et al., 2009; Kelley, 2013)
Figure 2. Eight tactor belt.
The current system uses an 8-tactor belt worn on the waist and provides to the helicopter pilot information derived from the aircraft sensors concerning the direction of helicopter drift. Standard tactor placement in the torso belt is umbilicus, center of back, left and right sides, and at the midpoints between these four cardinal locations. The direction of the helicopter drift vector in the azimuth is provided by the corresponding tactor on the torso within 8 equal segments of 45 degrees. The magnitude of the aircraft velocity vector is provided by three increasing pulse frequencies to indicate increasing velocity. Additional tactors placed in the seat and shoulder strap provide altitude information to indicate whether the pilot has gone below or above desired altitudes. Together the tactors permit a pilot to maintain a hover non-visually at a selected point in space. Flight paths and maneuvers can be directed by sequentially moving the selected point. Helicopter pilots who are current and proficient can learn within 10 minutes to maintain a hover non-visually. The task of hovering non-visually requires attention for the first several hours until it becomes a task similar to standing.
Clinical Solution to Balance Dysfunction Following a feasibility study (Rupert, 2010) to determine the effectiveness of the aviation-developed tactile cueing system to provide enhanced balance to patients, the Department of Defense provided funding for two small companies via the SBIR (Small Business Innovative Research) program to develop hardware for commercial applications, some of which is shown below.
Figure 3. Patient standing on force plate using tactile feedback to improve balance performance.
In Figure 3, the patient receives information via the tactile belt concerning deviations of center of pressure (COP) information as measured by the force platform on which he is standing. Data is transferred from the measuring platform to the battery powered vibrating tactors in the belt via a computer software interface and RF controller (see Figure 4). The same location of tactors is used as with the aircraft platform. The tactile stimulus frequency is increased as the patient COP deviates farther from the null condition of normal upright stance. The COP information is also displayed to the patient via a visual display situated in front of the patient.
Figure 4. Force platform is shown on left and remote controlled belt on right.
A schematic depiction of the cue is shown in Figure 5. As the patient sways forward, the initial center of pressure (black dot) moves forward on the platform (shaded dot), which triggers the forward tactor to activate, much as a rumble strip alerts a car driver when he is running off the road.
Figure 5. Depiction of tactile cue to patient.Figure 6. Visual display presented to patient.
In addition to the tactile cue, the patient is provided a visual representation of his position (Center of Pressure) on the force plate represented as a moving white ball (see Figure 6). Each of the 8 segments corresponds to a tactor channel on the vibrotactile belt. When the ball crosses out of the dark blue null zone into the grey zone the corresponding tactor activates. The size of the null zone can be set by the physiotherapist to match the skill level of the patient. Depending on the patient’s performance, the visual display may or may not be added to the tactile cueing stimulus.
The first study conducted to test the benefits of tactile cueing for balance rehabilitation was a dissertation by a physiotherapist (Atkins, 2010). In this study, 30 elderly balance patients derived a beneficial effect in the device intervention group, which showed improved Berg Balance Scale (BBS) and Dynamic Gait Index (DGI) scores after therapy (versus the control group) and better sit-to-stand performance. The BBS and DGI are standard measures of performance used by physiotherapists to assess balance function and locomotion.
The tactile cueing system for patients has been undergoing clinical trials for the past two years. The graph below shows preliminary evidence of treatment (versus control) improvement (i.e., change from baseline to the middle of the rehabilitation schedule) for three established functional clinical tests of balance (BBS, Functional Assessment Battery [FAB], DGI). This data is from a Phase II SBIR data collection site. As shown in Figure 7, the T1-T2 changes (from baseline versus testing at the mid-point of treatment at 6/12 PT sessions completed) in the treatment condition were significantly greater (versus changes in the control condition over the same period) in all tests (BBS, FAB, and DGI) at t ≥ 2.75 and one-tailed p < /= 0.017, n = 9).
Figure 7. Balance performance with tactile cueing (Treatment) or traditional therapy (Control).
DISCUSSION
The performance of the hover maneuver in a helicopter and standing upright are relatively equivalent tasks in that both involve dynamical unstable systems in which humans are constantly making small corrective actions to maintain control. The human balance system is so unstable that without continuous sensory inputs to provide information concerning slight shifts in center of gravity or center of pressure on the feet we will fall within seconds. Similarly the helicopter will depart from controlled hover without almost continuous corrective inputs. It has been necessary for control engineers to develop stability augmentation systems to reduce the workload of hovering so that pilots do not become fatigued. The similarity of the hovering and upright balance tasks from a central nervous system control response is also reflected in that the improvements made by pilots and patients alike, is accomplished using the same algorithm for tactile cueing.
Currently, patients can improve balance performance by repeated practice with tactile cueing. Ultimately, the goal to assist patients with dysfunctional balance is to incorporate the tactile cueing into an ambulatory system that will prevent falls while walking. The problem is that walking is essentially a controlled fall repeated over and over with each step. Normal walking requires a great deal of information regarding limb position and pressures on the soles of the feet. What will be required for this patient prosthesis to perform well is the knowledge of limb movement, terrain information, intention of movement, and highly advanced central programming, which are also the same issues faced by the engineering community when dealing with control of robots and exoskeletons.
RECOMMENDATIONS Aviation Applications
To fully achieve the potential of tactile displays in aviation, the further development, testing, and evaluation of the following technology areas and the human factors implications are recommended:
Integration of tactile instruments with HMD and 3D audio displays.
Tactor miniaturization and integration with flight garments.
Development of “smart” tactile algorithms to enable intelligent switching between various modes of SA information.
Demonstration of tactile cueing as a tool to assist SA with increasing levels of automation that remove pilots from active control of the platform.
Advanced modes such as threat warning and targeting which have been developed in the laboratory, but not yet tested in flight and/or in the field. The target presented on a radar screen requires visual attention and cognitive processing of the information before the operator decides where to direct his gaze to acquire the target visually. Users (e.g., pilots, soldiers) can be cued to the location of targets tactually and thus improve their performance.
Clinical Applications Research and development of tactile displays for military aviation will lead to better tactors and improved understanding and methods of presenting tactile information. This will directly impact and improve tactile displays for providing visual information for blind people, hearing information for deaf people, and as a vestibular prosthesis for people with balance disorders.
Many people with balance disorders benefit from treatment, but there is a significant number for whom treatment is not beneficial or who only obtain partial relief of their symptoms. However, there are many that could benefit in some way from a balance prosthesis. A tactile display, coupled with real time COP and COG and sole of feet information could provide a balance prosthesis that will re-supply motion information to those who have lost it because of inner ear disease or other sensory deficits.
This technology would also assist developers in the areas of ambulatory prostheses and humanoid robots. Patients using exoskeletons can be provided tactile cueing to be more aware of motion.
Space Applications In space, astronauts are denied any sensory reference to indicate “down” and the only sensory indication of orientation is currently provided by vision. Tactile displays could be interfaced with an inertial reference system to give astronauts a continuous perception of “down.” In Extra Vehicular Activities (EVA), tactile displays could improve safety and performance by presenting a constant point of reference, such as the airlock of the international space station and/or a companion astronaut. Not knowing the direction of an airlock, especially in an emergency or a collision with another astronaut could have fatal consequences.
CONCLUSIONS
Tactile cueing has the capability to assist in the complex task of balance to include maintaining upright posture or to assist pilots to continuously and intuitively maintain hover control in helicopters. Tactile cueing improves spatial orientation especially in high workload environments. Tactile displays may assist with problems of automation by maintaining pilots in the perceptual control loop.
In conclusion, tactile displays have the potential to save lives and increase performance in many different areas. Tactile stimulation has been identified as a very useful, low bandwidth communication modality when audio and visual capabilities are challenged or compromised, but has failed to get to market because of cumbersome human interfaces. Human interface has been the key challenge to deployment. Further research and development is warranted to fully realize the potential of this novel and intuitive display technology.
REFERENCES
Atkins, K. L. (2010). “Vibrotactile Postural Control in Patients that have Sit-to-Stand Balance Deficit and Fall”. ( Report No.
2010-18). Fort Rucker, AL: U.S. Army Aeromedical Research Laboratory. Benson, A. J. (1999). “Spatial disorientation - common illusions”. In J. Ernsting, A. N. Nicholson, D. J. Rainford (Eds.), Aviation medicine (3rd ed., pp. 437-481). Oxford: Butterworth Heinemann. Gillingham K. K. (1992). “The spatial disorientation problem in the United States Air Force”. J Vestib Res, 2, 297-306.
Jennings, S., Craig, G., Cheung, B., Rupert, A., & Schultz, K. (2004). “Flight test of a Tactile Situation Awareness System in a land-based deck landing task”. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, Santa Monica, CA, 142-146. Kelley, A. M., Cheung, B., Lawson, B. D., Rath, E., Chiasson, J., Ramiccio, J. G., & Rupert, A. H. (2013). “Tactile cues for
orienting pilots during hover over moving targets”. Aviation Space and Environmental Medicine, 84, 1-7. Krause, R. N. (1959). “Disorientation: An evaluation of the etiologic factors”. Air University, School of Aviation Medicine USAF, Brooks Air Force Base TX McGrath, B. J, Suri, N., Carff, R., Raj, A. K., & Rupert, A..H. (1998). “The Role of Intelligent Software in Spatial Awareness
Displays”. Proceedings for 3rd Annual Symposium and Exhibition on Situational Awareness in the Tactical Air Environment, Piney Point, MD. McGrath, B. J. (2000). “Tactile Instrument for Aviation”. (Report No. NAMRL-10-2000), Pensacola FL: Naval Aerospace Medical Research Laboratory. McGrath, B. J, Guedry, F. E, & Rupert, A. H. (2002). “Analysis of Spatial Disorientation Mishaps in the US Navy”.
Proceedings for NATO Symposium on Spatial disorientation in military vehicles: Causes, consequences and cures. April 2002, A Coruna, Spain. McGrath, B. J., Estrada, A., Braithwaite, M. G., Raj, A. K., & Rupert, A. H. (2004). “Tactile situation Awareness System Flight Demonstration Final Report” (Report No. 2004-10). Fort Rucker, AL: U.S. Army Aeromedical Research Laboratory. Rupert, A. H., Mateczun, A. J., Guedry, F. E. (1990) “Maintaining spatial orientation awareness”. In Situational Awareness in
Aerospace Operations, Copenhagen Denmark, 2-6 Oct 1990. AGARD CP-478: 21-1 to 21-5. Neuilly-Sur-Seine, France: Advisory Group for Aerospace Research and Development (AGARD). Rupert, A. H., (2000), “Tactile situation awareness system: proprioceptive prostheses for sensory deficiencies”. Aviation Space and Environmental Medicine, 71(9 Section 2Supplement):A92-A99. Rupert, A.H. (2000), “An instrumentation solution for reducing spatial disorientation mishaps”. IEEE Eng Med Biol Mag 19(2):
p. 71-80 Rupert, A.H., and Lawson, B.D. (2010), “Initial consideration of the feasibility and optimal application of tactile sway cueing to improve balance among persons suffering from disequilibrium”. USAARL Technical Report No. 2011-01, 15 pages. Schultz K., McGrath B.J., Cheung B., Craig G., Jennings S., Rupert A.H. (2009), “In flight evaluation of tactile situation
awareness system during high hover and simulated shipboard landing”. In: Proceedings of American Institute of Aeronautics and Astronautics conference on Guidance, Navigation and Control, Chicago, IL, USA
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Cognitive Engineering and Neuroergonomics
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2101-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional