Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Tactile Displays for Cueing Self-Motion and Looming: What Would Gibson Think?

U.S. Army Aeromedical Research Laboratory 6901 Farrel Road, Fort Rucker AL 36362 1 ↗
Publication year
Digital Object Identifier10.54941/ahfe100209

Abstract

James J. Gibson pioneered an approach to perception that treats stimuli as rich in information and considers perception to be tightly coupled with action, based upon the possibilities for action encountered in the environment. The relation between whole-field optical flow and the visual control of self-motion towards or away from significant objects has been central to the development of ecological psychology. This paper considers recent developments in tactile research and conjectures what Gibson might have to say about employing tactile displays to convey self-motion and/or looming. The author considers the ecological functions of touch (versus vision and hearing), especially in regards to the perception of self-motion and looming. Self-motion and looming are then contrasted to develop general principles for specifying these two percepts via touch, partly using Gibson’s visual kinesthesis principles as a point of departure. A few initial studies (by the author and others) are highlighted which imply that appropriately-designed tactile displays may augment the perception of self-motion and looming. The most successful tactile display designs are likely to be analogous to the natural stimuli specifying self-motion or looming in the tactile, visual, and/or auditory domains.

INTRODUCTION

James Jerome Gibson (1904-1979) was the pioneer of ecological psychology, which is an approach to perception that stresses the importance of the environment, focusing upon ambient, naturalistic stimuli, rather than isolated, static laboratory stimuli, as had been studied previously under conditions of tight control but limited context and generalizability. This more naturalistic approach emerged from Gibson’s applied military research on depth/distance perception problems in aviation, and his realization that the prevailing approach to perception had failed to predict the success of pilot candidates or make it easier for them to learn to fly (Gibson, 1979). This realization led Gibson away from a focus on perception of static visual objects in abstract space and towards a focus on dynamic visual motion of background features of the environment, such as terrain features, the horizon, and the relation of the ground to the sky. Contrary to much of the earlier empirical work in perception, Gibson developed a theoretical framework that considers stimuli to be rich (rather than impoverished), and seeks direct explanations for human per- ception and behavior that do not require extensive reliance upon learning and memory.2 Finally, ecological psychol- ogy considers perceptions and actions to be tightly coupled, based upon affordances, which are the possibilities for action that the environment offers to the perceiving organism while it moves through the world fulfilling its survival needs (for food, safety, etc.). While such concepts at first appeared to be more philosophical than scientific, Gib- 1 Views and findings expressed here are solely those of the author and should not be construed as positions, policies, or decisions of the U.S. Government nor any of its departments. Citation of trade names does not imply endorsement or approval of the use of such commercial items by the U.S. Government nor any of its departments. The author thanks the U.S. Army Medical Research and Materiel Command for its sponsorship and Deahndra Grigley for her assistance in editing the manuscript, through her ap- pointment to USAARL via the Oak Ridge Institute for Science and Education. 2 This has been summarized as a “bottom-up” approach to perception, albeit Gibson himself would disagree. Cognitive Engineering and Neuroergonomics (2019) son’s approach stimulated research that made quantitative contributions to visual psychophysics and indirectly to computational modeling of visual perception (Nakayama, 1994).

Central to Gibson’s approach is the influence of whole-field optic flow on the perception and control of self-motion, which has been applied to a number of related problems, including the control of one’s heading, distance, and time- to-contact with looming objects. Such concepts were described by Gibson starting in the 1950s, but the best single source is Gibson’s final book (Gibson, 1979). Since Gibson’s influential work on visual control of self-motion was carried out, the understanding of non-visual displays has advanced considerably. This paper considers tactile display design in the light of ecological psychology. The focus is on the use of tactile displays to convey self-motion (move- ment of the body through space) and looming (the approach of objects towards oneself). The question under consid- eration is: “If Gibson were alive today and extending his approach to the tactile modality, what might he think about the feasibility and optimal design of modern tactile displays for conveying self-motion and/or looming?” The present paper seeks to answer this question by considering the natural or ecologically realistic survival functions of vision, audition, and touch, especially in regards to the specification of self-motion and looming. The author then contrasts self-motion and looming to derive principles for specifying these two different but related percepts across non-visual modalities using Gibson’s visual kinesthesis3 principles (e.g., Gibson, 1979) as a framework for reason- ing. Finally, the author presents examples from his own recent research and the literature where self-motion or looming cues (including some that we know work in the visual or auditory modalities) appear to work in touch dis- plays, and notes the ecological validity of each successful example. By ecological validity, the author means the ex- tent to which the cues have been provided in a way that intuitively approximates how they would reach a person nat- urally (through touch or via analogies derived from another sense), i.e., during daily activities in the natural environ- ment in which humans evolved. This concept is highly relevant to usability and human-machine interactions (Flach, Hancock, Caird, and Vincente, 1995), and overlaps with established human factors concepts, such as congruence and compatibility (Wickens and Carswell, 2006).

SOME NATURAL FUNCTIONS OF VISION, AUDITION, AND TOUCH Vision, audition, and somatosensation (cutaneous and kinesthetic senses) work together to coordinate locomotion, body movement, and interception of targets or avoidance of threats. Nevertheless, each sensory modality has certain limitations and optimal ecological functions (or natural roles to play) when it comes to picking up information about the environment (Guski, 1992). These functions and limitations should be understood to foster optimal display de- sign. For example, vision is excellent at detecting distant objects or events, often before the objects or events are close enough for auditory or somatosensory detection. Also, vision permits excellent information for the mainte- nance of spatial awareness concerning the environment; for example, one can become disoriented underwater or when trying to walk through a dark room. On the other hand, audition excels at providing advanced warning and continuous monitoring of potential threats or significant events going on in a cluttered environment or outside one’s limited field of vision. For example, audition may play a critical role in the decision about whether one should move rapidly out of the way of an approaching threat or one has enough time to turn to face the threat instead (Guski, 1992).

Gibson (1979) thoroughly considered the natural functions of vision for controlling self-motion through the world and avoiding or intercepting objects. What are the natural functions (and limitations) of somatosensation during self- motion or looming? Kinesthesia (muscle, joint, and tendon sensations) and cutaneous senses (especially touch and pressure) are important to active locomotion, since they (along with the vestibular senses) provide critical informa- tion for the control of posture within the limits of stability, as well as the appreciation of one’s heading, speed, and relation to the substrate (e.g., firm/soft, even/sloping). Even during driving, somatosensory (skin pressure and kines- thetic) appreciation of body forces during acceleration/deceleration augments one’s ability to make controlled stops.4 A few of the predominant natural functions of vision, audition, and touch are summarized in Table 1, with emphasis placed upon survival-related functions in regards to the control of movement of the self relative to the environment

3 Gibson (1979) defined visual kinesthesis as the pickup of visual information related to self-movement, e.g., the sweeping of the field of view over the ambient optical array during head motion or the flow of the ambient optical array during locomotion. 4 This assertion disagrees with Gibson’s (1979) conjecture that “muscle-joint kinesthesis…does not function during passive loco- motion in a vehicle” (p. 125), but the present author considers it merely a mechanically-aided type of gap closing behavior with similarities to the landing of birds (Lee, 1998). Cognitive Engineering and Neuroergonomics (2019) and static or moving objects.5 The author’s goal is to consider the three senses from a general perspective consistent with Gibson’s ecological psychology.

Table 1: A few of the predominant survival-relevant functions of vision, hearing, and touch.
Original PDF Table 1:

Many of the natural functions of a given sensory modality shown in Table 1 will suggest optimal (i.e., most usable, interpretable, intuitive, or ecologically-relevant) cues to use in a modality-specific display. For example, auditory displays tend to be less efficient than visual displays at conveying information from graphs, maps, photographs, videos, equations, or complex symbols. Conversely, an auditory and/or tactile display would be superior to vision for alerting a vehicle operator who is looking elsewhere or is falling asleep. Table 1 can also be used to infer some ecological functions for one sensory system that might be successfully emulated via a different sensory modality, in order to avoid exceeding the processing capacity of a given sense (Cholewiak and McGrath, 2006). Humans rely ex- tensively upon vision, so we tend to design visual displays to help us control vehicles and machines. Unfortunately, our visual processing has become overloaded by this design practice. Currently, the pilot of a manned or unmanned aerial vehicle receives nearly all of the information concerning the state of his/her aircraft’s altitude, attitude, air- speed, heading, vertical speed, range/bearing to a target, or fuel level via numerous symbolic visual displays. It is important to determine whether some of this information could be replaced or augmented by non-visual displays, and if so, whether it can be conveyed in a way that involves direct perception (Gibson, 1979), rather than abstract processing requiring significant memory and interpretation. In the natural world, animals control their locomotion via direct, intuitive, subconscious, and multisensory feedback from vision (e.g., optical flow), audition (e.g., changes in the direction and distance of auditory frames of reference), somatosensation (e.g., skin pressure, shear, vibration, slip, limb kinesthesia), and vestibular sensation. Similarly, in the natural world, one perceives imminent object con- tact via intuitive feedback from optical flow, changes in loudness, and finally, skin pressure, pain, and/or forces on one’s limbs. Unfortunately, these non-visual cues are not typically exploited to convey veridical information con- cerning self-motion and looming during vehicle operations. Vehicle control tasks such as flying would be less prone to error if the displays concerning true vehicle state could be made more concordant with natural multisensory per- ception-action couplings and mechanisms.

This paper focusses on one aspect of the display design gaps identified above, viz., the evidence concerning the fea- sibility of using tactile displays to convey self-motion and/or looming. A key rationale for focusing on the sense of touch is because it is an under-utilized sense in aviation displays. The three main priorities of a pilot during normal operations are (in order) to “aviate, navigate, and communicate.” A military pilot must also operate displays and controls to identify and engage foes. Aviation, navigation, and enemy engagement are almost entirely visual tasks in flight, while communication is mostly auditory-verbal. Tactile displays are not currently used to help with these crit- ical flight priorities. It is worthwhile to explore touch as a non-visual flight display, since touch does not have the zone of fore-aft confusion associated with auditory displays. In fact, tactile torso sensations are localized most accu- rately at the midline, i.e., navel and spine (Cholewiak, Brill, and Schwab, 2004). Furthermore, reactions to touch stimuli are typically very rapid and are reflexively referenced to one’s body with good accuracy (Cholewiak and McGrath, 2006; Cholewiak, Brill, and Schwab, 2004; Brill and Terrance, 2007). Finally, in the natural world, so- matosensory stimuli are heavily involved in controlling self-movement already (e.g., how one’s skin is slipping or shearing relative to surfaces; Wasling, Norrsell, Göthner, and Olausson, 2005), so it is logical to suppose that an ecologically-valid tactile display design should be feasible. The next section below provides a consideration of the evidence that tactile stimuli can convey cues consistent with self-motion perceptions, followed by evidence that tac- tile stimuli can convey looming. The evidence is limited but interesting enough to warrant discussion.

EVIDENCE THAT TACTILE STIMULI ARE IMPORTANT TO SELF-MO- TION PERCEPTION Humans receive many environmentally-generated somatosensory cues concerning self-motion: wind or water flow- ing around the body, ground and engine vibrations (Riecke, Fuereissen, and Rieser, 2009), forces on the body gener- ated by acceleration, the degree of body leaning and head righting required at different speeds of linear or angular motion, the angle of contact of the foot to the substrate (e.g., to avoid foot slippage), joint/muscle forces during loco- motion, fingertip contact with walls and corners during locomotion, and the rhythmic information available during changes in speed of locomotion (Guedry, Rupert, and Reschke, 1998; Gray, 2009; Williams and Weigelt, 2002; Savelsbergh and Whiting, 1996). When people are deprived of a normal sense of touch in their feet, they have diffi- culty balancing and walking (Ments, Lord, St. George, and Fitzpatrick, 2004). Humans are even able to use remote tactile information from canes to aid their locomotion (Patla and Davies, 2004). Given the functional relevance of touch to self-motion in these situations, it is no surprise that neurophysiological evidence suggests that certain mo-

Cognitive Engineering and Neuroergonomics (2019) tion processing areas of the brain that respond to visual field motion are activated by tactile motion information as well (Soto-Faraco, Kingstone, and Spence, 2003).

Tactile stimuli can also modify ongoing perceptions of orientation and self-motion (Lawson, Rupert, Guedry, Gris- sett, and Mead, 1997; Lackner and DiZio, 2005). For example, Lackner and Graybiel (1978) reported that so- matosensory stimuli systematically altered vestibular illusions of motion associated with off-vertical rotation and perceptions of self orientation during the weightless phases of parabolic flight. When subjects are stationary inside a slowly rotating optokinetic drum in complete darkness, they may still perceive illusory self-rotation when they touch the rotating surrounding cylinder and follow its motion with one hand such that their arm moves about the shoulder joint (Brandt, Büchele, and Arnold, 1977). This illusion is tactile and kinesthetic in origin. The illusion is not readily elicited at higher velocities of drum movement when the participants reached out and followed the drum’s interior by “walking” both hands along alternately. Nevertheless, it is known that actively stepping with the feet along with the motion of a platform can enhance the perception of self-motion (Bles, 1981; Bles and Kapteyn, 1977; Riecke, Feuereissen, Rieser, and McNamara, 2011). It has been conjectured (Lawson and Riecke, in press) that ecological considerations may contribute to this difference in findings, since rhythmic leg motion relative to a horizontal sup- port surface is closely associated with self-movement over the stationary ground.

Since the most ubiquitous and versatile touch displays currently available rely on arrays of small vibrators (called vi- brotactors) on the body, it is important to determine whether artificial vibrotactile (vibrating touch) displays can modify or elicit self-motion perceptions as well. This is not a given, since such tactor arrays are typically very small compared to the number of optical texture elements available from even an inexpensive and small computer moni- tor, which means that some user error will be a function of resolution of the display rather than whether the display was tactile or visual (van Erp and Verschoor, 2004). Moreover, the activation of successive pixels on a visual screen can occur with very little interstimulus delay, eliciting a sensation of smooth visual motion, whereas vibrotactors take more time to ramp up and down and the sensation of tactile flow is likely to be more staccato than real tactile flow experienced in the natural world (e.g., trailing one’s fingers along a wall while walking in the dark). This is likely due to the small number of vibrators in an array and the vibrators’ speed of activation, rather than an inherent limitation of the cutaneous senses. In fact, it is probable that visual and cutaneous senses both have a similar ability to convey the idea of continuous apparent motion when two stimuli are presented in rapid succession, and that the interstimulus interval at which this sensation occurs is similar for vision and touch (Sherrick and Rogers, 1996; Lakatos and Shepard, 1997). Overall, the literature suggests that vibrotactile displays can be helpful when used as alerts, to augment visual displays, and to provide waypoint information in conditions of low visibility or difficult ter- rain (Prewett, Elliot, Walvoord, and Coovert, 2012).

Although the resolution of current tactile displays is primitive compared to visual displays, there is evidence that vi- brotactile flow fields can be exploited to modify feelings of self-motion. Kolev and Rupert (2008) reported that vi- brotactile flow could modify a visually-induced illusion of self-motion. The authors employed a vibrotactile belt (an array of eight vertical columns of tactors around the torso, with five tactors in each column) that activated successive tactors (two columns at a time) in the opposite direction from the rotation of an optokinetic stimulus. By this means, the stimulus emulated natural tactile flow, which tended to weaken the subjects’ (n = 12) visual vection illusion of moving in a direction opposite to the optical flow produced by real motion of an immersive optokinetic sphere. Also, 55% of the subjects’ showed evidence of an alteration in their visually-mediated optokinetic gaze reflexes when the tactile flow was activated. Similarly, when subjects (n =7) were presented with simulated optical flow cues (radial expansion of ~1,000 random dots on a 20-inch monitor) for forward self-motion, their estimates of the speed of illusory forward self-motion could be reduced or increased by varying the speed of front-to-back tactile flow (i.e., inter-stimulus interval of tactor activation in a 4 x 5 array of tactors) across the seat of their pants (Amemiya, Hirota, and Ikei, 2013), which is a very natural way to convey self-motion tactually.

EVIDENCE THAT TACTILE STIMULI CAN CONVEY LOOMING IN WAYS ANALOGOUS TO VISION OR AUDITION Somatosensory cues concerning collision or interception can be considered the final source of information for modi- fying action and for updating sensorimotor calibrations during the many looming judgments required daily during growth or during adult learning of new motor skills (Savelsbergh and Whiting, 1996; Kayed and Van der Meer, 2009; Fajen, Riley, and Turvey, 2009). When vision is degraded, touch becomes important, even when walking

Cognitive Engineering and Neuroergonomics (2019) through a familiar environment. During an evening power failure in one’s home, one raises their hands protectively and feels forward with one’s hands and feet, in order to be warned of obstacles and passages.

Although natural somatosensory cues are important to intercepting targets or avoiding obstacles, it has not been fully determined whether an artificial tactile display can convey useful moment-by-moment (graded and dynamic) ap- proach information (e.g., relative distance updates) prior to actual contact, as has been reported for visual and audi- tory displays. A few studies have been performed that address vibrotactile distance-to-target cues (e.g., Jansson, 1983; van Erp, 2007; Singh et al., 2010). From such studies, it appears that subjects can utilize a spatiotemporally- varying vibration stimulus as a cue concerning the approach of a target. In 2013, more specific and complete evi- dence emerged stating vibrotactile flow fields may be useful for conveying looming (Cancar, Diaz, Barrientos, Travieso, and Jacobs, 2013). Cancar et al. asked 12 participants to estimate time-to-contact of a radially-expanding tactile or visual flow field representing a simulated sphere approaching. Tactile flow was produced by radial activa- tion of an array of vibrotactors, which “expanded” from 4 to 12 to 36 active tactors in an ovoid pattern across the ab- domen. The visual flow was a computer monitor representation of the tactile flow field. This experiment found good accuracy in predicting the time-to-contact and in fact, did not detect significant differences between the visual and tactile conditions. In a second study in the same paper, the authors reported that similar tactile flow cues concerning the approach of a real ball were sufficient to enable participants (n = 12) to hit the ball at the correct time in 71% of the trials.7 The expanding tactile flow field Cancar et al. employed has two possible advantages over a simple on/off vibratory “looming warning cue” (such as the vibrating mode of a cell phone). First, the expanding flow field is a logical analogue of an approaching optical target or three-dimensional auditory target. Second, the perceived magni- tude of the stimulus will increase as more tactors are activated (Cholewiak, 1979), thus increasing saliency (and pos- sible urgency).

Employing a tactile array in a way analogous to optical looming is interesting, but it is not essential for a tactile looming cue to be an analogue of visual looming. There will be display needs where a smaller, cheaper, easier-to- implement tactile cue will be desirable. Between a full-featured tactile array and a primitive on/off tactile warning cue, there lies the intermediate possibility for a tactile display inspired by the auditory cues for looming. Blind peo- ple can detect when an automobile is rushing towards them at an intersection because the sound of its engine and tires seems louder and higher in pitch as the vehicle nears. Such auditory looming cues activate motor planning areas of the brain differently from receding auditory stimuli (Seifritz et al., 2002). Just as the perceived time to arrival of a rapidly-approaching visual object has been characterized in ecological psychology by a simple mathematical de- scription -- the rate of image dilation given by tau (Lee, 1976), so also does the auditory rate of change of noise in- tensity play a role in judging time of arrival via acoustic tau (Shaw, McGowan, and Turvey, 1991).8 This has practi- cal implications for display design, e.g., Gray (2011) reported that an auditory vehicle collision warning that in- creases in sound intensity in a way analogous to a real sound source approaching aids faster initiation of braking than any other cue he tested, with the exception of the sound of a car horn, which nevertheless produced a greater likelihood of false positive braking responses. Since simple loudness and pitch changes can help to convey looming, the author of the present report wondered whether a localized tactile stimulus of varying vibration frequency could convey information consistent with looming even when the tactile stimulus cues did not vary spatially over the sur- face of the body. The stimuli and certain parts of the experiment are summarized briefly, below.9 Six vibration stimuli were evaluated by 35 participants.10 The stimuli varied in terms of their tactor vibration fre- quency and duration of firing over the course of a 3 s train of 20 vibratory bursts (or beats). In all conditions, inter - burst intervals were at least 20 msec and the total duration of a vibration train was approximately 3 s. Condition 1 was the control condition (1a = a constant vibration stimulus felt over the 20 pulses; 1b = random stimulus). Condi- tion 2 was the vibration frequency change condition (2a = decreasing frequency from 250 Hertz (Hz) to 30 Hs over time; 2b = increasing from 30 Hz to 250 Hz). Condition 3 was the changing beat speed condition where vibration frequency was constant but the speed at which the burst of vibration came was altered (3a = decreasing beat speed from 300 msec duration to 30 msec; 3b = increasing from 30 msec to 300 msec). A trial was made up of either Con- dition pair #1a-b, pair #2a-b, or pair #3a-b, with the order of the a-b pair members randomized over trials. For ex-

7 The arc of trajectory of the ball was known to the participants, but not its time of contacts. 8 Note that other visual and auditory factors are important also (Guski, 1992; Hancock and Manser, 1997). 9 This study was approved by the Institutional Review Board, Office of Research Protections, U.S. Army Medical Research and Materiel Command. 10 Forehead and abdomen body sites were evaluated and auditory control conditions were employed. This report summarizes tac- tile findings only without distinguishing sites. Cognitive Engineering and Neuroergonomics (2019) ample, Trial 1 might consist of 3b followed by 3a; Trial 2 might consist of 2a followed by 2b, and so on. In this manner, six possible pattern pairs were constructed for repeated measures: 1a1b, 1b1a, 2a2b, 2b2a, 3a3b, and 3b3a.

Our goal was to establish whether a varying vibration cue will be interpreted as a meaningful tactile index or icon consistent with “approaching” or “receding” (Brewster and Brown, 2004). Participants were asked to rate what the stimulus felt like using a semantic differential scale, where (1 = “like a real object moving away from me” and 7 = “like a real object moving towards me”). We observed a main effect for the vibration stimulus pattern [control, fre- quency, or beat speed conditions; Wilks’ Lambda 0.6, F (5, 30) = 101.1, p < 0.001], mainly because of the changing frequency conditions. Increasing frequency was rated highest (i.e., it was rated the best “approaching” tactile icon or tacton), while decreasing frequency rated lowest (i.e., the best “receding” tacton). This contrasting result for the same stimulus (frequency change) lends validity to the findings. These two frequency conditions (increasing and de- creasing frequency) were different from all other stimulus patterns. It is hypothesized that frequency cue worked best because the increase in vibration saliency and frequency it delivers over time is roughly analogous to a natural auditory cue. As the frequency of the vibration stimulus increased over the stimulus period, the effect may be analo- gous to the perceived increase in pitch of an approaching sound source. Moreover, the vibration stimulus should in- crease in detectability over the stimulus period as it approaches the more salient higher frequency (Gescheider, Frisina, Hamer, and Verillo, 1978), which may emulate the perceived increase in loudness of an approaching sound source. We conclude that altering vibration frequency may be one type of ecologically-valid tactile cue for looming.

HYPOTHESIZED OPTIMAL VISUAL AND TACTILE CUES FOR SELF- MOTION VERSUS LOOMING We have considered the natural functions of touch versus vision and audition, and have highlighted some evidence suggesting it may be possible to employ tactile displays to convey cues consistent with self-motion and looming. The reader will recall from the Introduction that a central concern for the ecological psychology of Gibson (1979) was the influence of whole-field optic flow on the control of self-motion and the control of action relative to loom- ing objects. So far in this paper, these two aspects of movement perception and control have not been contrasted in detail. The final questions to be considered in this paper are how perception of self-motion differs from perception of an object looming, and whether design principles for optimal visual cueing can be extrapolated from these differ- ences to optimize tactile cueing of self-motion versus looming. Several general ecological visual display design principles can be extrapolated from Gibson (1979) and later sources concerning the optimal control of self-motion without collision while moving through space using optical flow information, versus the related issue of optimal control of object contact or collision avoidance. These are listed below for the sake of comparison to the author’s subsequent list of hypothetical requirements a tactile display should meet to be able to cue flow/self-motion versus object collision/contact.

General Visual Display Principles for Perception/Control of Self-motion without Collision

1) Ambient, coherent cues are important (Gibson, 1979). Coherent movement of the entire visual field tends to support the self-motion percept.11 Non-coherent, semi-random motion of optical texture (e.g., dust or leaves) is not likely to be interpreted as coherent motion of the visual field (Shiffrar, 2005). 2) Radial expansion is consistent with approaching, while radial contraction is consistent with receding (Gibson, 1979). a. Related Implication: the environment flows around you when you are not colliding with anything. b. Related Implication: flow during self-motion can go all the way up, down, or around your body, whereas collision often just hits one part of your body or all parts at nearly the same time. c. Related Implication: The limits of one’s field of view create a body-centered reference to aid the interpreta- tion of global flow cues concerning self-motion. 3) Faster or slower global flow specifies faster or slower self-motion, respectively (Gibson, 1979).12

4) The point of origin of optic flow is taken as current direction of self-motion (Gibson, 1979). 5) Optic texture elements on the ground are of special significance as a frame of reference (Gibson, 1979).

11 Although central radial flow can also elicit a perception of self-motion (Andersen and Braunstein, 1985). 12 However, the accuracy of this cue for indicating one’s speed declines as height above the ground increases (Warren, 1988). Cognitive Engineering and Neuroergonomics (2019)

6) Certain geometric cues relative to the global array can be of special significance, such as linear perspective cues or the position and motion of the horizon line (e.g., the relation of the horizon to the self and the target object help determine whether an opening is wide enough to pass through; Goldstein, 2005). 7) During self-motion, fixed objects in the environment (e.g., trees) appear to pass by more quickly than distant objects, due to relative motion parallax (Gibson, 1979). 8) Nearer objects occlude further objects visually. During self-motion, the amount of this near-far visual occlusion of regular ground objects (e.g., rocks or small shrubs during locomotion, trees during aviation) is a cue for self height or altitude (Goldstein, 2005). 9) Control and coordination of self-motion involves corroborating visual feedback with changing visual inputs from somotasensory system if active locomotion or if passive transportation at other than constant velocity, whereas collision can also occur when one is stationary and not receiving such changing somatosensory inputs.

General Visual Display Principles for Perception/Control of Object Contact or Collision

1) Visual angle increases to 180 degrees as an object reaches the face (Gibson, 1979), but this cue alone is less ef- fective at distances greater than 50 ft.13

2) Visual angle increases more and more rapidly as an object approaches (Gibson, 1979). 3) As an object draws nearer, binocular disparity/stereopsis cues increase (within 20 ft according to Wood and Chaparro, 2011), but note that rate of change of disparity provides a more distant cue (Goldstein, 2005). 4) An object occludes more and more of ambient visual environment as it approaches (Gibson, 1979). 5) The object and the ambient environment are distinguishable, e.g., there may be central optical expansion indi- cating a looming object relative to a stationary frame of reference (Gibson, 1979). 6) Two optical flows may be happening simultaneously if the observer is moving, e.g., radial expansion of envi- ronment during forward motion concurrent with faster radial expansion of an object approaching the observer. 7) Objects that are colliding with the observer do not flow around him/her – such objects generally stop at the ob- server’s body. Also, when the object contacts one’s body, this is confirmed via the tactile sense. Objects and apertures are distinguishable. Gibson (1979) describes how an object or aperture both have an edge relative to the surround and that edge will expand in the same way, but he identifies other cues to distinguish them. For ex- ample, the aperture reveals more and more of the vista beyond it as one gets closer, whereas the object occludes more and more of the vista beyond it. Also, the overhead (solar or artificial) lighting is generally different in the two cases.

HYPOTHETICAL DISPLAY PRINCIPLES FOR UTILIZING TOUCH TO CONVEY SELF-MOTION WITHOUT COLLISION VERSUS LOOMING (OBJECT CONTACT/COLLISION) This section lists the author’s tentative recommendations concerning ecological display design principles for optimal utilization of tactile (especially vibrotactile) displays to convey self-motion versus object collision/contact. These display design principles are based on the literature (see Introduction) and on personal communications with Roger Cholewiak, Ph.D.14 (in 2005 and 2010) and Bruce Mortimer, Ph.D. (in 2010).15

Vibrotactile Cueing of Self-motion without Specifying an Imminent Collision

1) The body site(s) chosen for stimulation should be chosen for maximum ecological validity, e.g., a self-motion sensation would be most ecologically elicited on the bottom of feet, the legs, and/or the buttocks, because these are natural body sites for inferring self-motion relative to a stationary substrate. (This is analogous to ecological psychology’s visual principle that the ground is of special significance.) 2) An ecologically-relevant self-motion cue need not be concentric or radially-changing, e.g., it can entail flow in a straight line from front to back on one’s seat, radial flow, or combinations of these, as appropriate. (This differs partially from the visual display principles previously discussed concerning radial change.)The portion of the 13 As when observing the tail lights of the car ahead at night (Wood and Chaparro, 2011). 14 Head of Cholewiak Consulting; retired Director of the Cutaneous Communications Laboratory at Princeton University. 15 Director of Research and Development, Engineering Acoustics, Inc.. Cognitive Engineering and Neuroergonomics (2019) field indicating self-motion should move coherently. (This is analogous to the important role of coherent flow in visual displays.) 3) Indicating self-motion does not require a stationary frame of reference within the tactile field itself, because one’s own body is the tactile frame-of-reference versus the self-motion cue. (This is analogous to the role of the field-of-view in vision.) 4) If one is upright, the tactile flow should go around to the side of one’s body, i.e., continue past one. (This is analogous to the visual display principle concerning optical flow passing around the observer.) a. Related Implication: In tactile flow, one could use cross-sections of body relative to environment. For the simple case where one is sitting in a cockpit flying straight-and-level with one’s hands and feet on the flight controls, the forward self-motion cue could be presented first at one’s feet, then flow up one’s legs and arms before passing over one’s torso from front to back. b. Related Implication: There is no “stopping point” where all tactors are on, i.e., no collision cue – rather the self-motion cue should be continual and homogenous. 5) The flow would be emulated by turning off previous tactors as new ones come on. (This is analogous to the lim- ited size of a visual display which necessitates entry of new optical texture elements from the center or edge of the display as other elements leave the outside edges of the screen.) 6) When object collision must be conveyed at one body site (i.e., without employing a spatial array of multiple tac- tors), changing the frequency of tactor vibration from low to high is more intuitive than changing the beat fre- quency of tactor pulses from low to high.

Vibrotactile Cueing of Self-motion for Specifying an Imminent Collision

1) The body site(s) chosen for stimulation should be chosen for maximum ecological validity, e.g., a head-on colli- sion perception would be most ecologically conveyed on the front of the torso or the face. 2) The origin of flow should be in center of the object and coming towards the observer’s face (This is analogous to the origin of visual flow). 3) Concentric expansion of the central tactile field should be suitable for conveying a colliding object (This is analogous to radial visual flow cues). 4) A sub-part of field should be moving, i.e., tactile flow should occur in one part of the display in relation to an ambient frame of reference designating the environment around the looming object. For example, a central tac- tor flow field could expand relative to an outside tactor ring that is always “stationary.” (This is analogous to the visual principles concerning expansion of the object relative to an ambient frame of reference and occlusion of the frame of reference by the object as it nears.) 5) The flow should stops at the front of the user’s body – it does not pass him/her. (This is analogous to the visual principle that the colliding object does not flow around you.) One implication is that the approaching object has a collision point (e.g., all tactors pulse or signal intensely at “collision” – this cue should be non-continual and heterogeneous versus prior cues leading up to collision). In the simple case discussed earlier where the flow cue travels up ones arms and legs towards one’s torso, it would stop at the front of the torso (and possibly the face).

CONCLUSIONS

If James Gibson were alive today, he would probably look at the available evidence and conclude that it is worth- while to explore the feasibility of using artificial touch displays to augment the perception and control of self-motion and looming, especially when visual information is degraded. He would be pleased to learn that the applications of tactile displays that seem to hold the most promise tend to be analogues of natural self-motion and looming cues. For example, the author’s research implies that a simple localized touch message consistent with looming can be identified that is roughly analogous to auditory cues for object approach. Similarly, a recent study implies that multi- point spatial arrays of tactile stimulators can augment the perception of looming in a way analogous to optical flow. Finally, recent research suggests that forward-to-backward tactile flow cues that are analogous to natural touch cues (e.g., sliding down a hill) and natural visual self-motion cues (optical flow) tend to enhance the perception of for- ward motion. These successful mappings, from vision and audition to touch, bode well for future tactile displays. Nevertheless, there are limitations to making straightforward analogies from visual to nonvisual looming perception (Jansson, 1983; Guski, 1992). Moreover, display designers should be aware that the optimal cues for self-motion are not the same as those for looming and significant design challenges lie ahead for creating a touch display that can unambiguously cue both percepts. Consideration by display designers of these challenges and the solutions for them Cognitive Engineering and Neuroergonomics (2019)

(some of which have been suggested in this paper) should foster the dissemination of future touch displays and re- duction of human error.

While this paper restricted discussion to vibrotactile displays, such displays are not necessarily the most ecologically valid touch cue in every situation. In addition to the sense of vibration, cutaneous sensations include light touch, deep pressure, stroking, temperature, and pain. The kinesthetic (joint, tendon, and muscle) sense is important also and is integrated with the touch modality during natural perception and action. Current vibrotactile displays will eventually become part of full-featured somatosensory (cutaneous and kinesthetic) display systems, and as this oc- curs, each sensory quality conveyed by the somatosensory display suite should be consistent the natural capabilities of the corresponding sensory modality so that a multisensory display is developed that is easy to learn and intuitive to use without error.

REFERENCES

Amemiya, T., Hirota, K., Ikei, Y. (2013). Perceived forward velocity increases with tactile flow on seat pan. IEEE Virtual Reality, 141-142. Andersen, G. J., Braunstein, M. L. (1985). Shape and depth perception from parallel projections of three-dimensional motion.

Journal of Experimental Psychology: Human Perception and Performance, 10, 749–760. Bles, W., Kapteyn, T. S. (1977). Circular Vection and Human Posture: 1. Does Proprioceptive System Play a Role? Agressologie, 18(6), 325–328. Bles, W. (1981). Stepping around: circular vection and Coriolis effects. In J. Long and A. Baddeley (Eds.), Attention and Performance IX (pp. 47–61). Hillsdale, NJ: Erlbaum. Boff, K. R., Lincoln, J. E. (1988). Engineering Data Compendium. Human Perception and Performance. AAMRL, Wright-Pat-

terson AFB, OH. Brandt, T., Büchele, W., Arnold, F. (1977). Arthrokinetic nystagmus and ego-motion sensation. Experimental Brain Research, 30(2), 331–338. doi:10.1007/BF00237260. Brewster, S., Brown, L. M. (2004). Tactons: structured tactile messages for non-visual information display. In A. Cockburn (Ed.),

Proceedings of the fifth conference on Australasian user interface: Vol. 28. Australian Computer Society, Inc. (pp. 15-23). Darlinghurst, Australia. Brill, C., Terrence, P. (2007). Discrete tactile versus auditory target cueing (Technical Report No. ARL-SR-0152). In R. Gilson, E. Redden, and L. Elliott (Eds.), Remote tactile displays for the Contemporary Soldier. Army Research Laboratory, Human Research and Engineering Directorate. Aberdeen Proving Ground, MD. Cancar, L., Diaz A., Barrientos, A., Travieso, D., Jacobs, D. (2013). Tactile-sight: a sensory substitution device based on dis-

tance-related vibrotactile flow. International Journal of Advanced Robotic Systems, 14, 21. Cholewiak, R. W. (1979). Spatial factors in the perceived intensity of vibrotactile patterns. Sensory Processes, 3, 141-156. Cholewiak, R. W., Brill, J. C., Schwab, A. (2004). Vibrotactile localization on the abdomen: Effects of place and space. Perception and Psychophysics. 66(6), 970-987. Cholewiak, R. W., McGrath, C. (2006). Vibrotactile targeting in multimodal systems: Accuracy and interaction. In Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE 14th Symposium, 413-420. Fajen, B. R., Riley, M. A., Turvey, M. T. (2009). Information, affordances, and the control of action in sport. Journal of Sport

Psychology, 40(1), 79. Flach, J. M., Hancock, P., Caird, J., Vicente, K. J. (1995). Global perspectives on the ecology of human-machine systems. Hillsdale, NJ: Erlbaum. Gescheider, G. A., Capraro, A. J., Frisina, R. D., Hamer, R. D., Verrillo, R. T. (1978). The effects of a surround on vibrotactile

thresholds. Sensory Processes, 2(2), 99-115. Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Boston: Houghton-Mifflin. Goldstein, E. B. (Ed.). (2005). The Blackwell Handbook of Sensation and Perception. Hoboken, NJ: John Wiley and Sons. Gray, R. (2009). How do batters use visual, auditory, and tactile information about the success of a baseball swing? Alliance for health, physical education, recreation and dance, 80(3), 491-501. Gray, R. (2011). Looming auditory collision warnings for driving. Human Factors: The Journal of the Human Factors and Er-

gonomics Society, 53(1), 63-74. Guedry, F. E., Rupert, A. R., Reschke, M. F. (1998). Motion sickness and development of synergy within the spatial orientation system. A hypothetical unifying concept. Brain research bulletin, 47(5), 475-480. Guski, R. (1992). Acoustic tau: An easy analogue to visual tau? Ecological Psychology, 4(3), 189-197. Hancock, P. A., Manser, M. P. (1997). Time-to-contact: More than tau alone. Ecological Psychology, 9(4), 265-297. Jansson, G. (1983). Tactile guidance of movement. International Journal of Neuroscience, 19(1-4), 37-46.

Cognitive Engineering and Neuroergonomics (2019)

Kayed, N. S., Van der Meer, A. L. (2009). A longitudinal study of prospective control in catching by full-term and preterm infants. Experimental brain research, 194(2), 245-258. Kolev, O., Rupert, A. (2008). The use of Tactile cues to modify the perception of self-motion. Proceedings of the 26th Army Sci-

ence Conference, Chicago. Lackner, J. R., DiZio, P. (2005). Vestibular, proprioceptive, and haptic contributions to spatial orientation. Annual Review of Psychology., 56, 115-147. Lackner, J. R., Graybiel, A. (1978). Some influences of touch and pressure cues on human spatial orientation. Aviation Space and

Environmental Medicine, 49(60), 798-804. Lakatos, S., Shepard, R. N. (1997). Constraints common to apparent motion in visual, tactile, and auditory space. Journal of Experimental Psychology: Human Perception and Performance, 23(4), 1050. Lawson, B. D., Riecke, B. E. (In press). The perception of body motion. In K.M. Stanney and K. Hale (Eds.), Handbook of vir-

tual environments: Design, implementation, and applications (2nd Edition). New York, NY: CRC Press. Lawson, B. D., Rupert, A. H., Guedry, F. E., Grissett, J. D., Mead, A. M. (1997). The human–machine interface challenge of using virtual environment (VE) displays aboard centrifuge devices. In: M. J. Smith, G. Salvendy, and R. J. Koubek (Eds.), De- sign of computing systems: Social and ergonomic considerations (pp. 945–948). Amsterdam: Elsevier Science. Lee, D. N. (1976). A theory of visual control of braking based on information about time-to-collision. Perception, 5(4), 437-459. Lee, D. N. (1998). Guiding movement by coupling taus. Ecological psychology, 10(3-4), 221-250. Nakayama, K. (1994). James J. Gibson: An appreciation. Psychological Review, 101(2), 329-335. Prewett, M. S., Elliott, L. R., Walvoord, A. G., Coovert, M. D. (2012). A meta-analysis of vibrotactile and visual information dis-

plays for improving task performance. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transac- tions, 42(1), 123-132. Riecke, B. E., Feuereissen, D., Rieser, J. J. (2009). Auditory self-motion simulation is facilitated by haptic and vibrational cues suggesting the possibility of actual motion. ACM Transactions on Applied Perception (TAP), 6(20),1–20:22. Riecke, B. E., Feuereissen, D., Rieser, J. J., McNamara, T. P. (2011). Spatialized Sound Enhances Biomechanically-Induced

Self-Motion Illusion (Vection). Proceedings of Conference on Human Factors in Computing Systems (pp. 1–4). Presented at the “Chi’11,” Vancouver, Canada: ACM Press. Savelsbergh, G. J. P., Whiting, H. T. A. (1996). Catching: A motor learning and developmental perspective. Handbook of perception and action, 2, 461-501. Seifritz, E., Neuhoff, J. G., Bilecen, D., Scheffler, K., Mustovic, H., Schächinger, H., … Di Salle, F. (2002). Neural processing of

auditory looming in the human brain. Current Biology, 12(24), 2147-2151. Shaw, B., McGowan, R., Turvey, M. (1991). An acoustic variable specifying time-to-contact. Ecological Psychology, 3(3), 253-

261. Sherrick, C.E., Rogers, R. (1996). Apparent haptic movement. Perception and Psychophysics, 1, 175-180. Shiffrar, M. (2005). Chapter Eight Movement and Event Perception. In E. B. Goldstein (Ed.). The Blackwell Handbook of Sensation and Perception. Hoboken, NJ: John Wiley and Sons. Singh, V., Paul, R., Mehra, D., Gupta, A., Sharma, V. D., Jain, S., … Manocha, D. (2010). 'Smart' Cane for the Visually Im-

paired: Design and Controlled Field Testing of an Affordable Obstacle Detection System. In TRANSED 2010: 12th Interna- tional Conference on Mobility and Transport for Elderly and Disabled Persons. Soto-Faraco, S., Kingstone, A., Spence, C. (2003). Multisensory contributions to the perception of motion. Neuropsychologia, 41(13), 1847-1862. Van Erp, J. B. F. (2007). Tactile displays for navigation and orientation: perception and behavior (Doctoral dissertation).

Utrecht University, Netherlands. Van Erp, J. B., Verschoor, M. H. (2004). Cross-modal visual and vibrotactile tracking. Applied Ergonomics, 35(2), 105-112. Warren, R. (1988). Visual perception in high-speed low-altitude flight (Report No. AAMRL-TR-88-066). Armstrong Aerospace Medical Research Laboratory, Wright-Patterson AFB, OH. Wasling, H. B., Norrsell, U., Göthner, K., Olausson, H. (2005). Tactile directional sensitivity and postural control. Experimental

Brain Research, 166(2), 147-156. doi: 10.1007/s00221-005-2343-5. Wickens, C. D., Carswell, C. M. (2006). Information processing. Handbook of human factors and ergonomics, 3, 111-149. Williams, A. M., Weigelt, C. (2002). Vision and proprioception in interceptive actions. Interceptive Actions in Sport: Information and Movement. Albany, NY: State University of New York Press. Wood, J., Chaparro, A. (2011). Chapter 28 Night Driving: How Low Illumination Affects Driving and the Challenges of Simula-

tion. In D. L. Fisher, M. Rizzo, J. K. Caird, and J. D. Lee (Eds.), Handbook of Driving Simulation for Engineering, Medicine, and Psychology. Boca Raton, FL: CRC Press.

Cognitive Engineering and Neuroergonomics (2019)

Cover of the proceedings volume Advances in Cognitive Engineering and Neuroergonomics

Explore the complete digital volume

Advances in Cognitive Engineering and Neuroergonomics

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2101-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →