Interacting in the perceptual process by artificial means (augmented perception) involves a) capturing and conveying environmental cues, and b) designing rele-vant interfaces toward the human receiver. Expanding communication beyond vision and hearing in the visocentric paradigm is valuable for a user living with deafblindness but in the long run also for the many opening up for new informa-tional pathways. Augmented perception, regarded as a process, is highly complex spanning incommensurable domains, mental and material, natural and man-made, active agency and passive. We deconstruct augmented perception with special emphasis on getting a common model in spite of these disparate domains. We show that translation is a proper metaphor for the sequence that is necessary for achieving a). We use this deconstruction for designing an interface for haptic communication to the receiving human including b). We hope this work will be supportive for future development of new means of communication.
Assistive communicative devices for capturing and conveying environmental information to a user living with deafblindness - and in the long run exploring new means of communicating beyond sight and hearing for all - involve a) capturing and conveying environmental cues, and b) designing relevant interfaces toward the human receiver. In effect, this is about perception – here taken in a broad sense, not just limited to neural processes – being the way a cognitive agent connects to the physical world in the role of being exactly cognitive. It involves neural activities in the central nervous system (CNS), afferent (inward going) but also efferent (outward going, for motor control) nerve processes, interactions to active search for clues in the surroundings, as well as physical phenomena (light reflection, acoustical wave propagation etc.) in the world outside the agent. Since long, classical Fechner (Fechner, 1860)] and models based on a canonical linear <distal stimulus - proximal stimulus – (waiting) receptors – nerves – CNS – brain> sequence are replaced by non- sequential network models of interacting, active, cue-searching, collaborative, distributed subsystems. By this, seemingly stable concepts like that of what a sense is, is questioned and human intentions as an ignition spark for perception is problematized. This said, with caution, it is still possible to consider parts of perception, and as an approximation treat them as sequential. For subparts only, it is possible to make efforts for replacing those parts by artificial means for congenital deficits or acquired losses.
The result is Augmented Perception (AP). It could vary vastly - be a neural electrode implant for jacketing into the human peripheral or central neural systems or be a camera and accompanying AI guided image analysis for people with vision impairment. In any case, AP systems are embedded in the natural perceptional network being a smaller or larger part thereof. AP can be regarded as an artefact i.e. a thing, a process or, as there is intention behind, an action or an event etc. We will mainly look upon AP as a process. A process is something taking place in temporal space having a start and an end involving change from some start state to some end state. AP – as well as perception – is difficult to deal with due to many reasons. It spans and unites real-world material entities (rocks, sunbeams and chairs) and mental entities (percepts, concepts, emotions, associations) which are seemingly incommensurable. It might involve the analogue as well as the digital. So also the plurality of the social and the singularity of the individual psychological. It is not to be passive only, as perception is active – searching for cues outside of the CNS. AP also involves the notoriously difficult concept of “information”. At least in the
“end”(?) a human being should make information out of the stimulus. Defining information as (at least) that which adds to knowledge (Marcos, 2011) of a human cognitive agent it involves interpretation, semantics – and the mental entity knowledge. If, such as for an electronic pulse in a sea cable at the bottom of the Atlantic, these latter components are not present how could there be said to be information and if this is the case, how could information be said to be conveyed? There is then a problem of informational pathways from a position of the surrounding in space time (x0, y0, z0 , t0) to another one at the front of or in CNS (x1, y1, z1 , t1), t0 < t1 and what keeps them cohesive as the information is not for sure existing before or elsewhere than (x1, y1, z1 , t1). In addition, from a practical point there is then the problem of bridging separated spatial and temporal distances.
For addressing this, deconstruction is a heuristic that both adopts to the AP characteristic of being a part of a larger system (perception) which is useful for designing any subprocesses thereof, and also agree with a wish for understanding the nature, narrative, context, prerequisites and potential biases of AP. We denote these two aspects of deconstruction, the artefactual deconstruction and cultural deconstruction, respectively. Cultural deconstruction reveals the social aspects of how the AP came into existence and how it is maintained as an AP. Therefore, the study object of the present work is the very process of deconstruction, rather than that and that specific realisation of AP for a given need. Still, we will give a concrete example of an AP as an illustration to the deconstruction approach. The aim is not here so much to solve the complicated problems raised above but present a framework for addressing these questions. We present a suggestion for a theoretical framework for deconstructing augmented perception processes. We then apply this on a case involving both of the a) (capturing) and b) (interfaces) in the first sentence, simultaneously developing general design rules. Using the cultural perspective, we make conclusions about the process of deconstruction of AP.
THEORETICAL FRAMEWORK Perception is the process of relating an objective (assuming realism), external world, W, with an internal, human, mental realm, M, and forming percepts, giving knowledge. Any digital world is included in W insofar there is capturing using visual screens, sound etc. perception is of unrivalled importance for survival of the human being and one of the characteristics a being a human being whatsoever. Perception is highly complex but physiological (as revealed by physico-chemical laws and symmetries) natural process with feedback, active search for input, lagging and hysteresis, non-linearity, emergence, cooperation between units (such as receptor cells and afferent nerves) influenced by mental state. No detailed discussion hereof will be performed. Even if it is a network, it is assumed that some parts can be decomposed for both analysis and for replacement. Perception is a coherent system but it is no monolith. It is decomposable.
Augmented perception (AP) is here any man-made efforts for interfering in perception outside of M, may it be for support, enhancement or expansion of natural perception. AP can most often be regarded as an artefact, i.e. a man-made physical thing or a system of things. AP can also be regarded as a process, which during operation supports a human being in getting information about the outside world of the human being. Augmented perception might be a new term but is something that already is existing and exercised. It embraces all efforts done by man to facilitate capturing the world; painting with distinct colours, having traffic lights, street lighting, alarm signals, glasses, magnifying glasses, hearing aids etc. and of course also language which is connecting concepts and visual symbol (written language) and connecting concepts and audial symbols(which is spoken language). Then the following Ansatz for a theory of deconstructing augmented perception can be formulated.
We first state that, being a system it is possible to deconstruct the AP in parts. For example in an AP, a part could be a microphone capturing acoustic clues from the surrounding. It is artificial and is giving an important contribution but is not in itself sufficient for perception. It is a part of perception, thus an AP. This is a kind of an existence theorem for our study object – the process of AP deconstruction. As this is a human endeavour one could further state that there is an agent, Obs, performing the composition. This is a human being (designer, inventor philosopher, engineer, manufacturer, bureaucrat, relative) or in the future humanoid (AI for planning) that have some kind of intention for interfering in the perception either for artificial decomposition or cultural decomposition.
There is without any further requirements, rules or measure no unique decomposition. It is instead based on the abilities, interest, skills, biases, prejudice and resources of Obs as Obs is a human being (or human influenced AI etc.) and no neutral objective stand-point. Decomposition is about partition, formally: grouping of the parts such that no part is left and all are included in exactly one of these groups. Partitions could be spanning and comprising more or less of perception. An example is including or not the powering, the fastening, the electrical connectors to a part with the microphone or having these as parts on its own. Mathematically, the number of possible partitions is given by the Bell number, which for a set of n is a quickly growing function B = B(n) having the first values 1, 2, 5, 15, 52, 203,…(i.e. for a set of 3 elements it is possible to decompose the set in 5 ways.). This we call horizontal decomposition in our context. The Obs has an enormous amount of choices. Below we will add criteria for restricting this, which is important from a design perspective. Parts need to be possible to handle (understand, design, realize, make, analyse) both from an artefactual perspective and a cultural perspective. We now make the following observation. Obs can continue the decomposition of those parts in the partition that are complex. This creates decompositions within decompositions. Here we call this in-depth decomposition. This could be done in an iterative way creating successive fine-graining. An example of the in-depth decomposition is to divide the microphone into a noise reducing subpart (including a simple windshield or advanced Active Noise Cancelling), membrane subsystem, signal amplifier and signal
Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 converter subsystem and so on.
We postulate the existence of a signal in the AP system. A signal is a temporal and/or spatial (repeatable) pattern, propagating in some channel, so that it is casually (i.e. law-like) formed by an encoder physically coupled to the channel. Optionally there is a decoder physically coupled to the channel able to detect the signal, fig 1.
encoder decoder
Figure 1. Encoder, channel, decoder, signal. I is a tentative and very general definition. Signals embraced are electromagnetic wave propagating, as well as sentences written in natural languages. Signal is an ontological existing entity without any references to the existence of human agents. It goes equally well for abiotic (such as pulses in an electrical cable) and biotic (pulse in a nerve) situations. Causality means that there is not stochastic unpredictable creativity. It is predictable. Reflected light waves from a surface with a certain index of refraction as well as a sequence of words in one language syntax are examples.
Figure available in the original PDF paper.
Using the nomenclature channel, encoder and decoder appeals to Shannon-Weaver theory (Shannon and Weaver, 1949) and are the same kind of general quantities. Encoders create a signal. Detectability with a decoder means that the decoder can handle typical levels (amplitudes, frequencies etc.) of the signal and its base line and in turn getting out some law-like outsignal from it. Decoders involve photocells, microphones, human hearing etc. i.e. both abiotic and biotic. In order to talk about channel in singular the channel should not be heterogeneous. This is important as this is a requirement for narrowing the openness of both horizontal and in-depth decomposition. If the channel is homogenous, there is one type of signal and one type of encoder and one type of decoder. Then the whole complex encoder-signal-channel- decoder is belonging to one ontological or technical domain, may it be electronics or mental activities within a human being. An AP is complex. There are many channels and many encoder-signal-channel-decoder complexes. However, each of them is according to what just was said to belong to its own ontological domain. Mutually such domains might be highly different and more or less incommensurable.
We state the existence of translation in an AP. In normal language translation means working with “texts” (or linguistic sound) that belongs to one domain so that someone in another (language) domain should understand something. This understanding is the overall goal of translation. Translation is performed by someone (or possible something, like an automatic translation software). This agent we call translator T. In our case, the “text” is the signal belonging to some domain. For bridging different more or less incommensurable domains, translation is a valuable metaphor.
It is important to differentiate between translation and communication, as for example defined in Shannon-Weaver (SW) model and theory {3]. Both relates to a signal. SW- Communication has as goal as sending a signal given to the system as undistorted as possible from one site to another. Translation on the other hand is to produce a completely new signal, not whatever signal but dependent on a signal given to the translator and, which is important, with someone else's, A, understanding in mind. Thus translation is not [Equation or mathematical symbol available in the original paper PDF]out = T([Equation or mathematical symbol available in the original paper PDF]in ) but [Equation or mathematical symbol available in the original paper PDF]out = T([Equation or mathematical symbol available in the original paper PDF]in |A is to understand). Translation is conditioned. One could say that translation is not “local” but has the reader (who could be far away spatially and temporally) in mind. T in our AP is not to be limited to be a human being.
HIPI AND DECONSTRUCTION ALGORITHM
Ultrasonic sensors
Matrix of vibrators
Camera
Rasp berry Pi Power bank
Ultrasonic sensors
Based on the theoretical framework we design a system described in detail elsewhere (Darányi et al 2022) denoted Haptic Intelligent Personal Interface, HIPI, fig 2. It is a portable, stand-alone textile construction for AP involving both of a) capturing and conveying environmental cues, and with camera and other sensors wirelessly connected to servers in the cloud and b) interfaces toward the human receiver in the form of a matrix of vibration elements on the back, shoulder and waist. Figure 2. A developed AP systems fulfilling both a) for capturing environmental cues by sensors and b) having tactile interfaces toward the human receiver as well as powering and control. Based on the theoretical framework a deconstruction algorithm was created that guided the design. 1 Identify the parts of the perception that needs to be made artificial, made to be an AP. This could be due to absence of a sense (such as for people with deafblindness),
Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 congenital or acquired impairments, time-limited malfunctioning senses or a wish for enhancement of existing perception capabilities. 2 identify what types of ontological real world entities in W that are to be involved in the AP. For example, it could be “physical things on ground or floor” for an AP for navigation and obstacle avoidance. Or it could be “human faces” for AP for face recognition for social interactions. 3 Jointly, 1 and 2 gives for-ness of AP. For-ness (Kroes 2012) is the aim of the AP, the overall answer to what it is able to do. 4 identify where the homogenous channels are. Find the minimum number of these. 5 Identify the translations there are in the AP. Translations are coupling different homogenous channels from each other and acts as encodes and decoders. 6 Identify the spatial and temporal localisation of these translations. For example, a camera should be placed so its field of view is compatible with 1. 7 This gives needed communication in-between. The camera outsignal will be sent to a data fusion unit. This algorithm works equally well for the classical and “normative” perspective as for the “descriptive”. In Fig 3 an example is given applying this.
Figure 3. A case (for the AP called HIPI we have developed) of translations and channel Shannon-Weaver like transfer of the informational pathway. In this peculiar case, cognitive capacities were replaced by image analysis and decision algorithm CULTURAL DECOMPOSITION
Figure available in the original PDF paper.
AP is man-made. AP is constructed for certain aims. It is used for certain aims and maintained as an AP during its operational life for certain aims. Its existence is thus not neutral. Nobody would say that lamps in houses are stigmatizing or assistive devices, rather signs of welfare and useful technological development to the benefit of the many. However, in fact lamps are also AP. They help us perceive the external world during night. They are part of perception. They support us getting informed about the surroundings. Nowadays lighting is to be regarded as an important and integrated and obvious part of society as witnessed by such diverse phenomena as NASA earth-at- night photos ability to drive cars at 100km/h in the middle of the night, going to evening theatre or clubbing at nights. Visual perception has been given a lot of AP attention and effort. Unfortunately, there are other kinds of perception and other kinds of needs that have not rendered the same interest. For people living with deafblindness very few APs have been developed and those that have, are to be regarded as a peculiarity. It is at this stage appropriate to relate cultural decomposition of AP to the two perspectives of disability denoted the Medical Model and the Social Model. The former relate disability to the individual who is in focus and is to be handled by medical interactions. This is of course good and well but is also related to exclusion, stigmatization, that someone need to be cured and that disabled need to be pitied and be exposed to charity. The Social Model of Disability on the other hand reframes disability as a social construct and highlights the equality issues in the context of civil rights. Exclusion is due to the way society is run and organized.
The lighting example makes it clear that society is biased. There is no counterpart to street lighting for navigation and getting impressions for those that have to rely on haptic experiences only, built-in in the society and few if any tactile amusements (except as a kind of bi-product of doing other things as handicraft in clay, knitting or gardening). Cultural decomposition of AP reveals all this.
CONCLUSIONS
Augmented perception (AP) which is the introduction of some man-made means in, or as a replacement for, the natural perception is regarded as a process. Deconstruction reveals both artefactual deconstruction and cultural deconstruction. It is the hope that this theoretical framework will support further work along these lines. For example, we need to explore solutions that enable the user with impaired perceptual modalities to be the decider of what information is received
ACKNOWLEDGMENTS
This work has been part of EU Horizon 2020 research and innovation program under grant agreement No 780814 SUITCEYES.
REFERENCES
Darányi S. et al (2022) Expanding Semantic Space for Navigation and Communication: Users With Dual Sensory Impairment Benefit From The Suitceyes Eu Project Informational Re-search forthcoming Fechner, G. T. (1860). ”Elemente der Psychophysik”, vol. 1. Leipzig: Breitkopf
und Härtel (English translation by H. E. Adler (Howes, D. H. & Boring E. G. (Eds.)) “Elements of psychophysics”. New York: Holt, Rinehart, and Winston (1966) Kroes, P. (2012) “Technical artefacts: Creations of mind and matter: A philosophy of engineering design (Vol. 6)”. Springer Science & Business Media. Marcos, A. (2011) Bioinformation as a triadic relation. Information and living
systems, pp 55-90. Shannon C.E., Weaver W. (1949) “The Mathematical Theory of Communication”.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8989-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional