Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Bimodal Affective Computing Interfaces for Emerging Artificial Intelligence Paradigms

Brown University, Providence, RI 02912, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1005363

Abstract

For nearly 30 years, Affective Computing has described a paradigm for human-computer interaction where machines are designed to respond to human affect. With recent acceleration in the development of artificial intelligence, the possibility of conscious machines with their own experiences and emotions is emerging. This is an opportune moment for engineers and designers to speculate and consider how affective computing paradigms might be expanded to also include machine affect and an emotional capacity of AI. We explore these themes through the development of a prototype that introduces human users to machine affect.

INTRODUCTION

Human Affect and Machine Affect Affective computing is characterized by systems and machines that can recog- nize, interpret, and respond to human emotions (Picard, 2000). But with new breakthroughs in AI, and the speculated emergence of Artificial Conscious- ness (i.e. AGI), new paradigms for affective computing must be developed that take into consideration both the affect of the human user (human affect) as well as the affect of the non-human machine (machine affect). We pro- pose the term Bimodal Affective Computing (BAC) for this new paradigm. Bimodal Affective Computing can be characterized as the deployment of Affective Computing modalities where both the machine and the human user can intuitively recognize, interpret, and respond to the feelings and intentions of each other.

Traditionally, technology has been understood instrumentally, as objects or tools that can be applied towards a given task. This has been the dominant paradigm since our ancestors first made stone tools. However, with the antic- ipated arrival of AGI, new relationships may emerge between machines and humans that could fundamentally challenge this conception. Somewhere on the horizon, between science fiction and science fact, are machines that can feel and have their own experiences. In advance of these possibilities, design- ers and engineers can begin to consider how interfaces for this new paradigm

© 2024. Published by AHFE Open Access. All rights reserved. 155

might be deployed. In intersubjective interactions, mutual empathy is neces- sary for the development of healthy relationships. We propose that the same may be true for the emerging relationships between humans and machines.

Affective Computing for Artificial General Intelligence Are machines capable of feeling? Can machines experience suffering or joy? And if so, are they legally and ethically persons, or is it ok to manipulate and exploit these machines however humans may see fit? These are questions that have been extensively explored in Science Fiction.

For example, in the Blade Runner franchise, and the book it is based on, ‘Do Androids Dream of Electric Sheep?’ (1968) the Voight-Kampff machine is applied as a kind of polygraph that can reveal whether a subject is a real or artificial human (Dick, 1968; Scott, 1982; Villeneuve, 2017). Much like our prototype, this device measures autonomic response (e.g. respiration, eye movement, and heart rate) in order to determine the degree of empathy the subject is capable of, and ultimately whether they are human or machine.

The question of whether an AI is actually thinking or feeling, rather than just simulating or imitating the actual processes of thinking and feeling has long been debated, famously by John Searle in his so-called Chinese Room thought experiment (Searle, 1980). In that thought experiment, he argues that the manipulation of symbols to simulate and represent thought is not the same as conscious thought, even if it appears as such to an outside observer. These questions are relevant to the discussion of our project as well. Is machine affect equivalent to authentic, artificial emotion? How would you know the difference?

In any case, the possibility, even probability, of affective empathy between machines and humans is a scenario that engineers and designers should be considering and anticipating. Authentic emotional response would seem to require, at a minimum, subjective experience, self-awareness, and conscious- ness, which are not yet technically possible (Bubeck et al., 2023). However, we can imagine machine affect as existing on a spectrum, in the same way that we might categorize animal sentience on a spectrum relative to its bio- logical complexity. A fish may have less feeling than a dog, for example. The hope is that this prototype may help shape questions about unconscious artificial affect, and push towards building a better understanding about the emergence of conscious artificial affect. This project, and this prototype, are meant as a first step across that wider spectrum.

INFLATABLE RESPONSIVE INTERFACES To explore and expand upon these evolving conceptions of empathy, we have developed the first generation of what we are calling Inflatable Responsive Interfaces (IRI). Our prototype integrates IRIs into a mouse form factor to validate the design (see Figure 1). These button-like interfaces employ an inflatable soft rubber bladder that can expand and contract, mimicking a living, breathing entity. Air pressure sensors provide a variable input sig- nal to the computer based on the amount of pressure applied. This works like a traditional button. However, these buttons also push back, offering

Bimodal Affective Computing Interface for Emerging Artificial Intelligence Paradigms 157

human users an intuitive and empathic experience of what the computer is feeling, calibrated to the relative inflation of the surface of the button. IRIs are capable of conveying machine affect as well as human affect.

Within the traditional Affective Computing paradigm, wearables, cameras, and other sensors are common interfaces that measure the arousal of the autonomic nervous system. This provides a measurement of human affect. Electrodermal Activity (EDA) and Electroencephalogram (EEG), for exam- ple, can reliably indicate anxiety or stress in users (Picard et al., 2015). Heart rate variability is another commonly used measurement of autonomic ner- vous system activity (McDuff et al., 2014). These unconscious physiological parameters can be correlated to other parameters that give a fuller picture of users’ overall affect and broader emotional state. These data can be applied to the design of interfaces that are more responsive and adaptive to (human) users’ emotional valence. We began our investigation by asking what the equivalent of these human autonomic responses would be for displaying and interpreting machine affect.

Figure 1: Mouse prototype with IRI buttons.
Original PDF Figure 1:

This first generation of IRIs are designed to communicate with the auto- nomic nervous system of the computer, so to speak. For example, CPU usage can be translated into feedback that informs the response of (human) users (See Figure 2). This kind of machine affect is somewhat analogous to pulse or respiration in humans. Other examples might include graphical user inter- faces that assign a symbolic language, such as a pinwheel or a pulsing LED, to indicate when a computer is “thinking.” In practice, however, this usually reads as “frozen.”In addition to adding friction and frustration to the human- computer experience, this visual feedback is not bimodal in a strict sense. They do not reflect true artificial consciousness or machine agency. However, they may give some insight into the “unconscious” affect of machines, and in doing so into human consciousness as well. This could provide a foundation upon which to further develop systems that are truly bimodal.

Figure 2: Screenshot of computer program that reads CPU utilization, sends data to the mouse, and receives and visualizes the data.
Original PDF Figure 2:

The mouse interface developed in this prototype is something equivalent to holding hands with the computer (see Figure 3). It provides a degree of embodiment to the AI system through touch and haptic feedback, by which the AI system could communicate its “feelings,” or at least its level of arousal to a (human) user. This prototype, based on the common computer mouse, offers a familiar interface that is intuitive and easy to use. With the change in CPU usage, for example, the computer “breathes”faster or slower, depending on how much it is being taxed. This is equivalent, in some sense, to the anxiety and fatigue a (human) user might experience, expressed through the auto- nomic nervous system. These inflatable bladders, in addition to other stimuli such as light and haptic feedback, allow the (human) user to respond based on their affective state, establishing a conceptual foundation for empathy between machines and people.

Figure 3: User and IRI prototype.
Original PDF Figure 3:

Bimodal Affective Computing Interface for Emerging Artificial Intelligence Paradigms 159

We have developed this interface prototype to validate the concept of Bimodal Affective Computing, which can be characterized as an HCI paradigm that establishes affordances for machines to recognize, interpret, and respond to human affect, as well establishing affordances for humans to recognize, interpret, and respond to machine affect. Questions about the degree to which computers have or will have emotions and affect, and true subjective experiences, is a subject of ongoing debate. And even though arti- ficial consciousness and AGI does not yet exist, its possibility does. It is our hope that this project will stimulate conversation about how relationships between humans and machines might develop in the months and years ahead, as the hard distinction between human affect and machine affect blurs.

IRI Technical Details The prototype includes an Arduino Pro Micro which can appear to a com- puter as a mouse. The Arduino is connected to two air pressure sensors - these sensors use HX710b ADCs and are rated for 0–40KPa. The Arduino is also connected to an L293D motor driver IC to control two peristaltic pumps (see Figure 4). Each inflatable button is connected to a pump and a pressure sensor. The software receives a target value for what pressure each button should be pressurized to through serial communication from a com- puter. The Arduino’s control loop for pressure runs the pumps at a speed proportional to the error between the target pressure and measured pressure. If the measured pressure rises above the target pressure plus an additional threshold, that is interpreted as a person pressing on the button. When a but- ton is pressed, the Arduino can act like a USB mouse and send a mouse press command. For our prototype, we used Processing, running on a computer to get the percentage of CPU utilization from the computer. Processing calcu- lates pressure setpoints for the mouse that rise and fall at a rate proportional to the CPU utilization, and the program sends the setpoints to the Arduino. The Processing program also receives data sent from the Arduino in order to visualize the measured pressure in each button and whether either button has been detected to be pressed.

Figure 4: Components of the mouse prototype.
Original PDF Figure 4:

Related Work

The applications of IRIs for Bimodal Affect Computing, and AI applica- tions more generally, situate themselves in relationship to several notable precedents.

At the University of Michigan, the Haptix Lab is researching the use of inflatable tubes applied to steering wheels for better feedback between the vehicle computer and the human driver (Báez et al., 2023). This device provides users with inflatable interfaces for tactile, task related feedback, inflating and deflating a bladder behind the steering wheel. These tubes also provide the computer with pressure data.

In their study, Shultz and Harrison developed a thin, tactile interface capa- ble of rapid and significant surface deformation within a millimeter-scale footprint, utilizing Embedded Electroosmotic Pumps (EEOPs) (Shultz and Harrison, 2023). This system showcases pumps that are directly controlled and powered by applied voltage, with a mere thickness of 1.5mm, yet capa- ble of moving fluid quickly, generating pressures sufficient to create dynamic millimeter-scale tactile features. These prototypes demonstrate how complex tactile feedback might be integrated into compact interfaces, crucial for the further development of BAC applications.

Further work demonstrates the viability of integrating systems simi- lar to IRIs into wearables. Shen, Rae-Grant, Mullenbach, Harrison, and Shultz introduce haptic technology that integrates high-density electroos- motic pump arrays into the fingertips of gloves, allowing for intricate tactile sensations in virtual environments (Shen et al., 2023). These include the perception of contact textures, pressure variations, and the movement of dynamic objects. The gloves achieve detailed tactile feedback with a high resolution of 20 tactile pixels per square centimeter through the use of electroosmotic pump technology.

CONCLUSION

Despite rapid technical achievements in the field of AI, our species still lacks a nuanced understanding of what intelligence and consciousness actually are. Indeed, the quest for a fuller understanding of intelligence and consciousness may be the greatest spiritual and scientific goal our species can set for itself. It may come to pass that the greater achievement in the development of artificial intelligence is not merely the arrival of AGI, but perhaps more importantly, the discovery of profound insights into the emergent mechanisms of con- sciousness and intelligence. Like fictional characters such as Dr. Frankenstein or Geppetto, we may only come to understand ourselves by looking into the mirror of our creations… as they look at us (Shelley, 2008; Collodi, 1989). Only then may we really know ourselves. We may come to understand that Artificial Intelligence isn’t so artificial after all, which is to say, alienated from what we assume to constitute the “natural”human subject. Like all technolo- gies, AI is a reflection and extension of our human selves, and we humans are a reflection and extension of nature and its unfolding processes. Perhaps we might understand the emergence of AI differently if we understood tech- nology as part of this more encompassing expression of nature; as a natural evolution of consciousness, in both its human and non-human forms.

Bimodal Affective Computing Interface for Emerging Artificial Intelligence Paradigms 161

The question of what constitutes intelligence and consciousness will likely need to take into account the role emotions and desires play in forming expe- rience. General intelligence (or g-factor) resists reduction to a specific set of criteria for intelligence (Jensen et al., 1994). Although certainly part of the bigger picture, new conceptual frameworks will likely be needed to take into account emotional intelligence as well. General Intelligence, like most of our experiences, is more than the sum of its parts. Intelligence and con- sciousnesses cannot be reduced to the mechanisms described by rationalism or materialism alone. As poets, artists, and others have long known, emotion plays a vital role in our awareness of, and investment in, our experiences. This is why a theory of machine affect is crucial to understanding Artificial Emotional Intelligence.

ACKNOWLEDGMENT

This work originated in the Brown University Computer Science course, CSCI1951C Designing Humanity Centred Technology taught by Professor Gonsher. The Computer Science department provided funding to support this project.

REFERENCES

Báez, H., Bhardwaj, A., Costa, J., Gideon, J., Modhrain, S., Sarter, N. and Gillespie, B. (2023). Communication is a Two-Way Street: Negotiat- ing Driving Intent through a Shape-Changing Steering Wheel. In: 2023 IEEE World Haptics Conference (WHC). Delft, Netherlands: IEEE, pp. 134–140. doi: 10.1109/WHC56415.2023.10224458. Blade Runner (1982) Directed by Scott, R. [Film]. Warner Bros. Blade Runner 2049 (2017) Directed by Villeneuve, D. [Film]. Warner Bros. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T. and Zhang, Y. (2023). Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint. doi: 10.48550/arxiv.2303.12712. Collodi, C. (1989). The Adventures of Pinocchio. New York, NY: Macmillan. Dick, P. K. (1968). Do Androids Dream of Electric Sheep? New York, NY:

Doubleday. Jensen, A. R. and Weng, L.-J. (1994). What is a good g? Intelligence, 18(3), pp. 231–258. doi: 10.1016/0160-2896(94)90029-9. McDuff, D., Gontarek, S. and Picard, R. (2014). Remote measurement of cognitive

stress via heart rate variability. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Chicago, IL: IEEE, pp. 2957–2960. doi: 10.1109/embc.2014.6944243. Picard, R. W. (2000). Affective computing. Cambridge, MA: MIT Press. Picard, R. W., Fedor, S. and Ayzenberg, Y. (2015). Multiple Arousal Theory and Daily-Life Electrodermal Activity Asymmetry. Emotion Review, 8(1), pp. 62–75. doi: 10.1177/1754073914565517. Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences,

3(3), 417–424. Shelley, M. (2008). Frankenstein or The Modern Prometheus. Edited by Michael Kennedy Joseph, Oxford University Press.

Shen, V., Tucker Rae-Grant, Mullenbach, J., Harrison, C. and Shultz, C. (2023).

Fluid Reality: High-Resolution, Untethered Haptic Gloves using Electroosmotic Pump Arrays. In: UIST’23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. San Francisco, CA: ACM, pp. 1–20. doi: 10.1145/3586183.3606771. Shultz, C. and Harrison, C. (2023). Flat Panel Haptics: Embedded Electroosmotic Pumps for Scalable Shape Displays. In: 2023 CHI Conference on Human Factors in Computing Systems. New York, NY: ACM, pp. 1–16. doi: 10.1145/3544548.3581547.

Cover of the proceedings volume Human Factors and Systems Interaction

Explore the complete digital volume

Human Factors and Systems Interaction

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-30-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →