Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Designing Music Training Systems for Deaf and Hard-of-Hearing Individuals: Insights from Multi-Element Perception Tasks

1 Tsukuba University of Technology, Ibaraki 3058520, Japan ↗
2 University of Tsukuba, Ibaraki 3058577, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1006949

Abstract

Many Deaf and Hard-of-Hearing (DHH) individuals enjoy listening to music and singing. However, the way music sounds to them varies from person to person, and even among those with the same level of hearing loss, perceptions differ. Furthermore, DHH individuals focus on different elements when listening to music. Although music training systems for DHH individuals have existed before, many were designed and implemented in laboratory settings. We aimed to create a music training system for DHH individuals that could be used playfully in real-world environments, such as on their own smartphones or tablets, and for researchers and educators to provide an easily extensible framework. This presentation describes the versatile framework and trial experiments with DHH participants. The results revealed the importance of selecting training materials and setting training difficulty levels, especially for DHH participants whose music perception is affected by several factors, including hearing levels, interest in musical elements, and musical experiences. We envisioned a music training system integrating subsystems designed to help DHH individuals hear more of what they want to hear when listening to music. The subsystems share the same user interface, presenting the target audio and four alternative audio samples. Separate instructions indicate which selection to make, without visual or haptic cues. As users must listen carefully to the target sound and compare it with the alternatives, we call this subsystem “Music Memory.” We created Music Memory for four musical elements using this simple interface. We set problems for each of these: 1) selective listening, 2) instrument identification, 3) recognizing melody or rhythm variations, and 4) tempo recognition. To investigate whether the difficulty levels set for each of the four Music Memories were appropriate and whether the acoustic data for the alternatives presented matched the target audio, experiments were conducted with DHH participants. The problem audio sources were selected from J-POP and anime songs, which DHH frequently listens to. Between 8 and 20 young DHH participants took part in the four experiments. The DHH participants in the four Music Memory experiments differed (with some overlap).The accuracy rates for the four memory experiments were: 1) mostly ceiling effect observed, 2) ceiling effect observed, 3) slightly above 50%, and 4) slightly below 25%, namely below the chance level. For condition 3, a tendency was observed where higher musical experience correlated with higher accuracy rates. The accuracy rates, though averages, indicate the problem settings that are either too easy or too difficult. This suppresses the user's motivation to continue training and makes it difficult for trainers to refine the provided materials. For use in training, the slightly above 50% accuracy rate observed in 3) might be appropriate among the four Music Memories. However, in 3), concerns were raised about the lack of criteria for creating alternatives involving melody or rhythm alterations, potentially leading to subjective provision of acoustic data.These findings highlight the necessity of carefully designing training difficulty levels and instructional materials before applying the system in practice. Not only for 3), but all three other Music Memories need the objective provision to provide alternatives. For trainees to keep using Music Memory, the interface will need to give feedback on their answers, and we may consider additional sensory modalities to reduce the burden of using Music Memory for DHH people. This research has established a foundation for the practical implementation of a music training system for DHH individuals. However, future challenges include refining the personalized adaptation algorithm, verifying long-term training effects, and conducting evaluations with a larger participant sample. Then the system will contribute to the realization of music training environments that are optimized for each individual DHH and to construct inclusive music education environments. This research indicates a new research direction in the interdisciplinary field of assistive technology for the DHH people and music education.

INTRODUCTION

With the advancement of hearing devices, whether hearing aids or Cochlear Implants, an increasing number of Deaf and Hard of Hearing (DHH) individuals can access music and listen to it. On the other hand, there are DHH people who enjoy music with little sound information. The inquiry to DHH university students about their music activities revealed that 88% of 59 students answered “yes” to the question, “Do you like to listen to music?” Regarding the other question, “Do you like to sing songs?” 88% of students also answered “yes.” On the other hand, the rates of other music activities, such as dancing, attending live performances, and playing musical instruments, were low. They seem to enjoy music in their own way. Bleckly

© 2025. Published by AHFE Open Access. All rights reserved. 1374

Designing Music Training Systems for Deaf and Hard-of-Hearing Individuals 1375

reported on a survey of 131 articles and showed that the appreciation of music is not well investigated (Bleckly, 2024).

Since DHH people are unable to have passive music experiences, such as hearing background music in a shopping mall or café, increasing their exposure to music may provide them with better access to music experiences. Several music training systems have been proposed recently (Boyer, 2022; Calvino, 2025; Bissmeyer, 2022). Both Boyer’s and Calvino’s studies used a training system called “Meludia”. The target audience spans a wide range of ages and musical experiences. Additionally, a Cochlear Implant company introduces it to Cochlear Implantees. Meludia intends to train in the five “dimensions” of melody, rhythm, harmony, specialization, and formulation. Meludia mentioned the use by Cochlear Implantees, but the major target users are hearing people. In order to build music training for DHH, we need to know how they perceive music. Bissmeyer’s system focused on the musical interval only.

Our undergraduate students are all DHH, and we understand their interest in music varies; some like to get rhythm, some pitches, etc. Therefore, we designed and developed our own music training system, which enables users to experiment with various music elements independently, all while maintaining a consistent user interface. Then, by accumulating those sub- training systems that handle separate music elements, users can enjoy training multiple music elements selectively.

Through the feasibility test of our music training system, we aimed to understand our research question: what we need to know to set the difficulty levels in music training. Our contribution to music training for DHH is that we demonstrated the feasibility of a simple user interface for music training incorporating several musical elements, and clarified the issues associated with setting difficulty levels for music training.

In this paper, we describe our sub-training systems, including the interface and experiments with four types of music elements. Then we describe the results of each Music Memory, focusing on the correct rates. It was found that determining difficulty levels is essential, but it is also very challenging. Additionally, we encountered the challenge of selecting materials for training.

Music Memory: A Music Training Subsystem for Single Musical Elements Using a Simple UI Since each DHH individual focuses on different music elements, such as some on beat and some on timbre, we developed a music training system with a uniform user interface that allows training of any music element. The subsystems share the same user interface, presenting the target audio and four alternative audio samples. Separate instructions indicate which selection to make, without visual or haptic cues. As users must listen carefully to the target sound and compare it with the alternatives, we refer to this subsystem as “Music Memory.”

For example, we describe a training session in the case of Music Memory, which asks users to judge the tempo of a target music. The instruction to users is “select a tempo that is the same as the target audio.” Users can listen to the

1376 Hiraga and Terasawa

target audio by clicking the button with the 8th note. Then they listen to four alternatives that show the regular tempi. Users can listen to both the target audio and alternative sounds as many times as they want to. Finally, a user chooses one of the four alternatives. Figure 1 displays a screenshot that shows the Beat Per Minute (BPM, tempo) of the target sound and its alternatives, as well as whether the user’s choice was correct. The trainer can provide as many sessions as necessary for the same instruction, using different sets of target sounds and alternatives.

In addition to tempo recognition, trainers can select the instruction and sound materials related to music elements, as described in the Experiment section.

Music Memory is implemented with Flutter and runs on iPhone or iPod touch.

Figure 1: User interface of music memory (for tempo recognition).
Original PDF Figure 1:

Four Types of Music Training With Music Memory We prepared four types of instructions with a common interface for Music Memory. So far, researchers who have an interest in music, computing, and music activities by DHH people (we call them trainees of Music Memory) have prepared four different musical elements to build Music Memory. The music elements are 1) selective listening, 2) instrument identification, 3) recognizing melody or rhythm variations, and 4) tempo recognition. From 1) to 3), the trainee used J-POP as music for the target sounds. The target

Designing Music Training Systems for Deaf and Hard-of-Hearing Individuals 1377

sounds of 4) are either a very simple note series or music from anime songs. Thus, we use the term “music elements” in a broader sense, encompassing not only how people listen to music but also the traditional sense of music elements, such as tempo, melody, and harmony. All trainees provide different levels of training sets.

Selective Listening Hearing people acknowledge melodies or lyrics while the music plays is the result of the accumulation of several instruments and singing voices. In real life, people can hear sounds that are a mixture of several types of sound and still identify the specific sound. In the training of selective listening, the target music is the original J-Pop music with the vocal part. The instruction is “to select one that is the same as the target music.” The alternatives are music of the same or different melodies. There are four levels, based on the number of mixed sounds in the target sounds: vocal only, vocal and drums, vocal and instruments other than drums, and vocal with drums and other instruments, ranging from the easiest (Level 1) to the most difficult (Level 4). Each level consists of 10 sessions. We call this Music Memory “M1.”

Instrument Identification There have been several studies on the perception of musical timbre by DHH people (Driscoll, 2012, Macherey, 2013, Jiam, 2019). Before using Music Memory for sustained music training, this experiment aimed to determine the suitable difficulty standard for instrument identification (Akaki, 2022). The target music is the melody of J-POP played by a single instrument. The instruction is “to select one that is the same instrument as the target music.” Alternatives played a different melody from the target music. There were three levels, each comprising five instrument combinations (for a total of 15 sessions). We call this Music Memory “M2.”

Recognizing Melody or Rhythm Variations The trainee focused on melody, specifically the melody contour and its rhythm pattern. Several past studies have demonstrated that Cochlear Implantees can recognize melody contours (Galvin, 2007) and the importance of rhythm patterns (Kong, 2004). In the past studies, the materials used in the experiments were simpler compared to the actual music we listen to daily. Besides, there were no investigations to see the effect of modifying a part of the pitch or rhythm in the melody. The target music was a J-POP melody lasting four to eight measures. The instruction was “to select one that is the music as the target music.” There were two types of alternatives: 1) modifying a part of the melody, and 2) modifying a part of the rhythm in the melody. Each type provided three alternatives. For both types, there were two levels. Each level consisted of eight sessions. We call this Music Memory “M3.”

Recognizing Tempo Some studies on hearing loss and tempo recognition have shown that children with hearing loss are affected in their rhythmic abilities (Hidalgo, 2021), and another study has demonstrated that rhythm processing is not intact for

1378 Hiraga and Terasawa

Cochlear implant recipients (Valentin, 2025). There are no widely accepted findings regarding the perception of tempo among DHH individuals. In this Music Memory, the target music varies from just the beat beep to anime songs. The instruction is “to select one that is the tempo of the target music.” Alternatives showed the tempo beep. There were three levels, each included five target music (altogether 15 sessions). We call this Music Memory “M4.”

EXPERIMENT Procedure of the Experiment

Although we conducted the four experiments using four different Music Memories, the procedure is common to all the experiments. Since this is not a laboratory-based, rigid experiment, we wanted to know how DHH individuals listen to music in their daily listening environments.

After explaining the experiment, including its purpose, how to use the system, and the rights of experimental participants, the experiment proceeded at the participants’ convenience, at any location, and at any time. We also asked them to put down inquiries to collect demographic data. The experiment was approved by the Ethical Committees of the trainers’ institutions.

We use the Music Memory task for “recognizing tempo” as an example to describe the experiment in more detail. The sound material as the target music are: 1) a regular beat, 2) a melody played by the piano, 3) a music piece with the melody and the accompaniment part played by the piano, 4) the melody of an anime song only played by the piano, and 5) the same music piece as 4) played by several instruments. The tempo of the target music is automatically obtained. The difficulty levels are determined based on the difference from the target music. If the tempo of the target music is x, then the tempo of the alternatives is as follows: Easy: either (x-15, x, x + 15, x+ 20) or (x-20, x-15, x, x + 15) Normal: either (x-10, x, x + 10, x + 15) or (x-15, x-10, x, x + 10) Hard: either (x-5, x, x + 5, x + 10) or (x-10, x-5, x, x + 5) Table 1 shows the number of sessions and participants of all four Music Memories. Although the group of participants for the four Music Memories differs, their hearing levels range from 70 dBHL to 120 dBHL for all groups. Most of them are prelingually deafened.

Table 1: Sessions and participants of the four MMs.

Table available in the original PDF paper.

The Number of Participants

Type of Music Memory

Total Number of Sessions

The Number of Levels DHH Hearing

M1 40 4 8 0 M2 15 3 20 0 M3 Melody 16 2 16 6 Me Rhythm 16 2 16 6 M4 15 3 20 3

Designing Music Training Systems for Deaf and Hard-of-Hearing Individuals 1379

RESULTS

Table 2 shows the participants and correct rates of DHH participants of all four Music Memories. The ranges found in the correct rates correspond to the levels.

Although the correct rate of M1, which involves selecting the same music as the target music (selective listening), exhibits a ceiling effect, the participants were not exceptional compared to the other three Music Memories. Among the eight participants, three listen to music every day. Their hearing devices vary; the combinations include a Cochlear Implant in one ear and none in the other, a Cochlear Implant and a Hearing Aid, and Hearing Aids in both ears. The correct rates of M2 and M3 indicate that the difficulty levels were well-defined. On the other hand, M4’s correct rate is very low. The results of three hearing peers participating in the experiment with M4 show a similar average.

Table 2: Results of the four MMs.

Table available in the original PDF paper.

Type of Music Memory Correct Rate (%) M1 Almost 100 M2 70∼85 M3 Melody 51∼59 M3 Rhythm 63∼72 M4 Less than 25

DISCUSSION

The Correct Rate and the Difficulty Level The level set in M1 and M4 did not work well. On the other hand, M2 and M3 set it better. The possible reason for the high correct rate of M1 is that we could not expect the results because there have been no studies on selective listening in music by DHH people. The very low correct rate of M4 is because we did not understand the threshold of tempo recognition. The difference between M3 Melody and M3 Rhythm aligns with the fact that time discrimination is easier for DHH than frequency discrimination. In general, because of the few studies related to music perception by DHH individuals, we have no established and common understanding of their music perception. Of course, there are not a few reports as in the references. However, we are not certain that the results of each experiment will always be the same across different groups of participants. Even M2 and M3 trainers decided the difficulty levels based on their own experiences. Thus, we need to explore the research on music perception by DHH in order to develop a stable music training system.

Toward a Music Training System For the research question of what we need to know to set the difficulty levels in music training, the experiment clarified the issues associated with setting difficulty levels for music training. Although the primary purpose of our

1380 Hiraga and Terasawa

experiments was to explore the difficulty level, several other issues must be addressed for Music Memory to be utilized as a real music training system. These include the gamification strategy to encourage trainees to continue training and a system to evaluate the effectiveness of the training. At least the evaluation of the training is not well established for the same reason as in the previous description (no established theory on music perception by DHH people).

User Interface The user interface of Music Memory is both simple and easy for trainers to design training content and materials, as well as for trainees to understand the system. To determine whether the common simple user interface is appreciated by participants, we may need to ask some participants to work on all four Music Memories.

Limitation We understand several limitations in these experiments. The number of participants in each experiment is very low, and we were unable to statistically analyze the results. We conducted all the experiments online. This yields the variety in the devices used in the experiments, and, of course, the location, which will introduce weaknesses into the results. On the other hand, we conducted the online experiments intentionally to observe how DHH people listen to music under everyday conditions.

Future challenges include improving individual adaptation algorithms, verifying long-term training effects, and conducting evaluations with larger participant groups. We will consider using other senses, such as haptics.

CONCLUSION

To improve the music accessibility of DHH people, we designed and implemented a music training system, “Music Memory.” With a common, simple user interface, we selected four types of music elements and conducted experiments to assess the feasibility of Music Memory. For the research question of what we need to know to set the difficulty levels in music training, we found that rich data from the study of music perception by DHH individuals is required, which is currently not available.

As described earlier, music activities for people with DHH are intensive in listening and singing. The music training system running on smartphones or tablets enables people to actively listen to music elements of interest anywhere, anytime. With the simple user interface, DHH children can also use it. Then, DHH individuals can increase their music accessibility independently.

ACKNOWLEDGMENT

This work was supported by JSPS KAKENHI Grant Number 21H00884.

Designing Music Training Systems for Deaf and Hard-of-Hearing Individuals 1381

REFERENCES

Akaki S, Hiraga R, Yasu K, Tabuchi K, Terasawa H. (2022) Design and Evaluation of Instrument Sound Identification Difficulty for the Deaf and Hard-of Hearing. Proc. APSIPA ASC, pp. 723–728. Bissmeyer SRS, Ortiz JR, Gan H, Goldsworthy RL. (2022) Computer-based musical

interval training program for Cochlear implant users and listeners with no known hearing loss. Front. Neurosci., 16. Bleckly F, Lo CY, Rapport F, Clay-Williams R. (2024) Music Perception, Appreciation, and Participation in Postlingually Deafened Adults and Cochlear Implant Users: A Systematic Literature Review. Trends in Hearing. 28. Boyer J, Stohl J. (2022) MELUDIA - Online music training for cochlear implant users.

Cochlear Implants Int. 23(5): 257–269. Calvino M, Gavilán J, Lassaletta L. (2025) Use of Meludia to evaluate the effect of music training in cochlear implant users: Study protocol of a randomised controlled study. BMJ Open. 15(4): e092207. Driscoll VD. (2012) The Effects of Training on Recognition of Musical Instruments

by Adults with Cochlear Implants. Semin Hear 33(04): 410–418. Galvin JJ 3rd, Fu QJ, Nogaki G. (2007) Melodic contour identification by cochlear implant listeners. Ear Hear. 28(3): 302–19. Hidalgo C, Zécri A, Pesnot-Lerousseau J, Truy E, Roman S, Falk S, Dalla Bella S,

Schön D. (2021) Rhythmic Abilities of Children With Hearing Loss. Ear Hear. 42(2): 364–372. Jiam NT, Deroche ML, Jiradejvong P, Limb CJ. (2019) A Randomized Controlled Crossover Study of the Impact of Online Music Training on Pitch and Timbre Perception in Cochlear Implant Users. J Assoc Res Otolaryngol. 20(3): 247–262. Kong YY, Cruz R, Jones JA, Zeng FG. (2004) Music perception with temporal cues

in acoustic and electric hearing. Ear Hear. 25(2): 173–85. Macherey O, Delpierre A. (2013) Perception of musical timbre by cochlear implant listeners: A multidimensional scaling study. Ear Hear. 34(4): 426–36. Valentin O, Vernam FNE, Intartaglia B, Schönwiesner M., Nozaradan S, Lehmann A.

(2025). Intact sensorimotor rhythm abilities but altered audiovisual integration in cochlear implant users. PLOS ONE, 20(4), e0320815.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-75-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →