Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Karaoke Game with Body Movement Tracking for Investigating Singing-motor Interactions: System Proposal

Graduate School of Engineering and Science, Shibaura Institute of Technology, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1002087

Abstract

This paper presents and discusses the design, implementation and initial results of a game system using a singing-motor paradigm, using karaoke singing combined with a human-computer interface through body movement recognition. The purpose of this system is to collect experimental data on how karaoke singing and body movement interact and how they affect each other, in order to analyze the data from the perspective of human-machine interface, immersion, gameplay and entertainment. At the end, the results and outputs of the development are presented, and some follow-up proposals are discussed.

INTRODUCTION

Along with the constant evolution and growth of the games industry, the human interfaces for gaming also evolved significantly, from simple 2 but- ton controllers to sophisticated multimodal gesture interfaces (da Luz, 2014). More specifically, the Music Games genre has taken advantage of the inter- face progress to innovate human interaction possibilities such as: real-like instruments, floor stepping sensor for dancing, hand movement in Virtual Reality and even full body tracking.

When it comes to music and rhythm game sessions, players usually are required to use a lot of focus, motor coordination and good timing (Sal- sabilla et al., 2020, Song et al., 2019). However, multitasking (also called dual-tasking in the cognitive science field) is a big challenge for the human brain, especially when it comes to tasks that require a high level of attention (Pashler, 1994). Therefore, considering the cognitive challenges of multita- sking and a multimodal body movement interface, this study introduces a game system with a singing-motor paradigm, in which players have to sing karaoke and perform body movements simultaneously. While singing, the player will be challenged to break 3D obstacles that spawn in rhythmic pat- terns, using his own body to control an in-game avatar and perform poses, punches and kicks in the correct timing. The purpose of the system is to inve- stigate how the singing-motor tasks interferes with each other, including how

© 2022. Published by AHFE Open Access. All rights reserved. 261

262 Pedroso and Horie

musical aspects of the game design can be used to obtain different results in terms of difficulty, entertainment, and game experience.

METHOD

The system configuration consists of 2 modules: The first module is a 3D game module developed in Unity 3D (Unity Technologies) environment, a game development engine, that handles the GUI and performs the body tra- cking tasks. For each song, the module receives as input a set of files that includes the music audio, lyrics, and the level timeline containing position and spawn timing of obstacles. As output, it saves a file containing the pla- yer’s full body positions over time for each game session. The second module runs in a smartphone and it is responsible for the karaoke management, pla- ying the background music and reading voice data from the player to evaluate the player singing skill. High scores mean the player has a good singing per- formance. The application receives as input the music audio and lyrics files, and saves the voice captured by the microphone during the game session. Additionally, the application sends music timing and score data in real time to the first module via TCP/IP communication.

During the game, a 3D humanoid avatar replicates player movements made in real time, while obstacles appear in the form of asteroids that are broken through the correct player interaction. The avatar resource asset is a 3D model named “Unity Chan” (© Unity Technologies Japan/UCL), and the obstacles asset is named “Breakable Asteroids”, both from Unity 3D asset store, under the Standard Unity Asset Store EULA License agreement (Unity Technologies 2020).

The game user interface is performed by a Microsoft Azure Kinect (Micro- soft Corporation). This device consists of a set of cameras and depth sensors that can retrieve image and 3D information from the environment and pro- vide the position data of all joints from detected human bodies (Microsoft Corporation 2021). Thus, in the game, the Kinect device detects the pla- yer’s full body position, controlling the in-game avatar in real time. In order to enable investigation of different motor coordination challenges, the game provides 2 different modes to be combined with singing: Posing mode and Fighting mode, to be detailed below.

Game Mode: Posing In this game mode, the player is challenged to break obstacles flying from the front. For break the obstacles, the player has to make instructed poses in particular timing while the keeping good performances of singing a karaoke song. Figure 1 shows the implemented game interface running in Pose mode (parts of showing lyrics are omitted in the figure).

For instructing poses, a horizontal bar guides the player to reproduce the body positions that are displayed, while obstacles (asteroids) fly in the ava- tar’s direction. Each pose corresponds to 1 obstacle. The guide bar scrolls pictures from right to left of the screen and the player should be at the correct position when the picture reaches the red line in the middle. At this moment, the player’s body joints orientations are registered and compared with the

Karaoke Game with Body Movement Tracking for Investigating Singing-motor Interactions 263

Figure 1: Game interface demonstration in pose mode (© Unity Technologies Japan/UCL).
Original PDF Figure 1:

target body position. Then, a precision score is generated from 0 to 100 as the evaluation of player’s motion and classified as Perfect, Good, Fair or Bad. A text feedback corresponding to the precision score is presented.

For feedback of singing performance, the microphone icon shown at the left part of Figure 1 is introduced. The system receives the singing evaluation data from the smartphone module in real time, ranging from 0 to 100 points and displays it by filling the microphone icon. An empty icon means bad singing performance, while a full icon means perfect singing score.

In addition, there’s a singing score threshold. Below the threshold, the microphone icon filling will be red, meaning disability of breaking obsta- cles even with making the pose successfully. Above the Threshold, the icon filling will be green and the avatar displays a particle aura, indicating that the singing score is good and breaking obstacles can be done by performing good posing.

Thus, the amount of points the player receives after each pose is calculated and in the following way, and shown in Table 1: If singing score is over the threshold and the pose score is classified as Good or Perfect, both scores will be multiplied and added to the game’s total score. If the current singing score is bellow threshold, resulting score will be 0, no matter the posing. Similarly, if the pose is considered Fair or Bad, the resulting score will be 0.

Finally, in order to encourage the player to work hard both in singing and moving, a mechanism was introduced: The Power Bar, showed in the right part of the screen in Figure 1. In Pose Mode, the resulting score is added to the game’s overall score and, also, fills the power bar with energy. At the end of each short song session, a big obstacle appears and can only be broken if the power bar has a minimum amount of energy. In other words, if the player focuses on only singing or only posing, the resulting score will be low, leading to a failure when breaking the big obstacle. The difficulty and amount

264 Pedroso and Horie

Table 1. Score calculation method for Posing mode.

Table available in the original PDF paper.

Singing Score input Pose Score input Result: Obstacle Result: Obtained Score Over threshold Good or Perfect Broken Singing * Posing Over threshold Bad or Fair Not broken Zero Below threshold Good or Perfect Not broken Zero Below threshold Bad or Fair Not broken Zero

of score/energy needed, as well as the threshold values are configurable to provide gameplay experience tweaks.

Game Mode: Fighting In this game mode, the player can move freely to control the avatar hori- zontally. The obstacles come flying in the avatar’s direction, and the player should hit punches and kicks to break it and earn points. Similar to thje Posing mode, it is required to have a good motion score and a good singing score combined in order to be able to break obstacles. Minimum requirement of singing performance is indicated by the microphone icon and the particle aura is same as in the pose mode.

Additionally, in order to benefit the immersion and encourage wide move- ments (in other words, prevent players from “cheating the system”), the speed of blows is calculated in real time and taken in consideration, meaning that slow and weak punches/kicks aren’t considered.

RESULTS

After the requirements planning and designing phases, the game system was developed, implemented and tested successfully. It started with a functional and simple prototype and, as the system was being tested and used, needs for new mechanics and features emerged, resulting in the configuration cur- rently presented in this study. For instance: The power bar and fighting aura mechanics were introduced to prevent players from focusing on only singing or only moving to earn points, while rewarding players who perform wide and strong gestures.

At last, for testing purposes, some subjects were invited to play the game, validating the functionality and other technical aspects of the system such as: game mechanics, communication between modules and computational performance.

The results are a playable and functional game system, correctly recording the data needed to carry out future experiments. A preliminary interview with the testers showed that, as expected, the multitask paradigm of singing- motor coordination is a hard challenge for the players, and the familiarity with the lyrics and the song is a key factor for the performance. Finally, a preliminary adaptation of the system for Virtual Reality (VR) environment was implemented. The game mechanics and architecture were not changed, but the deployment with a VR device using first person point of view was achieved successfully.

Karaoke Game with Body Movement Tracking for Investigating Singing-motor Interactions 265

DISCUSSION

Although pose and fight modes may seem a little similar, there’s a crucial dif- ference when evaluating it from the multitask paradigm perspective and its interaction with music: fighting means doing a quick movement at the right time, while posing means being still in an exact position at the right time. Because humans have a natural rhythmic connection with music through the pulse sensation named as beat (Ficth, 2011), the modes will result in different motor interactions between the body and the music, since fighting implies in a strong movement during the occurrence of the beat, while posing means be stopped in the target position during the occurrence of the beat. Thus, we beli- eve that these modes can provide freedom and a wider range of possibilities regarding design of experiments.

Human being presents a natural interaction with rhythm, that can be exemplified in terms of the cognitive process by inferring beats from pat- terned acoustic stimulus and generating motor outputs such as dancing, clapping or playing drum patterns (Fitch, 2011). Therefore, with these conce- pts in mind, the next steps of this study will be towards the design of experiments. More specifically, we intend to study and analyze the singing- motor paradigm under the perspectives of game design and human interface, including musical aspects of the song, such as strong beats, weak beats, out of beat, complexity of rhythmic patterns and similarity with the melody sung in karaoke. With the study’s results, we intend to seek answers for questi- ons such as “How to make the singing-motor challenge more entertaining?”, “How to design it easier or harder?”, “What are the best methods to generate rhythmic motor challenges considering the musical aspects of a song?”.

CONCLUSION

This study showed the design and implementation of a game system that uses karaoke singing and performing body movements simultaneously in order to investigate the singing-motor relation between these tasks and music. The system architecture was presented and its modules were explained, followed by introducing the game mechanics and modes. The successful implemen- tation of the system was presented and, to encourage players to strive in both singing and moving, two game modes were created. Moreover, the cognitive aspects of human and rhythm interaction are briefly discussed as a background for our motivation and the future design of experiments.

For next steps, we expect to design several singing-motor scenarios, conduct the experiments, further analyzing the collected data. For future improvements, it is planned to completely adapt the game system to Vir- tual Reality environment, increasing immersion and widening the system’s application possibilities.

REFERENCES

da Luz, A. (2014), “Evolution of the physical interfaces in videogames as a support to the narrative and the gaming experience.” Fitch, W. (2011), “The biology and evolution of rhythm: Unraveling a paradox.”

Language and Music as Cognitive Systems.

266 Pedroso and Horie

Microsoft Corporation 2021, “Azure Kinect DK hardware specifications”, viewed 12 January 2022, <https://docs.microsoft.com/en-us/azure/kinect-dk/hardware- specification>. Pashler, H. “Dual-task Interference in Simple Tasks: Data and theory.” Psychol. Bull.

116(2), 220–244 Salsabilla, G. A., Fabroyir, H., Herumurti, D., Kuswardayan, I., and Hidayati, S.

C. (2020). “Tachyon: Multiplatform Rhythm Game with Automatic Beatmap Generation.” CENIM 2020 - Proceeding: International Conference on Computer Engineering, Network, and Intelligent Multimedia 2020, 162–167. Song, D. H., Kim, K. B., and Lee, J. H. (2019). “Analysis and evaluation of mobile rhythm games: Game structure and playability.” International Journal of Electrical and Computer Engineering, 9(6), 5263–5269. Unity Technologies 2020, “Asset Store Terms of Service and EULA.” viewed 21

December 2021, <https://unity3d.com/legal/as_terms>

Cover of the proceedings volume Human Factors in Virtual Environments and Game Design

Explore the complete digital volume

Human Factors in Virtual Environments and Game Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-26-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →