Imran Ullah Khan, Sana Tahir, Jong Weon Lee (2023). Android-based Indoor Physical Activities Detection and Monitoring System using Pose Estimation. In: Human Factors in Aging and Special Needs. AHFE Open Access, vol. 88, pp. 142–152. https://doi.org/10.54941/ahfe1003671
In the last decade, innovation in the fitness and health industry has exponentially expanded, resulting in new devices like fitness trackers, calorie counters, diet planners, and running trackers. However, a large proportion of the population is elder people who need technical assistance to live their life healthy and independently. In order to progress the advancement in this research area, we concentrated on developing a mobile application that can recognize and assist human actions during exercise and store their records. In this study, a computer vision-based application is developed that can recognize and monitor indoor physical exercises of people especially elders through a smartphone camera. The application uses the pose estimation method to recognize the exercises and further counts the number of exercises and repetitions. This application can recognize 12 different physical exercises, which are further divided into three main categories such as upper body (dumbbell press, overhead arms, back twist, chest, shoulder, front arm), lower body (chair stand, leg raise forward/backward, side leg raise, squats, balance walk), and whole-body (pushups) exercises. An android application is developed which combines OpenPose, deep learning framework, and OpenCV for pose estimation, learning, and visualization while the android studio is used for application development. We use the Media pipe library for the detection and tracking of 33 different points in the human body, including face points. For pose estimation and activity recognition, we calculate different angles between body-detected joints, which are used to recognize the exercise. Moreover, we also measure how many repetitions the user performs in a specific exercise. Finally, the application was tested on 15 different users, which gave positive feedback, and the results were quite accurate for detecting and counting the exercises. Furthermore, the application is very useful for elderly people who mostly stay in their homes or people who can’t join the Gym outside due to their busy schedules.
Physical activity recognition is a field of study that aims to understand and track the movements and exercises of individuals. In elderly populations, it plays a crucial role in maintaining physical health and independence, as well as improving overall well-being. This can be achieved through wearable
devices (Attal et al., 2015; Lara & Labrador, 2012), such as smartwatches (Mekruksavanich & Jitpattanakul, 2020), smartphone sensors (Hassan et al., 2018; Kwapisz et al., 2011; Yang et al., 2010) or fitness trackers (Nguyen et al., 2015; Weiss et al., 2016), which can monitor physical activity levels and provide feedback and recommendations to encourage healthy habits. Additionally, physical activity recognition can also be used to monitor and detect potential health concerns, such as falls detection (Tran et al., 2016), and provide timely interventions to address these issues.
Android-based physical activity detection and repetition counting have numerous applications and uses in various fields. In healthcare (Subasi et al., 2018), they are used to monitor and track physical activity levels in patients and elders, such as those with chronic conditions or post-operative rehabi- litation. In sports (Ghazali et al., 2018; Hsu et al., 2018), they can be used to track training progress and identify areas for improvement. Additionally, these applications can be used in research settings to collect and analyze large amounts of physical activity data for studies on human movement and exercise. Overall, android-based physical activity monitoring and repe- tition counting provide a convenient and accessible way to monitor physical activity, improve performance, and promote healthier habits.
In fitness and wellness, these applications provide users with real-time feedback on their exercise form and repetition count. This information is particularly useful for users who are trying to build strength and improve their performance. The application can also be used to track overall phy- sical activity levels and provide users with personalized recommendations for healthy habits. The versatility and accessibility of android-based phy- sical activity detection and repetition counting make them a valuable tool for a wide range of applications. They provide users with the information they need to optimize their physical activity, improve their performance, and promote healthier habits. This application has a proper GUI where the user will register himself/herself, login to their account performed different acti- vities and the record of these activities will be stored in database for future use. There are 12 different types of physical activities used in this applica- tion which are mainly categorized in three categories such as upper body exercises, lower body exercises and whole body exercise. The upper body exercises are dumbbell press, overhead arms, back twist, chest, shoulder, and front arm raise. The lower bod exercises are chair stand, leg raise forward/ba- ckward, side leg raise, squats, and balance walk while whole body exercise is pushups. Our proposed application can accurately detection the human body joints and calculate the angle between the joint in order to accurately count the repetitions. This paper presents various contributions which are discussed below.
1. Android-based physical activity detection and monitoring using pose estimation can help individuals to improve their physical health by providing accurate and real-time feedback on their exercise form and intensity. 2. The use of pose estimation in physical activity detection and monito- ring allows for a more accurate tracking of individual movements and
exercises, helping users to optimize their workout routines and achieve their fitness goals. 3. The selected physical activities in this application can improve poten- tial health concerns, such as improving body strength, stability, and flexibility. 4. Using pose estimation in physical activity monitoring allows for the cre- ation of personalized recommendations for healthy habits, based on an individual’s unique physical activity patterns and records stored in app. 5. The data collected through these applications can be used in resea- rch settings to study human movement and physical activity patterns, inform the development of new exercise programs and interventions, and advance our understanding of the human body.
Proposed Methodology and System Design The proposed methodology for the android-based physical activity detection and monitoring system involves the integration of smartphone and compu- ter vision algorithms for accurate tracking and monitoring of individuals’ movements and exercises. The system uses pose estimation techniques to analyze different physical activities in real-time, providing valuable insights into the individuals’ physical activity patterns. The smartphone camera is used to capture and transmit movement data to the application where com- puter vision algorithms process the data, using machine learning and deep learning models, to track different body joints landmarks and calculate their angle in order to count the repetition of these physical activities, such as dumbbell press, front arm raise, squats etc. In addition to recognize body landmarks, the system will also provide real-time feedback on exercise form and intensity, including a repetition counting feature for different exercises. The collected data will be securely stored and can be used to track progress and inform treatment plans, optimize workout routines, and advance our understanding of human movement and physical activity patterns.
In this research our focus was to recognize 12 different kinds of pose during exercises performed by a user. The exercises are divided into three different categories i.e., upper body (dumbbell press, overhead arms, back twist, chest, shoulder, front arm), lower body (chair stand, leg raise forward/backward, side leg raise, squats, balance walk), and whole-body (pushups). User can also start some random exercise as an experiment or select the one category from the above mentioned one and start to perform exercise. The body points indexes are represented in Figure 1.
Media Pipe Media Pipe is an open-source framework developed by Google that makes it easy to build and deploy machine learning models on mobile and web appli- cations (Lugaresi et al., 2019). Media Pipe Pose is a specific model within the Media Pipe framework that is used to detect the pose of a person in an image or video. It uses a neural network to estimate the position of body landmarks,
Figure 1: Shows 33 keypoints of human body extracted through pose estimation.
such as the joints of a person’s limbs and torso, and then uses that informa- tion to infer the overall pose of the person. It can detect multiple people in an image or video and track their poses over time.
Pose Estimation To estimate the pose in the different exercises we need to calculate the angles between the different joints of human body. For every exercise the angle of body joints was different, so based on those angles, we judge the exercise position. For the calculation of angle, we used the formula mentioned below.
Equation reproduced from the original PDF.
if θ > 180 : θ = 360 −θ The expression above calculates the angle θ between two lines between joints. The lines are defined by two points each, α and β. The tan function calculates the tangent of the angle between the two lines. The atan function calculates the arctangent, which returns the angle in radians. The angle is then converted to degrees by multiplying it by 180 / pi. The abs function returns the absolute value of the angle, ensuring that it is always positive. Finally, the expression subtracts the arctangent of line α and β from the arctangent of line 0 and β, which gives the angle between the two lines.
Here is a step-by-step explanation of the expression:
atanθ(α[1] - β[1],α[0] - β[0]): calculates the arctangent of the slope of the direction vector α - β. This gives us the angle in radians of the line represented by the direction vector α - β.
atan θ 2(α[1] - β[1],0[0] - β[0]): calculates the arctangent of the slope of the direction vector 0 - β. This gives us the angle in radians of the line represented by the direction vector 0 - β.
tanθ / (α[1] - β[1],0[0] - β[0]) - atan θ 2(α[1] - β[1],α[0] - β[0]): subtracts the angles of the two lines represented by the direction vectors α - β and 0 - β to get the angle between the two lines.
abs(): calculates the absolute value of the angle between the two lines, as the angle between the lines is defined to be positive.
180 / π: converts the angle in radians to degrees, by dividing by π and multiplying by 180.
The result of the expression is the angle θ in degrees between the two lines represented by the direction vectors α and β.
Count Repetitions Along with the measurement of the angle, we also measure that how many times a particular exercise was performed. If a user selects squats and try to perform squat exercise the application will count the number of squats which were performed during that time. To measure the count accurately we added the counter while a specific angle is created during exercise. There are some rules defined for every activity calculation via pose estimation for which we take the example of Shoulders up and down exercise. The pseudocode of Shoulders up and down exercise is given below: Algorithm: Pseudo code for angle and count variable check
1. Check the value of the angle variable 2. if angle < 120 degrees then 3. Set the value of the stage variable to “Shoulders up” 4. Print the value of the stage variable 5. Call the setText function with the tv and stage variables as parameters 6. Set the value of the isDown variable to true 7. else 8. Set the value of the stage variable to “Shoulders down” 9. Print the value of the stage variable 10. if isDown == True then 11. Increase the value of the counter variable by 1 12. end if 13. Call the setText function with the tv and stage variables as parameters 14. Set the value of the isDown variable to false System Architecture
We have developed and android application where the first step is the process of user login functionality. The user is prompted to enter their login creden- tials, such as a username and password, to gain access to the application. The application then verifies the provided credentials against a database to determine whether the user is authorized to access the application. Once the
Figure 2: Shows the overall framework of our android application.
user is successfully logged in, the application stores their login information in the database for later use. This allows the application to track the user’s activity and provide personalized experiences. After the user is logged in, they are presented with three options for body exercise: Upper body, Lower body, and Whole body. The user can select one of these options to proceed with their workout. If the user selects the Whole-body option, they are presented with a list of exercises to choose from, such as pushups. If the user selects the Lower body option, they are presented with a list of exercises to cho- ose from, such as Leg raise, Side leg raise, squats, and Balance Walk. If the user selects the Upper body option, they are presented with a list of exercises to choose from, such as Dumble press, Overhead arms, Back Twist, Chest, Shoulders, and Front arm raise. The overall architecture of application is shown in Figure 2.
Exercise Logs User can also track the exercise logs and track his history that how many exercises and how many repetition he/she performed during the specific date. The user can go to Experimental logs section in application where the user can select the month and date in calendar and check the record of his activities. Figure 3 shows the screenshots of different Menu of application.
Application Performance Analysis The performance of this application can be analyzed based on several key metrics, accuracy, usability, reliability, and user satisfaction.
Accuracy refers to the ability of the system to correctly recognize the joint and count the physical activities and is typically measured as the percentage of correctly identified activities. High accuracy is important as it ensures that the system provides accurate information on physical activity levels and patterns and can be used to inform and optimize workout routines.
Figure 3: Shows different screenshots of applications menu.
Reliability refers to the consistency and stability of the system and is mea- sured as the percentage of correctly identified activities over time. High reliability is important as it ensures that the system provides consistent and accurate information, and that the data collected can be used to make meaningful comparisons and draw valid conclusions when used in future.
User satisfaction refers to the extent to which users find the system easy to use, comfortable to use and understand, and helpful in tracking and impro- ving their physical activity levels. High user satisfaction is important as it ensures that users are more likely to continue using the application, and that the data collected is of high quality.
In addition to these metrics, the performance of the system can also be analyzed based on factors such as battery life, ease of use, and data security. Battery life is important as it affects the user’s ability to use the technology for extended periods of time, while ease of use is important as it affects the user’s ability to effectively use the application and the quality of the data collected and stored.
The performance of the android-based physical activity monitoring system using pose estimation can be analyzed based on feedback from 15 different users which are shown in Table 1 which summarize that this application got positive feedback from all the users in term of accuracy, reliability, and user satisfaction.
Table 1. Shows the summary of user’s feedback on using this application.
Table available in the original PDF paper.
App Users Accurate Reliable User Satisfaction User 1 ■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
■Agree □Neutral □Disagree
User 2 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
User 3 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
User 4 □Agree ■Neutral □Disagree
■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
User 5 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
User 6 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
User 7 ■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
□Agree ■Neutral □Disagree
User 8 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
User 9 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
User 10 ■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
■Agree □Neutral □Disagree
User 11 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
User 12 □Agree ■Neutral □Disagree
□Agree ■Neutral □Disagree
■Agree □Neutral □Disagree
User 13 ■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
□Agree ■Neutral □Disagree
User 14 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
User 15 ■Agree □Neutral □Disagree
■Agree □Neutral □Disagree
□Agree ■Neutral □Disagree
DISCUSSION
The use of android-based physical activity detection and monitoring using pose estimation is a rapidly growing field, with numerous applications and benefits. The combination of smartphone technology and computer vision algorithms allows for accurate tracking and monitoring of individuals’ move- ments and exercises, providing valuable insights into their physical activity patterns. This information can be used to inform treatment plans, optimize workout routines, provide personalized recommendations for healthy habits, and advance our understanding of human movement and physical activity patterns.
One of the key benefits of android-based physical activity monitoring system is the ability to provide real-time feedback on exercise form and inten- sity. This feedback is particularly valuable for individuals who are trying to build strength and improve their performance, as it helps them to avoid common mistakes and ensure that they are performing exercises correctly.
Another important benefit of this technology is its potential to improve physical health. By continuously monitoring physical activity levels, these applications can detect potential health concerns, such as mobility problems and provide early interventions to address these issues. This is particularly important for individuals with chronic conditions or post-operative rehabili- tation, who require close monitoring and regular assessments of their physical activity levels.
In research, the data collected through these applications can be used to study human movement and physical activity patterns, inform the deve- lopment of new exercise programs and interventions, and advance our understanding of the human body. Additionally, the repetition counting fea- ture can be used to study the effects of different training programs on muscle strength and endurance.
Despite the numerous benefits and applications of android-based physical activity monitoring system, there are also some limitations and challenges that need to be addressed. One of the main challenges is ensuring the accu- racy and reliability of the tracking and monitoring technology, as errors and inaccuracies in the data can lead to incorrect conclusions and misinterpreta- tion of the results. Additionally, there is a need for standardized protocols and metrics for collecting and analyzing physical activity data, to ensure that the data is comparable across studies and can be used to make meaning- ful comparisons and draw valid conclusions. The potential benefits of this technology are significant and have the potential to greatly improve physical health, performance, and wellness in a variety of settings. Further resea- rch in this field is essential to fully realize the potential of this technology and to advance our understanding of human movement and physical activity patterns.
CONCLUSION
Android-based physical activity detection and repetition counting using pose estimation have the potential to greatly enhance our understanding of human movement and physical activity patterns. The applications and scope of this
technology are far-reaching and have the potential to improve physical health, performance, and wellness in a variety of settings. In healthcare, this applica- tion can be used to monitor and track physical activity levels in patients, such as those with chronic conditions or post-operative rehabilitation. The use of pose estimation allows for a more accurate tracking of individual movements, providing valuable insights into patients’ physical activity patterns and hel- ping to inform treatment plans. In sports, this application can be used by athletes to track their training progress and improve their form. The repeti- tion counting feature is particularly useful in strength training, where it helps individuals to accurately track their progress and avoid over-training. This information is particularly useful for users who are trying to build strength and improve their performance. The use of pose estimation allows for a more accurate tracking of individual movements, providing users with perso- nalized recommendations for healthy habits based on their unique physical activity patterns. This application can efficiently monitor 12 different types of physical activities. The activities performed by users are stored in the exe- rcise logs so that the users has the record of all exercises and repetition he has performed. This application is tested by 15 different users in terms of accuracy reliability and user satisfaction and the feedback from the all user was positive about this application. Overall, the applications and scope of android-based physical activity detection and repetition counting using pose estimation are wide-ranging and have the potential to greatly improve physical health, performance, and wellness in a variety of settings.
ACKNOWLEDGMENT
This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program(IITP-2022-RS-2022-00156354) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation) and by the Ministry of Trade, Industry and Energy (MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the International Cooperative R&D program (Project No. P0016038).
REFERENCES
Attal, F., Mohammed, S., Dedabrishvili, M., Chamroukhi, F., Oukhellou, L., & Amirat, Y. (2015). Physical human activity recognition using wearable sensors. Sensors, 15(12), 31314–31338. Ghazali, N., Shahar, N., Rahmad, N., Sufri, N. J., As’ ari, M., & Latif, H. M.
(2018). Common sport activity recognition using inertial sensor. 2018 IEEE 14th International Colloquium on Signal Processing & Its Applications (CSPA), Hassan, M. M., Uddin, M. Z., Mohamed, A., & Almogren, A. (2018). A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems, 81, 307–313. Hsu, Y.-L., Yang, S.-C., Chang, H.-C., & Lai, H.-C. (2018). Human daily and
sport activity recognition using a wearable inertial sensor network. Ieee Access, 6, 31715–31728. Kwapisz, J. R., Weiss, G. M., & Moore, S. A. (2011). Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter, 12(2), 74–82.
Lara, O. D., & Labrador, M. A. (2012). A survey on human activity recognition using wearable sensors. IEEE communications surveys & tutorials, 15(3), 1192–1209. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang,
F., Chang, C.-L., Yong, M. G., & Lee, J. (2019). Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172. Mekruksavanich, S., & Jitpattanakul, A. (2020). Smartwatch-based human activity recognition using hybrid lstm network. 2020 IEEE SENSORS. Nguyen, M., Fan, L., & Shahabi, C. (2015). Activity recognition using wrist-worn
sensors for human performance evaluation. 2015 IEEE International Conference on Data Mining Workshop (ICDMW). Subasi, A., Radhwan, M., Kurdi, R., & Khateeb, K. (2018). IoT based mobile healthcare system for human activity recognition. 2018 15th learning and technology conference (L&T). Tran, T. D., Truong, H., & Dang, T. K. (2016). Automatic fall detection using smart-
phone acceleration sensor. International Journal of Advanced Computer Science and Applications, 7(12). Weiss, G. M., Lockhart, J. W., Pulickal, T. T., McHugh, P. T., Ronan, I. H., & Timko, J. L. (2016). Actitracker: a smartphone-based activity recognition system for improving health and well-being. 2016 IEEE international conference on data science and advanced analytics (DSAA). Yang, J., Lu, H., Liu, Z., & Boda, P. P. (2010). Physical activity recognition with
mobile phones: challenges, methods, and applications. Multimedia Interaction and Intelligent User Interfaces: Principles, Methods and Applications, 185–213.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Aging and Special Needs
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-64-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional