Mathias Rieder, Norbert Bartneck (2022). Demonstrator for a Collaborative Human-Robot Picking System. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe100971
This paper introduces a demonstrator for a Collaborative Human-Robot Picking application consisting of a robot in front of a rack picking known objects into a bin. The major part of the system demonstrates the execution of picking orders carried out by human pickers and picking robots in a common workspace. A further part includes the training of Convolutional Neural Networks for object detection. A central concept of the system is an Emergency Call enabling the robot to request help if problems occur during object detection or grasping. The main goals of this demonstrator are to evaluate the interaction between human pickers and robots and to test the performance of object detection.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
INTRODUCTION
In several industrial applications processes have been automated during last decades by standardizing processes and reducing complexity e.g., automated transport was enabled by using transport equipment like pallets. Order picking lacks this potential of standardization, as it is the composition of items form the complete assortment according to customer orders (Association of Engineers, 1994). Therefore, a high amount of flexibility is required to handle different items, as everybody knows from shopping in supermarkets (Echelmeyer et al., 2008). Formulating rules for machines to handle this variety of items automatically was a challenge so that 80% of warehouses were run manually in 2015, 15% were operated mechanized and only 5% automated (Bonkenburg, 2016) and automating logistics processes is still a challenge (Zsifkovits et al., 2020). Nevertheless, there are trends for automation and advances in robotics and IT (Custodio & Machado, 2020). Only few approaches consider how to transform order picking from manual handling to partwise or complete automation without redesigning the whole logistics infrastructure. Rieder and Verbeet (2019) proposed a concept to introduce robots into a traditional rack picking environment and raise the percentage of robot picking by human-robot collaboration stepwise. To prove the approach’s suitability for operational use a demonstrator has been developed to display the human-robot collaboration as central element of the concept.
The remainder of this paper is organized as follows. First, the approach for a Collaborative Human-Robot Picking System is introduced briefly followed by a short overview of different demonstration setups in surroundings of modern logistics. Then the developed demonstrator is introduced. A conclusion and an outlook on next research steps close the paper.
RELATED WORK
This chapter introduces the central elements of the Collaborative Human-Robot Picking System realized as prototype by the demonstrator and gives a short overview of existing demonstrators in logistics surroundings.
COLLABORATIVE HUMAN-ROBOT PICKING SYSTEM The concept of a Collaborative Human-Robot Picking System enables the integration of robots into an existing picking system without investing much effort into warehouse infrastructure or process design (Rieder & Verbeet, 2019). In this concept, human pickers help to integrate robots starting at a rather low performance level to improve during operations especially in the process of detecting wanted items within the shelves.
To generate an appropriate starting performance for the robots, a Learning-Phase is applied gathering images in a laboratory environment - realized as Picture Recording Machine (PRM) - and training Convolutional Neural Networks (CNN) for object detection. These initial CNNs are used by robots in the Application-Phase to detect and pick items of assigned
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 picking orders from shelves. If detection or grasping fails, an Adjustment-Phase offers different solutions to achieve a successful pick by e.g., re-recording an image or changing the perspective to the shelf. Still failing leads to the Cooperation-Phase where a human picker is called for help (Emergency Call) to solve the robot’s problems (Rieder & Verbeet, 2020) and gather data for further process improvement. These data extend the set of training data from the Learning-Phase and are included into a re-training to improve CNN’s performance. The picking process consist of four phases shown in Figure 1. The interaction between the phases realizes the concept of a Feedback Loop (Rieder & Verbeet, 2019; Rieder & Verbeet, 2020).
Figure 1. Phases in the Collaborative Human-Robot Picking System (Verbeet et al., 2021) DEMONSTRATORS IN LOGISTICS SURROUNDINGS
Demonstrators typically are used to evaluate new technical solutions and concepts for industrial usage. A demonstrator, especially for software testing purposes, is developed in five phases: Definition, Design, Implementation, Test and Maintenance. Design and implementation phase have been run through recurringly (DeMarco, 1979). Maier et al. (2021) describe main challenges and the appropriate consequences during software demonstrator development.
Different popular technology fields exist with strong relation to robotic picking systems such as robotic picking (Roa-Garzón et al., 2019; Holm et al. 2021), device communication (Loriot et al., 2017, Scheible, 2021), cyber-physical systems (Vörös et al., 2018, Dobler et al., 2020) or the usage of camera data to optimize logistics processes (Lewin et al., 2017). Considering successful evolution of new technologies, human factors must be considered during design and development of production and logistics systems (Sgarbossa et al., 2020).
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
DEMONSTRATOR This chapter introduces the setup of the demonstrator. The first part considers the Training Environment, the second shows the setup of the prototypical Rack Picking and the third its software realization. All parts of the demonstrator have been developed according to the five phases describe by DeMarco (1979) with the current stage of iterative improvement while running through implementation and test. TRAINING ENVIRONMENT
One part of the demonstrator is a training environment consisting of a PRM and different computers enabling an efficient collection of image data to enable training of CNNs (Rieder & Verbeet, 2019). It has been developed and realized to support the startup of a collaborative picking system and realizes the introduced Training-Phase. The components and its communication interfaces are shown in Figure 2.
Figure 2. Training Environment, (modified from Rieder and Verbeet, 2020) An important extension to Rieder and Verbeet (2019) is the integration of a working station enabling training of CNNs much faster than using a standard desktop (approx. 2,000 iterations per hour instead of 70). Furthermore, the controller has been realized as multiagent- system described in detail by Verbeet et al. (2021). The PRM is independent of the Rack Picking. A connection of both systems can be achieved by updating the CNN trained by the working station and then be used for object detection during picking processes automatically. This is done manually at the current stage of research.
RACK PICKING The demonstrator for the picking process consists of a collaborative Universal Robot UR5e
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 mounted in front of a rack. Rack and robot’s table are built from aluminum profiles to enable changing the robot’s position as well as a reconfiguration of the rack and its shelves. The robot is equipped with a vacuum gripping system ECBPi by Schmalz and a two-finger- gripper 2F-140 by Robotique. For automated tool change, the tool-changing system WINGMAN by TripleA Robotics is integrated. In the rear of the robot, a mounting allows the installation of different camera systems for purposes of comparison, containing Microsoft Kinect One, Photoneo PhoXi 3D Scanner M and Roboception rc_visard 160 color. The camera mounting also serves as holder for a touchscreen enabling the technical interaction of robot and human operators. The setup is shown in Figure 3.
Figure 3. Rack Picking hardware setup
SOFTWARE REALIZATION As the feasibility of multiagent-systems in modern logistics applications is proven (Seitz et al., 2021; Fischer et al., 2020), the controller of the Rack Picking is realized as multiagent- system where each agent executes specific functions (Verbeet et al., 2021).
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
The Picking-Station-Agent is the central agent and coordinates the actions and communication of the multiagent-system. A special interface allows the manual creation of picking orders containing items from the database. This interface is shown in the left part of Figure 4. The picking orders are then distributed to the pickers in the system. If an Emergency Call is triggered, it is forwarded to a human picker. A further interface helps the human picker to understand the robot’s problem by displaying the image the robot has recorded from the shelf. This is shown as grey field on the right part of Figure 4. At the top of this field the name of the searched item is displayed, here exemplarily “Raspberry-Box”. Further- more, all data collected during Emergency Calls is saved to the database for further trainings. The Roboception-Agent realizes the interface to the Roboception sensor to trigger image recording and gripping point detection. The Photoneo-Agent realizes the interface to the Photoneo sensor to trigger image recording and gripping point detection. The UR5e-Agent realizes the interface to the robot controller initiating tool changes if necessary, picking and placing. The Prediction-Server-Agent uses an input image from Roboception- or Photoneo- Agent and a CNN to predict the Region of Interest (ROI) within the image where the searched object is supposed to be located. The ROI is then reported to the related agent to serve as basis for gripping point detection. All agents run on one single computer but could be distributed to different ones to integrate more entities of humans and robots. An agent operating Kinect One is realized at the PRM and can be integrated if necessary.
Figure 4. System interface for order creation (left) and Emergency Call handling (right)
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
CONCLUSIONS
The developed demonstrator described in this paper has been finalized recently and will enable exhaustive testing of the Collaborative Human-Robot Picking System. With a Picture Recording Machine, the efficient training of Convolutional Neural Networks is possible. The human-robot interface at the Rack Picking enables performance evaluation of object detection and grasping of a picking robot. The described demonstrator enables a series of experiments that will be carried out as future research. It is expected to estimate the probability of a certain setting of software, hardware and items to be handled in an automated or collaborative way.
ACKNOWLEDGMENTS
This work is part of the project “ZAFH Intralogistik”, funded by the European Regional Development Fund and the Ministry of Science, Research and Arts of Baden Württemberg, Germany (F.No. 32-7545.24-17/3/1). Many thanks go to Richard Verbeet for realization of the multiagent-system, inspiring collaboration and support.
REFERENCES
Association of Engineers (1994). “Picking systems – basics” (in German), standard no. 3590 Bonkenburg, T. (2016). Robotics in Logistics - A DPDHL perspective on implications and use cases for the logistics industry. Bonn: Deutsche Post DHL Group Custodio, L., & Machado, R. (2020). Flexible automated warehouse: a literature review and
an innovative framework. The International Journal Of Advanced Manufacturing Technology, 106, pp. 533-558 DeMarco, T.: (1979) Concise Notes on Software Engineering. New York: Yourdon Press Dobler, M., Schumacher, J., Büsel, P., & Hartmann, C. (2020). Supporting SMEs in the Lake Constance Region in the Implementation of Cyber-Physical-Systems: Framework and Demonstrator”. in: IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC). (pp. 1-8) Echelmeyer, W., Kirchheim, A., & Wellbrock, E. (2008) Robotics-logistics: Challenges for
automation of logistic processes, IEEE International Conference on Automation and Logistics. (pp. 2099-2103) Fischer, J., Lieberoth-Leden, C., Fottner, J., & Vogel-Heuser, B. (2020). Design, Application, and Evaluation of a Multiagent System in the Logistics Domain. IEEE Transactions on Automation Science and Engineering. 17, pp. 1283-1296
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Holm, M., Senington, R., Wang, W., & Lindblom, J. (2021). Real-World Industrial Demonstrators on Human-Robot Collaborative Assembly. In Wang, L., Wang, X.V., Váncza, J., & Kemény, Z. (Eds.). Advanced Human-Robot Collaboration in Manufacturing (pp. 413-438). Springer, Cham Lewin, M., Weber, H., & Fay, A. (2017). Optimization of Production-Oriented Logistics
Processes Through Camera-Based Identification and Localization for Cyber-Physical Systems. In: Lödding, H., Riedel, R., Thoben, KD., von Cieminski, G., Kiritsis, D. (Eds). Advances in Production Management Systems. The Path to Intelligent, Collaborative and Sustainable Manufacturing (pp. 168-176). Springer, Cham Loriot, M., Aljer A., & Shahrour, I. (2017). Analysis of the use of LoRaWan technology in a large-scale smart city demonstrator. Sensors Networks Smart and Emerging Technologies (SENSET) (pp. 1-4). Beirut, Lebanon Maier, J.T., Berger, S., Heuer, T., Nyhuis, P., & Schmidt, M. (2021) Decisions And
Characteristics During The Development Process Of A Software Demonstrator. In: Nyhuis, P.; Herberger, D.; Hübner, M. (Eds.) 2nd Conference on Production Systems and Logistics (CPSL 2021) (pp. 514-523). Hannover, Germany. Rieder, M., & Verbeet, R. (2019). Robot-human-learning for robotic picking processes, In: Kersten, W., Blecker, T., & Ringle C.M. (Eds.) Artificial Intelligence and Digital Transformation in Supply Chain Management (pp. 87-114). Hamburg, Germany Rieder, M., & Verbeet, R. (2020). Realization and validation of a collaborative automated
picking system In: Jahn, C., Kersten, W., & Ringle, C.M. (Eds.), Proceedings of the Hamburg International Conference of Logistics (HICL 2020) (pp. 521-558): Hamburg, Germany Roa-Garzón, M.A., Gambaro, E.F., Florek-Jasinska, M., Endres, F., Ruess, F., Schaller, R., Emmerich, E., Muenster, K., & Suppa, M. (2019). Vision-Based Solutions for Robotic Manipulation and Navigation Applied to Object Picking and Distribution. In: Künstliche Intelligenz, 33, 171-180 Scheible, J. (2021). LoRa and LoRaWAN - An evaluation of scalability and reliability.
http://dx.doi.org/10.18725/OPARU-38585 Seitz, M., Gehlhoff, F., Cruz Salazar, L.A., Fay. A., & Vogel-Heuser, B. (2021): Automation platform independent multi-agent system for robust networks of production resources in industry 4.0. Journal Of Intelligent Manufacturing, 32, 2023-2041 Sgarbossa, F., Grosse, E.H., Neumann, W.P., Battini, D., & Glock, C.H. (2020). Human
factors in production and logistics systems of the future. Annual Reviews In Control, 49. 295-305 Verbeet, R., Rieder, M., & Bartneck, N. (2021). Coping with Changing Articles in Automated Picking Systems by Adaptive Object Detection and Human-Robot Cooperation. https://ssrn.com/abstract=3821401 Vörös A., Bur, M., Rath, I., Horvath, A., Micskei, Z., Balgoh, L., Mazlo, Z., & Varro, D.
(2018) MoDeS3: Model-Based Demonstrator for Smart and Safe Cyber-Physical Systems. In: Dutle, A., Muñoz, C., & Narkawicz, A. (Eds.). NASA Formal Methods. (pp. 460-467) Zsifkovits H., Woschank M., Ramingwong S., & Wisittipanich W. (2020). State-of-the-Art Analysis of the Usage and Potential of Automation in Logistics, In: Matt D., Modrák V., & Zsifkovits H. (Eds.). Industry 4.0 for SMEs (pp. 193-212)
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional