Shifan Li, Seemal Asif (2024). How Robot Arms can be used for Bin Picking and Shelf Picking. In: Human Factors in Robots, Drones and Unmanned Systems. AHFE Open Access, vol. 138, pp. 22–36. https://doi.org/10.54941/ahfe1005005
Recently, there has been a substantial amount of concern regarding the utilization of robot arms for bin picking and shelf picking. The escalation in the volumes of e-commerce, online grocery, and automated warehousing has brought increased attention to this issue. In this paper, typical bin picking and shelf picking systems are reviewed, their limitations analyzed, and a novel solution inspired by the grasp detection algorithm is presented.
Bin picking and shelf picking are two common problems that require automa- tion for decades. It is a popular systematic problem, most of the research is focusing on algorithm to recognize and localize the objects. Algorithm is important for bin picking and shelf picking. However, in order to com- plete solve the problem in industry, it is necessary to review the whole system including the hardware set-up, then combine with the bin picking algorithm.
The predominant focus in extant literature resides in the evaluation and analysis of bin picking methodologies, as opposed to shelf picking. Fur- thermore, it is noteworthy that a substantial portion of reviewers exhibit discernible predilections towards algorithmic facets within the scope of the presented problem. In (Buchholz, 2016), the historical context of the bin picking problem was meticulously examined, with a detailed exposition on the contemporary bin picking approach specifically from the vantage point of grasp point detection algorithms. (Alonso, Izaguirre and Graña, 2019) delved into a comprehensive review of the bin picking problem, empha- sizing the perceptual aspect, albeit still within the purview of algorithmic perspectives. (Cordeiro et al., 2022), in a similar vein, scrutinized the bin picking problem, this time concentrating on the domain of deep learning algorithms. Notably, the research categorized extant bin picking systems based on the specific type of deep learning techniques employed in their implementations.
This paper will undertake a comprehensive review of the robotic system designed for both bin picking and shelf picking. It is emphasized that the scope of review extends beyond the singular examination of the bin picking
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 23
problem. Inclusive within this analysis is the shelf picking problem, perceived as akin to the bin picking problem, and both are subject to joint scrutiny. The investigation is approached systematically, positing that a holistic under- standing of the challenges posed by bin picking and shelf picking necessitates a concurrent examination of both hardware and software components. Furthermore, the analysis extends to encompass current industry cases ready for deployment, providing a practical context that is more closely aligned with real-world applications when compared to cases within the research community.
The paper will be structured in the following sequence. In Sec- tion II, an examination of the systems implemented in the contem- porary industrial landscape will be undertaken. Section III will then focus on reviewing systems within the research community, recogniz- ing that systems in industry inherently exhibit a lower degree of intel- ligence than their counterparts in the research domain. Section IV engages in a comprehensive analysis of the topological aspects of cur- rent systems employed in both industry and the research community. Through this exploration, the paper culminates in a discussion of the identi- fied limitations inherent in the present systems, positing them as the primary impediments to the widespread deployment of bin picking and shelf picking systems.
In Section V, attention shifts towards the elucidation of the prevailing grasp detection algorithms employed in robotic picking. By amalgamating the limitations delineated in Section IV with insights derived from grasp detec- tion algorithms, the paper introduces an innovative proposition aimed at addressing the challenges posed by bin picking and shelf picking systems. Finally, Section VI encapsulates the paper with a conclusive summary and outlines potential avenues for future research and development.
HOW ROBOT ARMS ARE BEING USED FOR BIN PICKING AND SHELF PICKING IN INDUSTRY Bin picking and shelf picking are required extensively in industry. For example, in logistics, baskets and boxes are two fundamental elements as container for transportation. But in warehouses and supermarkets, shelf is convenient for displaying the item. The following section will illustrate how robot arms are using in industry for basket picking and shelf picking.
Bin Picking In the configuration illustrated in Figure 1 from Ocado, an online grocery company, a robotic arm is affixed to the top of a bin grid tailored for the storage of grocery items (Introducing On-Grid Robotic Pick - YouTube, 2022). The robotic arm is equipped with an extension rod culminating in a suction cup, enabling the efficient transfer of goods between basket grids. This deployment not only results in cost savings in labour but also serves to augment operational efficiency within the online grocery industry.
In the succeeding configuration in Figure 2, the robotic arm is positioned atop a cylindrical base, executing the task of retrieving e-commerce items from a bin and transferring them onto a conveyor belt (Bin-picking and packaging - Powered by robots - Enabling e-commerce growth - YouTube, 2017). The bin is situated on a table with a surface elevation lower than the base of the robot arm. This ‘underneath bin’ configuration allows the robot arm to seamlessly execute the bin-picking motion. The grippers affixed at the extremity of the robotic arm feature three fingers and an accompanying suction cup. Notably, the suction cup necessitates a cable connection to an external vacuum source.
Figure 2: Bin picking system in e-commerce. (Bin-picking and packaging - powered by robots - enabling e-commerce growth - YouTube, 2017).
In the third configuration depicted in Figure 3, the robotic arm under- takes the task of selecting a screw from a bin, manipulating it to achieve a desired angle, and subsequently vertically inserting the screw into a base (Bin picking, kitting and sorting of bolts with an Epson industrial robot—multiple object finishes - YouTube, 2023). The grip- per is tethered to an external power source via a cable routed through the body of the robot arm. Noteworthy is the robotic arm’s adept execution of the assigned task, facilitated by a straightforward setup configuration.
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 25
Figure 3: Screw bin picking system (bin picking, kitting and sorting of bolts with an Epson industrial robot—multiple object finishes - YouTube, 2023).
Shelf Picking For the shelf picking the initial configuration presented in Figure 4, the robotic arm is situated upon a mobile platform (Robotic Materials Inc.’ GPR-1 in a shelf restocking task - YouTube, 2019). The robotic arm adeptly retrieves a bottle from a shelf, employs gravity to invert the bottle’s orienta- tion, and subsequently places it into a bin positioned on the mobile platform. Conversely, the robotic arm is capable of retrieving a bottle from a bin, utiliz- ing gravity to invert its orientation, and then situating it onto the shelf. This task is executed with finesse, particularly within the confines of a shallow shelf area.
Figure 4: Shelf picking with two-finger grippers (Robotic Materials Inc.’ GPR-1 in a shelf restocking task - YouTube, 2019).
In the subsequent scenario depicted in Figure 5, distinct from the initial scene involving shelf picking, the robot arm is equipped with an extended suction cup at its extremity (Robotic Shelf Picking - IAM Robotics Auto- mated Storage & Retrieval System (AS/RS) - YouTube, 2015). This iteration of the robotic arm employs suction to extract an item from the shelf, exe- cutes a joint twist, and subsequently deposits the item into a bin. Notably, in contrast to the first scene, this pick-and-place procedure involves no flipping motion, thereby mitigating the risk of inadvertently dropping the item.
26 Li and Asif
Figure 5: Shelf picking with vacuum suction cup (Robotic Shelf Picking - IAM Robotics Automated Storage & Retrieval System (AS/RS) - YouTube, 2015).
HOW ROBOT ARMS ARE BEING USED FOR BIN PICKING AND SHELF PICKING IN RESEARCH COMMUNITY Amazon Bin Picking Challenge
In the realm of robotic bin picking within the research community, it is note- worthy to highlight the prominent challenge hosted by Amazon, known as the Amazon Bin Picking Challenge. This challenge encompasses tasks related to both bin picking and shelf picking.
The outcomes of this challenge, specifically the noteworthy systems employed, have been extensively detailed in (Fujita et al., 2020). The litera- ture underscores the prevalence of grippers that integrate a suction cup with a two-finger structure, a configuration that has gained popularity within the context of the challenge. The evaluation of system performance in the chal- lenge predominantly relies on two primary metrics: the successful picking rate and Mean Picks Per Hour (MPPH). In addition to these mainstream metrics, the author introduces supplementary measures for performance assessment, namely Mean Time Between Failure (MTBF) and Mean Time to Repair (MTTR). The paper comprehensively presents the methodologies and outcomes of four participating teams in the challenge.
Building upon the discourse of robotic bin picking and the Amazon Bin Picking Challenge discussed in the previous paragraph, it is pertinent to high- light the noteworthy approach adopted by Team MIT-Princeton, as depicted in Figure 6. Their system features a distinctive gripper design incorporating a two-finger structure in conjunction with suction cups. Notably, the system adheres to a fundamental principle known as “grasp-first-then-recognize.” This principle dictates that the system initially detects the grasping pose using one neural network and subsequently recognizes it with another network after the item has been picked up.
Figure 6: Team MIT-Princeton bin picking system (Fujita et al., 2020).
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 27
Team Nanyang employed a dual-arm system featuring suction cups for the Amazon Bin Picking Challenge as illustrated in Figure 7. The operational space was partitioned into two distinct sections, each corresponding to one of the bins (Mahler et al., 2019). The primary grasping strategy employed by Team Nanyang involved approaching the object in a direct vertical descent from the top.
Figure 7: Team Nanyang bin picking system (a) is the physical cell layout (b) system architecture (Fujita et al., 2020).
Team MC2 implemented a dual-robot-arm system, augmented by two lin- ear sliders for the purpose of picking and placing items into the bin (Mahler et al., 2019). Their approach as illustrated in Figure 8 involved simultane- ous recognition of object category and grasping points, facilitated by RGB-D sensors. To accommodate diverse object shapes and sizes, the team employed three distinct types of grippers. Furthermore, the integration of a tool changer within the system allows for seamless transitions between different gripper types, enhancing adaptability and versatility in the bin picking task.
Figure 8: Team MC2 bin picking system (Fujita et al., 2020).
Illustrated in Figure 9, Team NAIST-Panasonic employed a 7-degree-of- freedom (7 DOF) robot arm paired with a custom-designed gripper for their participation in the Amazon Bin Picking Challenge. Enhancing their
28 Li and Asif
perception capabilities, the team integrated five RGB-D sensors to collect comprehensive information from the scene. Notably, their system incorpo- rates a shelf featuring weight sensors, a key element contributing to the object recognition process within their framework.
Figure 9: Team NAIST-panasonic bin picking system (Fujita et al., 2020).
Other Bin Picking and Shelf Picking Research In the expansive landscape of research dedicated to bin picking and shelf picking, it is observed that a substantial portion of investigations has tended towards non-systematic approaches, thereby presenting a notable divergence from the practicalities associated with real-world deployment. The ensuing section of this discourse will conscientiously examine research endeavours characterized by a systematic methodology. This analytical lens seeks to con- tribute to the scholarly discourse by elucidating the strides made in research endeavours that are more closely aligned with the imperatives of real-world applications.
An innovative strategy-based system, equipped with a custom gripper designed for pick-and-place operations from a shelf, is expounded upon in (Zhu et al., 2016). This has been illustrated in Figure 10. The detailed config- uration of their gripper, as depicted in Figure 11, reveals a design comprising two fingers for grasping, linked to a linear slider controlled by an actuator. Positioned at the base of these fingers is an additional actuator governing the open and close functionality. At the upper extremity of the fingers, the gripper integrates two suction cups oriented in different directions—one fac- ing downward and the other forward. Both suction cups are manipulated by a four-bar linkage connected to yet another actuator. This specialized grip- per, aptly termed “The Mantis Gripper,” is ingeniously orchestrated with three actuators. The intricacies of this gripper design lay the foundation for a strategic system, enabling the robot to execute diverse item-picking behaviours.
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 29
Figure 10: Strategy based shelf picking system (Zhu et al., 2016).Figure 11: The Mantis Gripper in (Zhu et al., 2016).
Dex-Net 4.0 stands as a prominent grasping algorithm widely recognized within the research community, particularly in the context of bin picking (Mahler et al., 2019). The physical setup has been illustrated in Figure 12. The algorithm’s efficacy is showcased through its implementation on a dual- arm humanoid robot, a configuration employed to achieve what the authors term “Universal Picking (UP)” in their thesis. The robot is outfitted with a suction cup gripper on one arm and a parallel jaw gripper on the other, with a depth camera positioned at the robot’s head, directed towards the assortment of objects within the bin.
30 Li and Asif
Figure 12: Dex-Net 4.0 dual arm bin picking system (Mahler et al., 2019).
The algorithmic pipeline employed by Dex-Net 4.0 follows a two-fold process: sampling and evaluation. Initially, the algorithm samples grasp can- didates for both the suction cup and parallel jaw grippers from the depth image. Subsequently, a neural network named GQ-CNN is utilized for the selection of the sampled grasp candidates. According to their specified pipeline and physical setup, the authors assert the attainment of bin picking capabilities for up to 25 novel objects. Remarkably, the reported reliability exceeds 95 percent, achieved at a rate surpassing 300 mean picks per hour, equivalent to an impressive 12 seconds per item. Such outcomes underscore the notable success of the Dex-Net 4.0 algorithm in the realm of bin picking.
THE LIMITATION OF CURRENT SYSTEM While numerous bin picking and shelf picking solutions exhibit proficiency within controlled laboratory environments, the seamless translation of these capabilities to real-world industrial applications remains a formidable chal- lenge. Consequently, many practical scenarios continue to rely on fixture configurations to facilitate the picking procedure. For instance, elevating the base of the robot arm relative to the bin or table is a common practice, particularly effective for shallow bins. However, this approach encounters limitations when confronted with deeper bins, necessitating the adoption of intricate configurations such as the on-grid system employed by Ocado as illustrated in Figure 1 to ensure smooth picking operations.
Similar challenges persist in shelf picking scenarios, where the robot arm or gripper is prone to collisions with the edges of the shelf. Despite efforts by researchers to devise specialized grippers for efficient item retrieval in con- fined spaces, these solutions are often tailored to specific situations. In the face of highly complex real-world conditions, the robustness of the system is invariably tested.
A comprehensive review of applications in both industry and the research community reveals a prevalent trend in bin picking and shelf picking methodologies—most systems employ a straight-approaching manner. This approach, characterized by a top-down method for bin picking and a left- right method for shelf picking, is conducive to collision avoidance and
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 31
is facilitated by straightforward path planning strategies. This orientation aligns with the industry’s demand for robust system performance in diverse and challenging operational environments.
Nevertheless, when dealing with bins or shelves of greater depth, or when items are positioned within the deeper recesses of shelves, the inclusion of a small extension rod becomes imperative. This extension serves to create sep- aration between the bulkier body of the robot arm and the picking tip. While this solution effectively reaches objects in deeper locations, it introduces a notable limitation: the naturally occurring distance between the robot arm body and the objects in the bins or on the shelves necessitates additional investments in configuring the factory or warehouse to accommodate this distance, ensuring the proper functionality of the Pick-and-Place procedure.
As illustrated in Figure 13, a robot arm without an extension rod adeptly retrieves items from shallow bins but faces limitations when tasked with items in deeper bins. A parallel scenario unfolds in shelf picking, where a robot arm without an extension rod is efficient in accessing items on shallow shelves but encounters challenges with those in deeper locations, as demonstrated in Figure 14. Deploying a robot arm equipped with an extension rod presents an added complication, as the risk of collisions becomes more pronounced.
Figure 13: Illustration of the limitation from the robot arm system without extension rod.Figure 14: Illustration of the limitation from the robot arm system with extension rod.
32 Li and Asif
In response to this predicament, certain systems, exemplified in Figure 14, incorporate a bin grid beneath the robot arm, ensuring that the extension rod does not collide with the environment during operation. Alternatively, as illustrated in Figure 14, some configurations position the robot arm atop a raised cylindrical base, mitigating collision risks and safeguarding the proper execution of the picking procedure.
Nonetheless, we posit that a more streamlined solution can be devised, one that avoids extensive hardware setups such as the ‘bin grid underneath’ or ‘cylinder base underneath.’ In the ensuing section, we aim to present a novel approach to address this challenge in a more refined and efficient manner.
NOVEL IDEA INPIRED BY GRASPING ALGORITHM The grasping detection algorithm stands as a pivotal component in the intel- ligent robotic pick-and-place procedure. A review of 3D grasp detection (Sundermeyer et al., 2021), (Fang et al., 2019), (Li et al., 2021) reveals that these algorithms typically generate grasp candidates, representing the loca- tion where the gripper needs to reach relative to the target object. This is illustrated in Figure 14. These grasp candidates consist of two fundamental elements: grasping points and approaching direction.
Upon reviewing the industrial cases presented in Sections II and III, it is evident that prevailing approaches involve employing either a top-down or left-right manner, often facilitated by extension rods or specialized grippers with linear sliders to reach the target object. Synthesizing insights from grasp- ing algorithms and these industrial use cases, it becomes apparent that a straight, controlled approach is crucial for most robotic Pick-and-Place sce- narios. This approach is imperative to prevent collisions between the robot and its surroundings, ensuring seamless operation in cluttered environments within confined spaces.
In light of this observation, we propose a novel systematic solution for bin picking and shelf picking tasks. We advocate for the integration of telescopic joints within robot arms, where these joints can extend to several times their original length. The utilization of telescopic joints serves to address the chal- lenge of straight approaching without necessitating additional length. Unlike linear sliders, telescopic joints offer a more efficient solution by mitigating interference with the environment, thereby enhancing the adaptability and versatility of robotic systems in bin and shelf picking applications.
Figure 15: Grasp candidate for grasp detection algorithm (Fang et al., 2019).
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 33
The fundamental functionality of a telescopic joint is elucidated in Figure 15. This mechanical configuration comprises a rod at the top, and when subjected to rotational motion, the cylinder component of the joint extends beyond its original length. This telescopic extension mechanism proves highly conducive for applications in bin picking and shelf picking scenarios.
In practical terms, as the upper rod undergoes rotation, the telescopic joint in Figure 16 allows for the elongation of the cylindrical section, offering a considerable reach advantage. This particular attribute is particularly advan- tageous in situations where extended reach is essential, as is often the case in bin and shelf picking tasks. The telescopic joint, by design, provides a versatile solution to the challenge of accessing objects located within vary- ing depths and heights, rendering it a valuable addition to the repertoire of robotic pick-and-place systems.
Figure 16: Telescopic joint model.
To simply implement the telescopic joint on a robot arm, a rough inverse kinematic strategy is presented in Figure 17. To fully utilize the collision avoidance characteristic of telescopic robot arm, it is recommended to pro- vide two points without orientation, which are pivot point and target point. And the pivot point resides in the plane that is vertical to horizontal plan and cross through the line between base point and target point as shown in Figure 18. When the inverse kinematics is implemented in the robot arm with telescopic joint. The θ4 and p5 are determined by the provided pivot point and target point, where θ4 is the angle of the fourth joint and p5 is the telescopic joint extension. And the first three joints’ angles, which are θ1, θ2 and θ3, are determined by triangular method.
Figure 17: Simple illustration model for robot arm with telescopic joint.
34 Li and Asif
Figure 18: Pivot point selection plane for the inverse kinematic of the robot arm with telescopic joint.
CONCLUSION AND FUTURE WORK The paper comprehensively evaluated bin picking and shelf picking systems across both industry and the research community. While industry-based sys- tems typically exhibit enhanced robustness, those presented in the research community stand out for their ingenuity in gripper and software design. Notwithstanding these advancements, both sectors face a shared limitation in their native configurations—difficulty in picking items from deeper sections of bins and shelves. Furthermore, systems with extensions tend to encounter heightened collision risks in real-world scenarios.
Addressing this challenge, certain industrial systems allocate significant financial resources to mitigate collisions, ensuring the safe operation of robotic arm systems. However, this paper proposes an alternative solution— integrating a telescopic joint into the robotic arm. This novel approach is posited as a potentially more elegant resolution to the trade-off between depth reach and collision avoidance. From our knowledge, we are the first one to introduce telescopic joint at the end of the robot arm. Unlike the existing telescopic base in robot arm market in Figure 19 (OnRobot, 2022), telescopic joint at the end of the robot arm can enable simplest deployment for pick-and-place task in robot arm system.
Moving forward, our future endeavors will involve the implementation and evaluation of the proposed telescopic joint concept in real world. Rig- orous testing and benchmarking against established metrics for bin picking and shelf picking systems will be conducted to ascertain the efficacy and practicality of the introduced solution. Through these efforts, we aim to con- tribute to the advancement of robust and versatile robotic systems tailored for real-world deployment.
How Robot Arms Are Being Used for Bin Picking and Shelf Picking 35
Figure 19: Existing telescopic base for robot arm in the market (OnRobot, 2022).
ACKNOWLEDGMENT
We express our gratitude to Daniel Oakley, Lab Technician, and James Fowler, Senior Technical Officer at the Intelligent Automation Lab, for their invaluable support and technical expertise, which were essential for the project’s success.
REFERENCES
Alonso, M., Izaguirre, A. and Graña, M. (2019) ‘Current Research Trends in Robot Grasping and Bin Picking’, Advances in Intelligent Systems and Computing, 771, pp. 367–376. doi: 10.1007/978-3-319-94120-2_35. Bin picking, kitting and sorting of bolts with an Epson industrial robot—multiple object finishes - YouTube (2023). Available at: https://www.youtube.com/watch?v=xCw64NYGxWM (Accessed: 26 November 2023). Bin-picking and packaging - Powered by robots - Enabling e-commerce growth - YouTube (2017). Available at: https://www.youtube.com/watch?v=Jkos2DVSn8g&t=28s (Accessed: 26 November 2023). Buchholz, D. (2016) ‘Bin-Picking — 5 Decades of Research’. doi: 10.1007/978-3-
319-26500-1. Cordeiro, A. et al. (2022) ‘Bin Picking Approaches Based on Deep Learn- ing Techniques: A State-of-the-Art Survey’, 2022 IEEE International Confer- ence on Autonomous Robot Systems and Competitions, ICARSC 2022. IEEE, pp. 110–117. doi: 10.1109/ICARSC55462.2022.9784795. Fang, H.-S. et al. (2019) ‘GraspNet: A Large-Scale Clustered and Densely Annotated Dataset for Object Grasping’. Available at: https://arxiv.org/abs/1912.13470. Fujita, M. et al. (2020) ‘What are the important technologies for bin pick-
ing? Technology analysis of robots in competitions based on a set of perfor- mance metrics’, Advanced Robotics. Taylor & Francis, 34(7–8), pp. 560–574. doi: 10.1080/01691864.2019.1698463.
36 Li and Asif
Introducing On-Grid Robotic Pick - YouTube (2022). Available at: https://www.youtube.com/watch?v=UjczYsjO6fQ&t=13s (Accessed: 26 November 2023). Li, Yiming et al. (2021) ‘Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation’, IEEE International Conference on Intelligent Robots and Systems. IEEE, (3), pp. 3571–3578. doi: 10.1109/IROS51168.2021.9636012. Mahler, J. et al. (2019) ‘Learning ambidextrous robot grasping policies’, Science
Available at: https://www.youtube.com/watch?v=CTKSGX2lVpU (Accessed: 15 April 2024). Robotic Materials Inc.’ GPR-1 in a shelf restocking task - YouTube (2019). Available at: https://www.youtube.com/watch?v=BH9IGpZ9tLg (Accessed: 26 November 2023). Robotic Shelf Picking - IAM Robotics Automated Storage & Retrieval System (AS/RS) - YouTube (2015). Available at: https://www.youtube.com/watch?v=h9eRDyZJzSA (Accessed: 26 November 2023). Sundermeyer, M. et al. (2021) ‘Contact-GraspNet: Efficient 6-DoF Grasp Gen-
eration in Cluttered Scenes’, Proceedings - IEEE International Conference on Robotics and Automation. IEEE, 2021-May (Icra), pp. 3133–3139. doi: 10.1109/ICRA48506.2021.9561877. Zhu, H. et al. (2016) ‘Strategy-based robotic item picking from shelves’, IEEE International Conference on Intelligent Robots and Systems. IEEE, 2016-Novem, pp. 2263–2270. doi: 10.1109/IROS.2016.7759354.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Robots, Drones and Unmanned Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-14-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional