Nicholas Schloer, Benedict Bauer, Carsten Wittenberg (2023). Mixed reality control of a mobile robot via ROS and digital twin. In: Human Factors in Robots, Drones and Unmanned Systems. AHFE Open Access, vol. 93, pp. 85–93. https://doi.org/10.54941/ahfe1003751
In recent years, mobile robotics has been increasingly used in agricultural production due to technological progress and increasingly powerful IT systems. Today, drones are already being used to spray agricultural areas with pesticides, for example. In the field of mobile robotics, control is currently based primarily on a controller or an app on a smartphone or tablet. If spatial target coordinates are to be specified, which the robot has to control, both systems quickly reach their limits. Using a mobile robot developed for the management of monocultures, this paper explains how spatial target coordinates can be approached with the help of a digital twin, ROS and a mixed reality interface using an MS HoloLens. The fusion of the technologies accordingly opens up new possibilities for a "human-robot interface".When developing an operating concept for the mobile robot, problems quickly arise regarding the determination of coordinates. The explanation for this is: The coordinate systems of the HoloLens and the mobile robot are not identical and, above all, are not known to the user. However, these obstacles can be eliminated by using a Digital Twin, which is displayed on the real robot using the HoloLens. By this measure, the coordinate system of the mobile robot is made known to the mixed reality device and it is possible to send the position coordinates as an offset to the current position of the robot. However, it should be noted that Unity (on the HoloLens) uses the left-hand rule, while ROS (on the robot) uses the right-hand rule. Therefore, the position data must be transformed accordingly.The same applies to the rotations, which must be converted as the equivalent of the coordinate transformation. Afterwards, the converted pose, which is now available in a "base_link" frame, can be transferred to the mobile robot via ROS.For the communication between the mixed reality device and the mobile robot, a ROS Bridge Client, which provides the messages in JSON format, is used. When sending target poses, it is important to note that the current time-stamped must be mandatory included in the message, otherwise the transformation from the "base_link" frame will not be converted to "map" coordinates. The transformation of the pose must also be done for the visual solution of the path planning. For the visualization, a distinction is made between the local and the global path planner A digital twin is superimposed on the real mobile robot for control and path planning. In the course of the project, problems with accuracy or performance have been discovered. In the intended application area - agriculture - the performance is weighted higher after an evaluation, so that slight position inaccuracies are accepted. The paper presents the concept as well as the interaction possibilities between the human and the mobile robot via the digital twin.
Small mobile robots are more and more used to assist humans or to per- form autonomous tasks in various domains. As a team of the Heilbronn University of Applied Sciences (HHN), we developed a small mobile robot for the use in These competitions focus on navigating in monocultures and performing specific agricultural tasks, such as detecting foreign objects like weeds in these rows of plants.. Here, the robot is typically guided autono- mously through the plant rows, while he is automatically detecting certain situations or irregularities in the plant row. Thus, emphasizes that the robot works autonomously.
However, small mobile robots can also be used in tasks that are only parti- ally autonomous or where the robots are fully controlled entirely by humans. For these purposes, we investigated whether a Mixed Reality could serve as an interaction platform to connect humane control with the mobile robot. In the user’s view, a digital twin is displayed additional to the reality, which is used for interaction such as path planning by means of gestures. The advantage of this solution is that the same interaction can also be used for interacti- ons via virtual reality, i.e. users in a distant location can control the mobile robot.
In recent years, mobile robotics has been increasingly used in agricultu- ral production due to technological progress and increasingly powerful IT systems for instance drones are already used to spray agricultural areas with pesticides. In the field of mobile robotics, control is currently based prima- rily on a controller or an app on a smartphone or tablet. If spatial target coordinates are to be specified, which the robot has to control, both systems
quickly reach their limits. Here, we show how spatial target coordinates can be achieved with the help of a digital twin, ROS and a mixed reality interface using an MS HoloLens. The fusion of the technologies accordingly opens up new possibilities for a “human-robot interface”.
THE MOBILE ROBOT Mobile robots are commonly used in various industrial areas including agri- culture. The mobile robot FloriBot 4.0 that was developed at HHN is such a mobile robot develop for the Field Robot Event (FRE) (Straten, 2004) and shown in Figure 1 in its natural habitat.
The FRE is an international competition that aims to promote research and development in the field of mobile robots for agriculture. The event is split into several tasks that are designed to test the capabilities of the parti- cipation robots in various aspects of precision agriculture. In the Navigation Task the robots have to navigate autonomously in the field environment of a monoculture crop such as maize without the use of a Global Navigation Satellite System. So the event tests the capabilities of the robots in tasks such as mapping, planting and weeding. FloriBot 4.0 was designed to excel in these tasks. For that reason, one of the key features of FloriBot 4.0 is its articula- ted steering with active wheel and passive joint configuration, which is called passive articulated steering. In common with the standard articulated stee- ring this type of steering allows the robot to manoeuvre in tight spaces and navigate around obstacles with great precision. Typical for this drive confi- guration is its small turning radius in relation to its wheelbase. Therefore, it is excellent for navigating through monocultures. The kinematics of passive articulated steering is shown in (Bauer, 2018).
Such a specialized robot needs a special interface between the human user and the mobile robot. For that reason, a mixed reality interface by embedding a digital twin was developed
Figure 1: FloriBot 4.0 at the FRE 2022. (Photo: Phillip Rösner.)
DIGITAL TWIN OF THE ROBOT The digital twin is based on the CAD data, which allows for a very accu- rate appearance of the digital twin. To create the digital twin, as shown
Mixed Reality Control of a Mobile Robot via ROS and Digital Twin 87
Figure 2: Digital twin of the FloriBot in unity.
in Figure 2, the data is imported into Unity, where all further paramete- risation and adjustments are made. Unity is a development environment that is capable of creating applications for various classes of devices using the C# programming language. Unity was originally developed as a game engine, but is now also increasingly being used in film production and in the industrial contexts.
When developing an operating concept for the mobile robot, problems qui- ckly arise regarding the determination of coordinates. This problem is caused by the fact that the position of the two different coordinate systems of the HoloLens and the mobile robot do not coincide, which is also unknown to the user. These obstacles can be overcome by using a digital twin, which is displa- yed on the real robot using the HoloLens. For this reason, FloriBot 4.0 has to be tracked. Tracking is an important aspect when using a mixed reality device to interact with a real machine such as mobile robot. So the mobile robot is tagged with a QR code whose position and orientation can be tracked by the Mixed Reality device. Through this tracking, the pose of the mobile robot’s coordinate system is communicated to the Mixed Reality device and converted.The converted pose, which is now available in a “base_link” frame, can then be transferred to the mobile robot via ROS (Liu et al. 2020). The coordinate systems of the two systems (HoloLens and FloriBot 4.0) are not identical. Therefore, the coordinates of the FloriBot 4.0 have to be published in its coordinate system to the HoloLens system. Only after this step can the coordinates be transformed into the other system. This results not only in a deviation from the respective origin, but also in a dif- ferent definition of the system. However, it should be noted that in Unity uses the left-hand rule, while ROS uses the right-hand rule (Xue Er Sha- maine et al. 2020). The different coordinate system definitions are shown in Figure 3.
The transformation of the pose is done with quaternions. This makes the transformation of the points efficient.
Figure 3: Different coordination system (left-hand and right-hand-rule).
MIXED REALITY PRESENTATION Mixed reality is an extension of augmented reality. In contrast to augmented reality, mixed reality not only display holographic objects, but also makes possible to interact with these objects (Milgram and Kishino, 1994). Various types of input are supported. The application for controlling the FloriBot 4.0 relies on gestures, eye tracking and voice commands. The user interface is essential for augmented reality applications, because the mixing of the real and digital worlds (mixed reality) must not result in a flood of information for the user and the sources of information must also be clearly distinguishable. To avoid an abundance of messages, it is advan- tageous to highlight the relevant information and minimise others. This can be done by varying the size and colour of the messages or their placement in the field of view. It is also helpful to be able to activate and deactivate information paths, as this ensures that the user is presented with a clear user interface (Ens et al. 2014). These settings can be made in Figure 4. For this reason, the settings for the mobile robot are also included as a tab in the set- tings menu. In addition to the speed settings and the current position, toggles for displaying and manually placing the digital twin can be set here. If the pose is changed manually, the new pose must be overwritten in the variable using the “Change orientation” button.
Figure 4: World settings menu.
Mixed Reality Control of a Mobile Robot via ROS and Digital Twin 89
Menu fields that are called up are similar to those described in (Bareis et al. 2022) and are first projected into the field of view. If these fields are to move continuously or remain in one place, this can be set using the button with the tacks as symbol (see Figure 5).
INTERACTION For the communication between the Mixed Reality device and the mobile robot, a ROS Bridge Client is used, which provides the messages in JSON for- mat. This is shown in Figure 6. The HoloLens 2 cannot establish a native ROS 1 connection. Therefore, a ROS bridge server is operated on the ROS master. A ROS client is also integrated into the Mixed Reality application. As this is a UWP application, ROS-Sharp is used to send ROS messages in JSON format (Koubâa, 2015).
Figure 5: Digital twin floriBot setting.Figure 6: Schematic communication structure.
In this application, the mixed reality control is used to set goals for the mobile robot for path planning. When sending target poses, it is important to note that the current timestamp must be included in the mes- sage, otherwise the transformation from the “base_link” frame will not be converted to “map” coordinates. Mixed reality control also allows the user to visualise the path to the target planned by ROS in the real environ- ment. The visualisation distinguishes between the local and the global path planner.
The white path in Figure 7 signals the result of the local path planning algorithm, while the blue path represents the global one. Unity’s line rende- rer function is used to implement the path planning display. The data of the planned path, which is determined by a ROS node, is available as a multidi- mensional array. However, in order for the line renderer to work with this data, and for new waypoints to be added at any time, this data is transfer- red into a dynamic list. To enable this kind of transfer of poses that match the robot’s coordinate system, the digital twin must be superimposed on the real robot.
Basically, no control elements are initially displayed after starting the appli- cation. These can only be displayed by using the correct voice commands or gestures. The main menu for controlling the mobile robot is displayed by looking at the palm of the hand (see Figure 8). The main functions of the application, such as defining or deleting target poses for the FloriBot 4.0, can be carried out in this menu. But also, the resetting of the position of the digital twin to the real mobile robot. The first step in defining a tar- get pose is to create a “Goal” object, drag it to the target coordinate and set the orientation. The target object (see Figure 9) has an arrow to clearly
Figure 7: Path planning visualisation.
Mixed Reality Control of a Mobile Robot via ROS and Digital Twin 91
Figure 8: Main menu.
Figure available in the original PDF paper.
Figure 9: Goal object.
define the rotation, which indicates what the orientation of the mobile robot should be.
To ensure the data integrity of the target poses, only one target object can be created. For the same reason, as mentioned above, the current timestamp is required, as otherwise there is no guarantee that the target pose can still be approached, or to which initial pose the transformation refers to. This is because the coordinates that are sent are always in the “Base_Link” coordi- nate system. The origin of this coordinate system is on the front axis of the mobile robot. It is therefore essential to know what the robot’s current pose
was when the target pose was sent. If the time stamp exceeds the tolerance threshold, the motion command is ignored. With regard to the target pose, it should be noted that the Z-component can be ignored because it is a 2D path planner. Therefore, this coordinate component is always defined as zero before the target pose is transmitted.
CONCLUSION AND OUTLOOK Controlling a mobile robot with Mixed Reality glasses in combination with ROS seems to offer some advantages. One of the advantages is that it allows the robot to work with its hands free to perform other tasks, such as inspe- cting the plants and their fruits. By looking in the direction of the FloriBot 4.0, the robot can also monitor when the actual view is obstructed, for example by maize plants, as it is possible not to allow let the digital twin to be covered by real objects.
Another advantage of using ROS instead of a proprietary protocol is that it is relatively easy to allow the application to control other robots, as long as they use the same ROS nodes. Further work is to extend the gesture control, as well as to make a communication possible on basis of ROS 2. However, it is still problematic that the digital twin gets larger and larger position errors over time. This deviation adds up from various causes. One is the inaccuracy of the HoloLens with dynamic objects, as well as the placement of the digital twin, whose deviation is small in the near field but increases with distance from the HoloLens. This can cause exorbitant problems because the accuracy of the holographic placement of objects decreases dramatically with distance from the anchor. In most cases, dynamic objects are assigned to the superordi- nate coordinate system, which is the main coordinate system of the HoloLens presented here. In this case, positional accuracy is accepted for the sake of performance. In this way, unwanted pose errors are preserved. This will be further optimized in future work. Therefore, it is considered to track the QR code not only once at the beginning, but continuously as long as it is in the field of view of the HoloLens, in order to increase the positional accuracy. On the other hand, the low update rate of the current pose is also a limitation that needs additional improvements. The latter should be solved by a ROS 2 implementation and the TF-Topic.
One feature that would optimise the Mixed Reality application would be the visualisation of the laser scan or even the map data from the robot’s slam node. This would allow the user to follow the mobile robot’s decision with greater evidence and would also potentiate the fusion between robot and human.
Presenting the cost map as a mini-map, in one of the two upper corners pro- vides further significant information to the user, especially when the mobile robot is out of sight. It should be noted that this does not make the interface confusing.
With a fully kinematised model and object recognition using the HoloLens, more information from the digital twin should also be used in the future. For example, the HoloLens’ object recognition (obstacle detection) cloud be used
Mixed Reality Control of a Mobile Robot via ROS and Digital Twin 93
to incorporate data from the AR glasses into the path planning. Monitoring engine torque via the digital twin and other data could also be of interest.
REFERENCES
B. Bauer, Diesse, M., Heverhagen, T. & Wittenberg, C. (2018). Ein alternatives Antriebskonzept für einen mobilen Roboter – Der Knicklenker ohne aktives Stellglied. Tagungsband AALE 2018. VDE Verlag GmbH. Berlin. 127–133. Bareis C., Uhl F., Zeyer M., Bauer B., Wittenberg C. (2022) Human-Machine Intera-
ction: Controlling of a Factory with an Augmented Reality Device. In: Ahram T., Taiar R. (eds) Human Interaction, Emerging Technologies and Future Systems V. in: IHIET 2021. Lecture Notes in Networks and Systems, vol. 319. Springer. Barrett Ens, Juan David Hincapié-Ramos, and Pourang Irani (2014). Ethereal planes: a design framework for 2D information space in 3D mixed reality environments. in: Proceedings of the 2nd ACM symposium on Spatial user interaction (SUI ‘14). Association for Computing Machinery, pp. 2–12. C. Xue Er Shamaine, Y. Qiao, J. Henry, K. McNevin and N. Murray (2020).
“RoSTAR: ROS-based Telerobotic Control via Augmented Reality,”, in: IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP). pp. 1–6. Koubâa, A. (2015). ROS As a Service: Web Services for Robot Operating System. P. Milgram, F. Kishino (1994). A Taxonomy of Mixed Reality Visual Display Paul Milgram und Fumio Kishino. In IEICE Transactions on Information Systems, Vol E77–D, No. 12. van Straten, G. (2004). Field robot event, wageningen, 5–6 June 2003. Computers
and electronics in agriculture, 42(1), 51–58. Y. Liu. G. Novotny, N. Smirnov, W. Morales-Alvarez and C. Olaverri-Monreal (2020). “Mobile Delivery Robots: Mixed Reality-Based Simulation Relying on ROS and Unity 3D,” in IEEE Intelligent Vehicles Symposium (IV). pp. 15–20.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Robots, Drones and Unmanned Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-69-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional