Abstract

۱ Introduction

۲ Theoretical Background

۳ Related Works

۴ Methodology

۵ Experiments and Results

۶ Conclusions

Declarations

References

Abstract

     RoboCup 3D Soccer Simulation is a robot soccer competition based on a high-fidelity simulator with autonomous humanoid agents, making it an interesting testbed for robotics and artificial intelligence. Due to the recent success of Deep Reinforcement Learning (DRL) in continuous control tasks, many teams have been using this technique to develop motions in Soccer 3D. This article focuses on learning humanoid robot behaviors: completing a racing track as fast as possible and dribbling against a single opponent. Our approach uses a hierarchical controller where a model-free policy learns to interact model-based walking algorithm. Then, we use DRL algorithms for an agent to learn how to perform these behaviors. Finally, the learned dribble policy was evaluated in the Soccer 3D environment. Simulated experiments show that the DRL agent wins against the hand-coded behavior used by the ITAndroids robotics team in 68.2% of dribble attempts.

Introduction

     RoboCup is an international academic competition created to foster robotics and artificial intelligence research [27]. It has an ambitious long-term goal of having a team of humanoid robots beating the human soccer World Cup champions by 2050. There are many leagues with different game rules and constraints on robot designs to accelerate progress towards this objective.

RoboCup 3D Soccer Simulation (Soccer 3D) is a league of RoboCup based on a robot soccer simulator with a high-fidelity simulation model of the Nao humanoid robot. The particular contributions to RoboCup reside in being a research environment for high-level multi-agent cooperative decision-making, and humanoid robot control [44]. A simulation environment is convenient for machine learning algorithms due to their need for large amounts of data [36]. Dealing with real robots is time-consuming due to the need to recharge batteries or reallocate robots manually to set up experiments. Moreover, experience collection may be largely accelerated by running many simulations in parallel and executing in faster than real-time. Unfortunately, transferring behaviors learned in simulation to real robots is challenging due to the so-called reality gap. Still, some works have succeeded in doing so, usually by executing a final fine-tuning process on the real robot [36].

Conclusions

     Our main objective was to learn high-level soccer behaviors using reinforcement learning in this work. We addressed the problem with state-of-the-art model-free deep reinforcement learning algorithms, namely DDPG, TRPO, and PPO. Therefore, we learned behaviors while dealing with the complex dynamics of a humanoid robot.

     To facilitate, we used a hierarchical approach where the agent learns to command a model-based walking engine based on the Zero Moment Point (ZMP) concept. The walking engine receives the desired velocities in forward, lateral and rotational directions and outputs the joint angles.

     We developed a DRL framework for integrating DRL algorithms with the RoboCup 3D Soccer Simulation environment to accomplish our objective. In our results, PPO achieved the best performance, which was expected, and effectively learned humanoid robot behaviors.