This paper proposes a novel teleoperation method that allows users to guide robots hand by hand along with speech. In this method, the virtual robot modeled according to the remote real robot is projected into the real local environment to form a 3D operation interface. In this case, users can directly interact with virtual objects by their hands. Furthermore, since the Leap Motion is attached to the augmented reality (AR) glasses, the operation space is greatly extended. Therefore, users can observe the virtual robot from an arbitrary angle without blind angle in such a mobile pattern, which enhances the users’ interactive immersion and provides more natural human-machine interaction. To improve the accuracy of the measurement, an unscented Kalman filter (UKF) and an improved particle filter (IPF) are used to estimate the position and orientation of the hand, respectively. Furthermore, Term Frequency-Inverse Document Frequency (TF-IDF) and maximum entropy model are adopted to recognize the speech and gestures instructions of the user. The proposed method is compared with the three human-machine methods on various experiments. The results verified the effectiveness of the proposed method.

Introduction

Nowadays, robots have played an increasingly pivotal role in the development of technology. They are employed not only in manufacture, but also in other various domains such as search and rescue, mine and bomb detection, scientific exploration, entertainment and hospital care. Particularly, many new application areas need to provide some interactions between human and machines. For example, human and robots share the workspace to improve the quality and efficiency of complex tasks. Currently, various available methods have been proposed to implement human-machine interactions. Some methods are focusing on contact interaction between human and machine [1], [2]. Hyunki et al. [3] developed a soft wearable robot to replace a full-body rigid-frame exoskeleton device, improving the comfort of human-machine interaction. In [4], Zhang et al. proposed inertial/magnetic sensor module for improving the pedestrian tracking of the human-machine interaction. Gabriele et al. [5] attached the magnetic inertial unit to the waist for providing the effective sensor data fusion. Zhao et al. [6] used a brain-machine interface to capture the EMG signals, which were classified to generate robot commands. Xu et al. [7] used a haptic device named Phantom Device to control the remote manipulator, while force feedback was used to aid the operator. Rebelo et al. [8] developed a wearable arm exoskeleton master to achieve endto-end control of the robot.