Vision-Based Hand Gesture Recognition With Deep Machine Learning for Visual Servoing

Vision based robot motion control (Visual Servoing) is a challenging research direction in utilizing vision information to control the robot motion. In this work, we successfully controlled the real-time pose of an end-effector based on a hand gesture detector that we trained with acquired training data in the lab environment. Meanwhile, a machine learning model for hand language translation based on convolutional neural network is proposed and utilized in this paper. SSD is the suggested meta-architecture that uses single feed-forward convolutional network for straightly predicting categories. The proposed model is evaluated on Tensorflow platform along with Pascal VOC 2012. In addition, an image-based vision servoing system based on Lyapunov’s theory is developed to control velocities of the robot’s joints. In the experimentation, the integration of the above systems and MobileNet network as a convolutional feature extractor shows good performance in identification, tracking and motion control of the robot. The model achieved 98% mAP and 0.8 for Total-loss while visual servoing also demonstrated good performance during experimentation.