Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System
Abstract
The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block's pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row.
- Publication:
-
Sensors
- Pub Date:
- January 2023
- DOI:
- 10.3390/s23020752
- arXiv:
- arXiv:2211.07977
- Bibcode:
- 2023Senso..23..752M
- Keywords:
-
- Jenga;
- robotic arm;
- deep instance segmentation;
- visual servoing;
- sensor fusion;
- Computer Science - Robotics;
- Computer Science - Computer Vision and Pattern Recognition;
- Electrical Engineering and Systems Science - Systems and Control
- E-Print:
- Sensors. 2023