[2025-01] Motion Capture Teleoperation System

Note: To protect commercial confidentiality, public information is limited.

Project Overview

Developed as a key feature within the InternUtopia project, this system enables real-time teleoperation of a Franka robotic arm through monocular RGB camera-based hand gesture recognition, while simultaneously collecting trajectory data during operation.

Caption: Real-time gesture-to-arm motion mapping demonstration

Caption: Useful tools in system

Key Objectives:

• Low-cost solution for collecting robotic arm trajectory data in complex or long-horizion tasks

Highlights:

• Cost-effective: Requires only a single RGB camera without additional hardware
• Intuitive Controls: Special gestures enable:
  • Third-person view adjustment
  • Coordinate system recalibration
  • Motion precision tuning
• Optimized Performance: Human-robot motion semantic consistency with enhanced precision and real-time responsiveness

Technical Documentation provides detailed setup instructions and gesture lexicon. Below is a system architecture overview:

Hardware Requirements

Recommended:

• 2× (≥)NVIDIA RTX 4060 Ti GPUs (1 for Hamer, 1 for GRUtopia)
• 1× RGB camera (USB webcam or built-in laptop camera)

Note:

• Single-GPU operation possible with reduced frame rate
• Camera specifications are flexible

Implementation Workflow

1. Launch real-time video streaming server
2. Initialize hand gesture recognition service
3. Start Franka control program

Gesture Lexicon

• Right Hand: Directly controls end-effector
  • Thumb-index pinch: Close gripper
  • Pinch release: Open gripper
• Left Hand: Auxiliary functions
  • Thumb-index pinch: Adjust third-person view
  • (See documentation for complete gesture library)