EECS C106A

Handshake Bot

Team 38 · EECS C106A Final Project

Tech: ROS, Python, MoveIt, MediaPipe, Intel RealSense, Robotics

Project Overview

This project was completed for EECS C106A: Introduction to Robotics. Our goal was to build a robot system that could perform a human-like handshake by tracking a human hand in 3D and generating smooth, constrained robot motion in real time.

The write-up below is adapted from our team report and reformatted for this portfolio. Original reference: Handshake Bot project page.

Goal

Robotic motion is often precise but not human-like. We aimed to bridge that gap by building a handshake behavior that looked fluid and natural while staying safe and controllable.

This required combining:

  • real-time hand perception,
  • frame transformations between camera and robot coordinates,
  • smooth path planning with safety constraints.

Main Challenges

  1. Computer vision perception: reliably track a hand and estimate 3D position despite noise, lighting variation, and occlusion.
  2. Motion planning quality: generate handshake motion without unnatural joint twists, while still reacting to live input.

System Design

Functional Requirements

  • Track a human hand using vision and depth sensing.
  • Map that tracked point into robot coordinates.
  • Mirror or follow the motion to create handshake behavior.
  • Keep movement smooth and safe within a constrained workspace.

Four-Module Architecture

  1. Hand Tracking
    Use MediaPipe landmarks plus RealSense depth to estimate hand position in camera coordinates.
  2. Hand Transformation
    Transform points from camera frame to robot frame using calibrated transforms.
  3. Handshake Procedure / Mirroring
    Apply configurable mirroring behavior and publish target robot points.
  4. Path Execution
    Use MoveIt planning and constraints to execute smooth trajectories.

Design Trade-offs

  • A single RealSense camera gave a practical setup but introduced calibration sensitivity.
  • Wrist-point tracking was stable but did not capture full hand orientation.
  • Cartesian waypoint planning reduced jitter and improved smoothness, but introduced latency.
  • Additional filtering and thresholds reduced noise at the cost of responsiveness.

Implementation

Hardware

Software Components

  • cam_transform.py: saves/loads camera-to-robot transform.
  • realsense_tracking.py: publishes tracked hand pose.
  • realtime_move.py: transforms hand pose to base frame.
  • handshake_procedure.py: applies handshake/mirroring logic.
  • move_hand.py: plans and executes constrained motion with MoveIt.

Workflow

  1. Start robot and AR tracking infrastructure.
  2. Calibrate camera transform and save parameters.
  3. Run hand tracking and frame transformation nodes.
  4. Configure handshake mirroring procedure.
  5. Execute motion with constrained Cartesian planning.

Results

The system successfully tracked a human hand in 3D and mirrored it with the Sawyer arm to produce handshake-like motion. Behavior was generally smooth and safe within the defined workspace. The largest practical limitation was latency from real-time planning and execution.

Reflection

What worked well

  • End-to-end integration from perception to control.
  • Smoothness improvements from waypoint queueing, thresholding, and Cartesian paths.
  • Safety improvements from collision objects and joint constraints.

Difficulties

  • Camera calibration robustness in imperfect lighting and marker conditions.
  • Planner occasionally choosing inefficient or delayed trajectories.

Future improvements

  • Faster online planning for better responsiveness.
  • Better calibration pipeline with less manual setup.
  • Richer hand representation (orientation, more landmarks) for more natural interaction.

Team

  • Saurav Suresh — RealSense tracking
  • Naman Satish — Handshake procedure
  • Alexander Lui — Motion execution (move_hand.py)
  • Evan Chang — Camera transform and frame conversion