Skip to content
Changda Tian

Learning from Human Motion

HMAMP: Manipulate as Human

Task-oriented manipulation skills learned from human motion with adversarial motion priors, deployed on a real Kinova Gen3 arm.

Period
Mar 2023 – Sep 2023
Platform
Kinova Gen3 · Simulation
Methods
Adversarial Motion Priors · Reinforcement Learning · Keypoint Alignment
With
Ziqi Ma (SJTU), Yue Gao (SJTU)
Paper Video
HMAMP architecture: human video keypoints are aligned to the robot arm, and an AMP discriminator provides a style reward combined with the task reward for reinforcement learning.

Overview

Robots that work alongside people should manipulate tools the way people do — not only reaching the goal, but moving with recognizably human style. HMAMP (Human-style Manipulation with Adversarial Motion Priors) learns task-oriented manipulation skills whose style comes from human video and whose competence comes from reinforcement learning.

Core idea

From human manipulation video clips we extract keypoints of the arm and the tool, then align them to the robot arm’s kinematics in simulation. An AMP discriminator is trained to distinguish real human motion sequences from policy-generated ones; its output becomes a style reward. The policy is trained with the sum of the style reward and the task reward, so it must accomplish the task and fool the discriminator with human-like motion.

Experiments

We evaluated HMAMP on tasks where style and dynamics interact strongly:

  • hammering a nail;
  • clawing a nail out;
  • ball throwing and catching.

The learned policies outperform state-of-the-art baselines on benchmark tasks and generalize to novel objects.

Real robot experiment: human knocking motion clips used as priors (top), the policy in simulation, and the deployed Kinova Gen3 hammering a nail (bottom rows).

Real-world deployment

The policy transfers to a physical Kinova Gen3 arm, which completes the hammering task with the strike trajectory we designed for. The video below shows the human priors, the simulated policy, and the real arm side by side.

Publications

  • Manipulate as Human: Learning Task-oriented Manipulation Skills by Adversarial Motion Priors. Ziqi Ma, Changda Tian, Yue Gao. Robotica, 43(6):2320–2332, 2025. DOI
Project video.