Learning-assisted control under unknown payloads

A Franka Panda reaches for a target outside the region of its training data. Left: the policy alone stops 13 cm from the target. Right: a monitor gives an alarm at 0.40 s, a model-based controller takes control, and the tip stops 0.1 cm from the target. This is a typical test run, not the best one.

The question

A model-based controller is only as good as its model. A policy that learned from demonstrations has no model, but it expects the arm to respond as it did in the demonstrations. When the arm holds an object of unknown mass, both go wrong. I asked three questions, each on top of the one before:

1. Correct the model
   Is it better to estimate the payload mass, to train a neural network on the model error, or both? What occurs when the controller uses a learned model outside its training data?
2. Monitor the policy
   A diffusion policy controls the arm. Which model must a monitor use to detect its failure: a physics model of the empty arm, or a learned model? When must the monitor give control to the model-based controller?
3. Change the platform
   Are the results also true for a 7-joint Franka Panda with joint impedance control, at two joint stiffnesses?

Setup

• Arms
  A planar two-link arm driven by joint torques, and a Franka Panda (MuJoCo Menagerie model) driven, like the real robot, by a joint impedance loop. On both, the controller’s model gives the same motion as the simulator, step by step, when it knows the true payload. Thus each remaining error comes from a model error that I set.
• Model errors
  Three levels: the payload mass only; the mass and an offset of its centre of mass; and both of these plus joint friction that the model does not have.
• Controller
  A sampling-based model predictive controller (MPC). All model-based methods use the same MPC and the same settings. Only the model changes.
• Policy
  A closed-form diffusion policy (CFDP) that copies an expert MPC. It learned from 400 episodes on the empty arm.
• Monitors
  A physics monitor compares the measured motion with the motion that a model of the empty arm predicts. A learned monitor adds a neural network that learned the model error from the same demonstrations. A consistency monitor needs no model: it compares two overlapping action chunks of the policy. Each monitor adds its signal over time (CUSUM), and its threshold comes from clean calibration runs, for a false-alarm rate of 5%.
• Handover
  The policy controls the arm until the first alarm. Then the MPC takes control, with the payload mass estimated from the data of the run.

Part I: correct the model

Two-link arm. Median final tip error of 24 runs against the quantity of training data. Solid: targets inside the training region. Dashed: outside. C1 estimates the mass; C2 learns the model error; C3 does both; C3-E averages five networks.

• The quantity of data decides the best method. With 9 s of motion data, the mass estimate is more accurate. With 30 s or more, physics plus a learned correction is the most accurate inside the training region: 0.07 cm final error, against 1.95 cm for the mass estimate.
• Outside the training region, the MPC finds and uses the errors of the learned model. Along the paths that it selects, the prediction error increases by 8 to 13 times, and the worst runs stop 50 to 66 cm from the target. For every wrong model, physical or learned, the MPC expects a result that is better than the real result by half to all of the model error. An ensemble of five networks decreases the worst failures.

Part II: monitor the policy

Two-link arm, 32 tasks for each point. At 0 kg the alarm rate is the false-alarm rate (target 5%, dashed). Left: the model of the empty arm is exact. Right: the model does not know the joint friction.

• A policy that learned on the empty arm fails with a 1 kg payload: the median error increases from 0.4 cm to about 3 cm. Its action chunks mostly stay consistent while the arm moves down under the load.
• The two types of monitor detect two different failures. The residual monitors detect the payload. Only the consistency monitor detects that the policy is outside its training region (94% of runs).
• When the physics model does not know the friction, the learned monitor detects much smaller payloads: 63% of runs at 0.05 kg, against 13% for the physics monitor.
• A detected failure does not always justify a handover. When the MPC that takes control does not know the friction, it can be worse than the policy.

Part III: the Franka Panda

The Panda with a 2.5 kg payload, target inside the training region. Left: the policy alone stops 7.3 cm below the target. Right: the physics monitor gives an alarm at 0.08 s, and the MPC takes control.

Share of runs that are not within 1 cm of the target after 3 s. Each bar shows 96 runs: 16 tasks, 3 model errors, 2 joint stiffnesses.

• The results of Part II are also true on the Panda. With a 2.5 kg payload, every handover recovers every run, within 0.04 cm of an MPC that estimates the payload from the start. Outside the training region, the consistency monitor recovers the policy.
• On the Panda, a learned model is of little use for control. The impedance loop removes most of the model error, so a learned model can decrease the final error by 3.5 mm at most. But it is useful for monitoring: when friction hides a 1 kg payload, the learned monitor gives the alarm after 0.2 s, and the physics monitor after 1.6 s.
• The monitors do not detect all failures. In one run, the policy stopped 20 cm from the target with consistent action chunks, and no monitor gave an alarm.

Method

Before each test run in Parts II and III, I wrote down and committed the expected result: 26 statements in total. Of these, 16 were correct, 8 were partly correct and 2 were wrong. The report gives all of them, with the reasons for the wrong ones. All settings were selected on development tasks and did not change for the tests. Test tasks have their own random seeds. The code is public. A single command runs all experiments again (about 6 to 7 hours on a laptop CPU), and 131 tests check the models against the simulator.

What this is not

This is a simulation study, not a real robot. All tasks are reaching tasks, and for these an MPC with an online mass estimate is as good as the best handover. Thus the policy is not necessary here; the study only asks if monitors and a handover can recover a policy that fails. The payload is on the arm from the start of each run. No new method is proposed.

• Report (PDF, 6 pages)
• Code (GitHub)