Train an ACT Policy for an SO-101 Robot using LeRobot

Part 3 in Trelis Robotics Series

This is the third video in the Trelis series on robotics!

I describe how to collect data for training, and then I train the ACT policy for an SO-101 robot!

I then evaluate performance and talk about what works and what doesn’t with this lightweight ACT policy.

Cheers, Ronan

🤖 Purchase ADVANCED-robotics Repo Access

---

🗝️ Get Trelis All Access:

1. Access all SEVEN Trelis Github Repos (-robotics, -vision, -evals, -fine-tuning, -inference, -voice, -time-series)

2. Support via Github Issues & Trelis’ Private Discord

3. Early access to Trelis videos via Discord

---

Timestamps:

00:00 Introduction to Training the SO-101 Robot with ACT

00:21 Overview of the Video Series

01:16 Scripts and Repo Access: Trelis.com/ADVANCED-robotics

01:57 Cloning and Installing LeRobot Libraries

06:07 Connecting and Configuring the Robots

08:53 Calibrating the Motors and Arms

12:33 Teleoperation Setup

18:04 PID Controller Calibration

27:10 Recording and Managing Data

39:05 Training the ACT Model

44:33 Style settings and KL Weight (ADVANCED)

49:06 Running Training on a Mac (or cpu)

50:54 Setting Up Validation and Output Directories

53:44 Running Training on Mac and Handling Issues

55:31 Monitoring Training Progress

57:25 Calculating Training S teps and Epochs

58:28 Analyzing Training and Validation Loss

01:04:02 Setting Up Training on GPU

01:08:19 Connecting to Remote Host and Cloning Repo

01:12:48 Running Training on CUDA

01:14:48 Handling Issues Running on CUDA

01:23:28 Inspecting Results after Running on CUDA

01:27:04 Evaluating Model Performance

01:28:37 Replay and Evaluation of Training Examples

01:35:21 Challenges with Generalization and Data Requirements

01:36:28 Using Image Augmentations and Jitter

01:37:08 Deciding Number of Rollout Steps

01:40:06 Ensembling Predictions for Smoother Trajectories

01:43:43 Conclusion and Next Steps

---

Trelis Links:

🤝 Are you a talented developer? Work for Trelis

💡 Need Technical or Market Assistance? Book a Consult Here

💸 Starting a New Project/Venture? Apply for a Trelis Grant

---

Training a Robot Arm with ACT: From Data Collection to Evaluation

A detailed walkthrough of training an Action Chunking Transformer (ACT) policy on an S0-101 robot arm for a pick-and-place task, using the lerobot library. The process involves data collection through teleoperation, model training, and performance evaluation.

Initial Setup

1. Install LeRobot library from HuggingFace

2. Set up Conda environment with Python 3.1

3. Configure USB ports and permissions for robot communication

4. Calibrate robot motors and arm positions

Data Collection Process

1. Record training examples via teleoperation

2. Capture both wrist camera and front camera views

3. Recommended: 50+ training examples for good performance

4. Can append additional examples to existing dataset

5. Validate recorded episodes through replay

Training Configuration

Key parameters:

1. Chunk size: 50 steps (predicts ~1.66 seconds ahead)

2. Action steps: 15 (uses first 0.5 seconds of prediction)

3. Batch size: 4 for small datasets, up to 32 for larger ones

4. Training steps: Based on dataset size (e.g., 7,500 steps for 3,000 frames)

5. Validation split: 10% of data or max 32 episodes

Model Architecture Considerations

1. ACT uses a style encoder to handle multiple motion patterns

2. KL weight controls style encoding strength

3. Small datasets may not benefit from style encoding

4. Image transforms (jitter) only recommended for large datasets

Performance Characteristics

1. Works best when test conditions match training examples

2. Limited generalization to new cube positions

3. Requires comprehensive training data for position variations

4. Ensembling can affect smoothness but may reduce accuracy

5. Style encoding may be unnecessary for simple, consistent demonstrations

Practical Tips

1. Start with exact training scenario reproduction

2. Gradually test generalization boundaries

3. Monitor validation loss for training convergence

4. Consider reducing model complexity for small datasets

5. Maintain consistent lighting and camera positions

The results demonstrate ACT's capability to learn specific tasks but highlight its limitations in generalisation without comprehensive training data.