Train an ACT Policy for an SO-101 Robot using LeRobot
Part 3 in Trelis Robotics Series
This is the third video in the Trelis series on robotics!
I describe how to collect data for training, and then I train the ACT policy for an SO-101 robot!
I then evaluate performance and talk about what works and what doesn’t with this lightweight ACT policy.
Cheers, Ronan
🤖 Purchase ADVANCED-robotics Repo Access
Access all SEVEN Trelis Github Repos (-robotics, -vision, -evals, -fine-tuning, -inference, -voice, -time-series)
Support via Github Issues & Trelis’ Private Discord
Early access to Trelis videos via Discord
Timestamps:
00:00 Introduction to Training the SO-101 Robot with ACT
00:21 Overview of the Video Series
01:16 Scripts and Repo Access: Trelis.com/ADVANCED-robotics
01:57 Cloning and Installing LeRobot Libraries
06:07 Connecting and Configuring the Robots
08:53 Calibrating the Motors and Arms
12:33 Teleoperation Setup
18:04 PID Controller Calibration
27:10 Recording and Managing Data
39:05 Training the ACT Model
44:33 Style settings and KL Weight (ADVANCED)
49:06 Running Training on a Mac (or cpu)
50:54 Setting Up Validation and Output Directories
53:44 Running Training on Mac and Handling Issues
55:31 Monitoring Training Progress
57:25 Calculating Training S teps and Epochs
58:28 Analyzing Training and Validation Loss
01:04:02 Setting Up Training on GPU
01:08:19 Connecting to Remote Host and Cloning Repo
01:12:48 Running Training on CUDA
01:14:48 Handling Issues Running on CUDA
01:23:28 Inspecting Results after Running on CUDA
01:27:04 Evaluating Model Performance
01:28:37 Replay and Evaluation of Training Examples
01:35:21 Challenges with Generalization and Data Requirements
01:36:28 Using Image Augmentations and Jitter
01:37:08 Deciding Number of Rollout Steps
01:40:06 Ensembling Predictions for Smoother Trajectories
01:43:43 Conclusion and Next Steps
Trelis Links:
🤝 Are you a talented developer? Work for Trelis
💡 Need Technical or Market Assistance? Book a Consult Here
💸 Starting a New Project/Venture? Apply for a Trelis Grant
Training a Robot Arm with ACT: From Data Collection to Evaluation
A detailed walkthrough of training an Action Chunking Transformer (ACT) policy on an S0-101 robot arm for a pick-and-place task, using the lerobot library. The process involves data collection through teleoperation, model training, and performance evaluation.
Initial Setup
Install LeRobot library from HuggingFace
Set up Conda environment with Python 3.1
Configure USB ports and permissions for robot communication
Calibrate robot motors and arm positions
Data Collection Process
Record training examples via teleoperation
Capture both wrist camera and front camera views
Recommended: 50+ training examples for good performance
Can append additional examples to existing dataset
Validate recorded episodes through replay
Training Configuration
Key parameters:
Chunk size: 50 steps (predicts ~1.66 seconds ahead)
Action steps: 15 (uses first 0.5 seconds of prediction)
Batch size: 4 for small datasets, up to 32 for larger ones
Training steps: Based on dataset size (e.g., 7,500 steps for 3,000 frames)
Validation split: 10% of data or max 32 episodes
Model Architecture Considerations
ACT uses a style encoder to handle multiple motion patterns
KL weight controls style encoding strength
Small datasets may not benefit from style encoding
Image transforms (jitter) only recommended for large datasets
Performance Characteristics
Works best when test conditions match training examples
Limited generalization to new cube positions
Requires comprehensive training data for position variations
Ensembling can affect smoothness but may reduce accuracy
Style encoding may be unnecessary for simple, consistent demonstrations
Practical Tips
Start with exact training scenario reproduction
Gradually test generalization boundaries
Monitor validation loss for training convergence
Consider reducing model complexity for small datasets
Maintain consistent lighting and camera positions
The results demonstrate ACT's capability to learn specific tasks but highlight its limitations in generalisation without comprehensive training data.


Hi Ronan, have you purchased Reachy Mini? It also looks very interesting, haha