Dexbotic Tutorial

This tutorial is designed as a hands-on quickstart. Follow it step by step to set up the environment, run a pre-trained model, train a model on provided simulator data, and test your trained model in simulation.

---

1. Environment Setup (Docker)

We recommend using Docker to ensure a consistent environment and avoid CUDA/Python dependency issues.

Prerequisites:

• Ubuntu 20.04 or 22.04
• NVIDIA GPU: RTX 4090 / A100 / H100 (8 GPUs recommended for training; 1 GPU for deployment)
• NVIDIA Docker installed

Steps:

# Clone repository
git clone https://github.com/Dexmal/dexbotic.git

# Launch Docker container
docker run -it --rm --gpus all --network host \
  -v /path/to/dexbotic:/dexbotic \
  dexmal/dexbotic \
  bash

# Inside container
cd /dexbotic
conda activate dexbotic
pip install -e .

---

2. Run a Pretrained Model

Download Pretrained Weights

Here we use the Libero CogACT model as example.

mkdir -p checkpoints/libero
cd checkpoints/libero
# Clone from Hugging Face
git clone https://huggingface.co/Dexmal/libero-db-cogact libero_cogact

Single-Sample Inference

CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_cogact.py \
  --task inference_single \
  --image_path test_data/libero_test.png \
  --prompt "What action should the robot take to put both moka pots on the stove?"

Expected output: 16 continuous 7D action vectors.

Deploy Mode (Server + Client)

Start server:

CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_cogact.py --task inference

Query via curl:

curl -X POST \
  -F "text=What action should the robot take to put both moka pots on the stove?" \
  -F "image=@test_data/libero_test.png" \
  http://localhost:7891/process_frame

---

3. Train on Provided Simulator Data

Download Datasets

We provide processed simulation data on Hugging Face:

• Libero

Organize into the following structure:

data/
  libero/
    libero_10/{video,jsonl}
    libero_goal/
    libero_object/
    libero_spatial/

Launch Training (Libero Example)

torchrun --nproc_per_node=8 playground/benchmarks/libero/libero_cogact.py

• Checkpoints will be saved under ./checkpoints/.
• For 8×RTX 4090, use scripts/deepspeed/zero3_offload.json to reduce memory usage.

---

4. Test Your Trained Model

Change Checkpoint Path for Testing Your Own Model

By default, playground/benchmarks/libero/libero_cogact.py uses the pretrained checkpoint path for inference.
To test your own trained checkpoint, you need to modify the model_name_or_path in the LiberoCogActInferenceConfig class.

How to do this:

1. Open playground/benchmarks/libero/libero_cogact.py.
2. Find the following code block:

   @dataclass
   class LiberoCogActInferenceConfig(InferenceConfig):
       # You should put the inference model path here
       model_name_or_path: str = field(
           default='./checkpoints/libero/libero_cogact')
       port: int = field(default=7891)

3. Change the model_name_or_path to your own checkpoint path, for example:

   model_name_or_path: str = field(
       default='./user_checkpoints/dexbotic/libero_all_cogact/your_checkpoint_folder')

4. Save the file.

Now, the inference server will use your trained model weights for testing.

Local Inference Check

Start inference server with your trained weights:

CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_cogact.py \
  --task inference \

Send a request:

curl -X POST \
  -F "text=What action should the robot take to put both moka pots on the stove?" \
  -F "image=@test_data/libero_test.png" \
  http://localhost:7891/process_frame

Simulation Benchmark Evaluation

Use dexbotic-benchmark:

cd dexbotic-benchmark
docker run --gpus all --network host -v $(pwd):/workspace \
  dexmal/dexbotic_benchmark \
  bash /workspace/scripts/env_sh/libero.sh /workspace/evaluation/configs/libero/example_libero.yaml

This will run the trained model against the Libero environment and report benchmark results.

---

✅ After completing these steps, you will have:

• A working Docker environment
• Pretrained model inference results
• A trained checkpoint on simulator data
• Benchmark evaluation of your trained model