Ahmed Sohail Butt

Computer Science, Wilfrid Laurier University (2027) · software engineering → embodied AI

GitHub Hugging Face ahmedsohaill2003@gmail.com

One simulated industrial work-cell. Five ways to teach a robot.

The WorkCell series: one robot (SO-ARM100 in MuJoCo), one manipulation setting, taught five different ways — imitation learning, a vision-language-action foundation model, classical 3D perception, reinforcement learning with a from-scratch world model, and a ROS 2 deployment layer with a C++ port. Every project publishes its datasets and models, and reports failures as carefully as successes: corrected measurement bugs, negative results with forensics, and a claim-by-claim paper reproduction.

Stack: PyTorch · LeRobot · Hugging Face · MuJoCo · Stable-Baselines3 · ROS 2 · C++ · NumPy/SciPy

ACT policy performing pick-and-place from pixels (front and wrist cameras)

PART A · IMITATION LEARNING

Sim2Cell — ACT from pixels

End-to-end imitation pipeline built from scratch: env → IK → scripted expert → LeRobot datasets → ACT → eval forensics.

A domain-randomization study where DR training turned out to be regularization, not a tax: identical recipe, only the data changed, +27 points on the nominal task. Includes two documented negative results — a temporal-ensembling sign flip on multimodal data, and a targeted-data "coverage whack-a-mole" verified episode-by-episode on a fixed eval seed.

90% (45/50) from pixels87–100% robust under DR4 datasets + 3 models on 🤗free-tier T4 training

Repository ACT reproduction report Best model

SmolVLA executing a language-conditioned pick-and-place

PART B · VISION-LANGUAGE-ACTION

TalkToTheCell — SmolVLA fine-tune

"Pick up the red block and place it in the blue tray" — a 450M VLA taken from 0% zero-shot to 90%, on a free Kaggle T4.

The headline is a controlled cross-architecture study: the identical recovery-demo dataset moved the two architectures in opposite directions — chunked-regression ACT fell 65→50 while SmolVLA's flow-matching action expert learned both behavior modes and jumped 55→90. Same data; the architecture decides whether behavioral diversity is signal or poison.

0% → 55% → 90% (18/20)language-conditionedACT −15 vs SmolVLA +35 on same data12k steps on one T4

Repository Model (90%)

RGB-D perception pipeline picking cubes camera-to-tray

PART C · CLASSICAL PERCEPTION

GraspSight — RGB-D → 6-DoF grasp

Vision-guided picking with zero runtime ground truth — every geometric algorithm implemented from scratch in NumPy/SciPy.

Pinhole back-projection, RANSAC plane removal, DBSCAN clustering, and symmetry-aware table-constrained ICP recover object pose from depth alone; hand-eye extrinsics come from an industrial precision-sphere calibration routine (0.32 mm). A self-calibrating look-then-move executor closes the loop with a slide-in grasp against 1.9 mm of jaw clearance.

20/20 camera-to-tray picks0.58 mm ICP pose error0.32 mm hand-eye calibrationno Open3D, no PCL

Repository

RL policy driving the arm to randomized targets

PART D · RL + WORLD MODELS

PolicyForge — model-free vs model-based

PPO and SAC benchmarked against a from-scratch world model (PETS-style dynamics ensemble + CEM-MPC) on reach and contact-rich lift.

The learned world model hits 100% reach success from 8k environment steps — 37× fewer interactions than PPO — reproducing the sample-efficiency hierarchy that matters when robot time is the cost. Contact-rich lift was solved by root-causing a reset bug and running a documented four-stage reward-design study (watch the lift-and-hold rollout).

100% reach @ 8k steps37× fewer than PPO, 12× than SAClift: 3-seed SAC, mean 73%4-stage reward study

Repository Learning curves

Nav2 navigating to goals on a self-built SLAM map

PART E · ROS 2 + C++

CellOps — SLAM, Nav2, and a C++ service

The deployment layer: autonomous mapping and navigation, plus Part C's pose estimator ported to C++ and served over ROS 2.

slam_toolbox maps the world under a closed-loop lidar-guided explorer, then Nav2 + AMCL hits goals derived automatically from the map — fully scripted, 12/12 across two independent runs. The C++ port (from-scratch KD-tree, no PCL) is numerically identical to the Python reference and 3.3× faster; nine silent-failure modes found en route are documented in the build log.

12/12 Nav2 goals0.000 mm C++/Python parity14.0 → 4.33 ms (3.3×)9 failure modes documented

Repository Build log

Research & open source

Reproducing ACT (Zhao et al., RSS 2023) on a $150-class arm Claim-by-claim reproduction study with verdicts, Wilson CIs, and honest NOT-TESTED scoping — including an extension the paper's regime never exhibits: temporal ensembling's benefit inverts on multimodal data, and per-step replanning without aggregation collapses the policy to 5%.
Upstream fixes to LeRobot (Windows) Three bugs found, root-caused, and fixed while building Part A: #4174 (POSIX-formed Hub patterns), #4177 (torchcodec load guard), and an independent Windows verification review of #4121 (checkpoint symlink → junction fallback).
4 LeRobot datasets + 5 models on Hugging Face Every headline number on this page is reproducible from public artifacts — including the negative results, which ship with the same quality of model card as the successes.
Peer review welcome Spotted a flaw, an overclaim, or something that wouldn't survive real hardware? I'm actively collecting technical feedback in this open thread — blunt criticism appreciated.