Humanoid perception & RL engineer — robot vision, locomotion policies, sim → sim → hardware, industrial CV
Everything here has to survive a physical robot — actuators that answer late, a head camera feeding a policy, a CPU with no GPU beside it. That constraint is what the work is about. The recurring theme is not a model architecture, it is refusing to trust a number that has not earned it: the policy that never falls until actuation delay is modelled, the gait reward that quietly rewards marching in place, the validation split that shares images with training.
The whole path from a robot description to a policy on the robot. The URDF is converted to MJCF and an automated verifier checks the model against the CAD before anything trains on it, because a policy trained on a wrong model learns the wrong robot. Velocity-tracking and motion-tracking tasks are trained with PPO in MuJoCo Warp / mjlab, with mass randomisation, datasheet torque limits and an actuation-latency model. The policy is exported to ONNX and run by a C++ state-machine runner behind a hardware-abstraction layer, so the same runner can face a second simulator or the hardware.
The sim → sim step earns its place. A policy with 0% falls in an idealised sim fell 100% of the time once a 45 ms actuation delay was modelled; retrained with the delay, it was back to 0%. Robot, CAD, customers and internal code stay with my employer.
gait_params_at_speed blends the cycle
0.6 s → 0.4 s and stance 0.56 → 0.40 between 1.0 and 2.0 m/s. The feet run
half a cycle apart, so once stance drops below 0.5 the two stance windows stop overlapping: double support
disappears and a flight phase opens. Under the fixed clock that gap can never exist, so a running gait
is unrewardable however the reward is tuned. Values from the task configuration; this animates the clock
(slowed 4×), not a recording of a trained policy.
Stand, walk forward and backward, run, and take hits from any direction — trained from scratch, no pretrained policy and no imitation data. Robustness comes from two different disturbances, not one: instantaneous push impulses, and a sustained random-heading drag that holds force for seconds and keeps stealing momentum. A shove and a drag fail a policy in different ways, so each gets its own curriculum, layered over a three-stage command-envelope curriculum so difficulty rises on two axes at once.
Anti-reward-hacking is a design axis. Every gait reward ships alongside the term that closes its exploit — marching in place, foot sliding, hard landings, posture drift. When the policy failed backward pushes, it got a dedicated isolation run and a targeted curriculum fix before any retraining: debugging a policy like a system rather than a slot machine. Trained end-to-end on 2×T4 with a checkpoint/resume harness that survives 9–12 h session kills.
source · github.com/shivpratapsinghpanwar/g1-robust-locomotion
The perception-to-action path of a humanoid, end to end in simulation: a Tiangong 2 Pro in MuJoCo with an RGB-D head camera producing metric depth, feeding SmolVLA inside a 50 Hz control loop. Actions go through a 17-joint upper-body action contract resolved by joint name, not by array position, so a reordered model or robot description cannot silently drive the wrong joint, and then through a safety filter — a ±0.30 rad envelope plus a rate limit — before anything reaches an actuator.
Status, stated plainly: this is a plumbing milestone. The loop, the contract, the filter and the latency harness work; the motion is not trained yet. The harness exists to measure what the model costs on the kind of CPU an edge robot actually has — and on this one, the base model is nowhere near real time. That number is the reason for TernaVLA.
source · github.com/shivpratapsinghpanwar/Humanoid_Perception_for_Edge
How large can a VLA backbone be and still keep up on a CPU? Ternary (1.58-bit) backbones are benchmarked on a laptop CPU at three sizes. The tier numbers use random weights at real layer shapes — they measure compute, not competence — and at 8 threads the 0.5B tier runs an action chunk in 0.44 s, fast enough for real time. The real 2B model runs under bitnet.cpp at 2.10 s per 91-token step, 21% faster than ONNX Runtime.
The vision side of the ternary stack. Checkpoints carry only the projector, the LoRA and optimiser state, so a session resumes from the exact data cursor — epoch, shard, row — and the trainer stops itself before the platform's session limit so a run never dies mid-write.
Multi-camera, multi-task perception — detection, instance segmentation, pose and depth — built and quantised from FP16 down to INT4 for real-time inference on constrained edge targets. Architecture, customers and deployment specifics stay with my employer.
99.4% similar, rot90), not a fuzzy-hash guess.
running…
Inspection data often comes from video of one line, where adjacent frames are near-copies that leak between splits unless something checks. A 10-second habit instead of a post-mortem: exact duplicates, near-duplicates, train/val leakage and COCO structural defects (duplicate ids, annotations referencing missing images, degenerate boxes and polygons, zero-annotation images) in one command. Exit code 1 on failure, so it drops straight into CI. Rotated, flipped, re-encoded and resized copies are caught, and the hash distance sets how much recompression it tolerates.
install · pip install vision-data-doctor
source · github.com/shivpratapsinghpanwar/data-doctor
The whole dataset workflow for an inspection line in one tool: annotate, import a model's predictions and correct them, convert between formats, deduplicate, split without leakage, resize, tile, augment, and report. One rule decides what ships: if a feature needs configuring before you can draw the first box, it does not ship. Mainstream annotation tools ask for an organisation, a project, a label schema and a task first. Here a class is created by drawing a box and typing its name.
The model loop. Annotate a few hundred images, train, then load the model's predictions back in — YOLO, VOC, LabelMe, COCO, or the bare detection-results list that evaluation scripts emit. Predictions render dashed and fade with confidence; one key accepts what is visible and moves on. Every import reports what did not match, because a file-name mismatch that imports zero shapes looks exactly like a model that predicted nothing. Export takes reviewed images only by default: feeding a model its own unchecked predictions as ground truth is how it poisons itself, and nothing in the output looks wrong when it happens.
Model agreement tells you when to retrain and when to stop. It counts only images a person has reviewed, and it counts deletions, since deleting a false positive is the commonest correction there is. Segmentation models such as SAM propose outlines from a separate process, so an out-of-memory crash takes the tool down, not the editor, and masks are simplified to editable polygons with a vertex cap rather than thousand-point outlines. Existing folders can be annotated in place, writing back to their own VOC, LabelMe or YOLO files atomically.
install · pip install easyannotate
source · github.com/shivpratapsinghpanwar/EasyAnnotate
Hobby project: conveyor-belt rock segmentation and sizing (Mask R-CNN, GigE camera, operator dashboard).
A delta only means something when it clears both its own spread and the class's per-image quantum (1/support). Held to that rule, the honest summary of this experiment is: no macro effect, one small real on-target gain, one small real on-target loss, and a large off-target gain with a mundane explanation.
Melanoma sensitivity is the clinically critical metric in this domain, and +0.116 would have been the
headline. The writeup declines to claim it, names the confound, and specifies the balance-matched control
(oversampling real rare classes) needed to separate a rebalancing effect from a realism effect. The
generation side is built to the same standard: every synthetic image passes memorisation gates before it may
be used, and each one is tracked in a provenance manifest flagged synthetic: true.
source · github.com/shivpratapsinghpanwar/Synthetic_Data_Factory
First-author work published at IEEE ICoEIT 2025: crop and fertilizer recommendation, leaf-photo disease detection, yield prediction, weather and live mandi prices behind one assistant. The assistant is not asked to know agronomy — it is handed thirteen tools and retrieval over the project's own knowledge base, so every answer traces back to a trained model or a dataset, and an offline rules engine keeps it answering with no API key or connectivity. Model choice was run as a seven-classifier comparison under 5-fold CV, not a single fit. Field-validated with 150+ farmers.
paper · doi.org/10.1109/ICoEIT63558.2025.11211713
code · github.com/shivpratapsinghpanwar/KrishiDisha.ai
Perception for a medical screening robot. Details withheld.