I fine-tuned π0.5, a vision-language-action model, to run on my SO-101. The arm makes a nice smooth approach and a confident descent onto my blocks, closes the gripper, and carries them across to the wall it's building on the other side. In a controlled test with the fence already loaded, it got the block there five times out of six.
That clip is the first time it worked, and it's not even the final policy. It's the first grasp of the first run: the v1 checkpoint at step 2,000, only an hour into training. The policy that finally did the job reliably came out of the second of four fine-tunes.
In between those two sat the worst roadblocks, and that's what this post goes through in detail. Trained on to the end, v1 turned out to be picking from memory. I later realised the literature already has a name for this: shortcut learning, or causal confusion when it happens in imitation learning.
It would make the same confident descent onto a patch of bare table, the exact spot where a pink block had sat a minute earlier, and try to grasp air. Its loss curve, on the other hand, fell the way the textbooks promised me it should, and didn't alert me to anything being off. You can't trust your loss curve...
After eleven hours of training on 65,810 frames, it had still hardly learned to actually look at what the cameras were showing it. Getting out of that rut was lesson #1 in robotics for me, and it took two fixes:
- I engineered entropy into the training data. A script generated randomised block layouts and I staged every new wall from one, so the model could no longer guess a block's position from the state of the stacked wall and had to actually look at the scene.
- A serving change fixed how it finished. With the RTC execution horizon at 20 instead of 5, single-block transports went from zero in six to five in six, with no retraining.
My demonstrations were its training data, and every disciplined decision I thought I was making in good faith is exactly what quietly taught the arm to cheat.
Like I said, the loss curves said almost nothing through all of this, but the probes I hand-rolled said a great deal. The cheat turned out to be measurable to a fraction of a joint unit. The fix was scene entropy to make the policy look, and a serving setting to make it finish.
This post is what I learnt along the way, with the short list in Takeaways. It covers how to record data so the model can't take a shortcut, which signals to trust (not the loss), and how to find out what is actually going wrong. Findings come first, then the task, the data and the evaluation protocol, then the experiments.
You can check every number here, because the models and datasets are public on Hugging Face. The four checkpoints are pi05_wall, pi05_wall_v2, pi05_wall_v3_no_dagger and pi05_wall_v3_rel, and Assets lists the five training datasets.
One warning before we start: more than one plan in this post is overturned by a section that comes after it, and at least one number stated with confidence gets questioned later. At the end of the post, I've dumped my Build Log for anyone interested :)
Key Findings
- A serving setting moved task completion more than any training change did. On v2b, the second fine-tune, raising the RTC (Real-Time Chunking) execution horizon from 5 to 20 took single-block transports into an already-loaded fence from zero in six to five in six. There was no retraining, and I measured it once at n=6 (Chunk Seams, Gut Versus Data).
- Scene entropy changed where the policy looks, not how well it executes the grasp. After 115 card-staged scatters and 29 corrections, blind closes on empty table fell from 75% to 27% of all closes under one detector. Look-down-and-refuse aborts went from 16 to 73, and the remaining misses had the block in view (Veto First).
- The data itself contained the shortcut, and I could measure it. Wall position alone explained 51 to 66% of the variance in the pick pose. 84% of first picks sat in one lateral band, with a standard deviation of 2.8 joint units against 13.1 overall (Script Measured).
- Eval loss could not see the failure. v1's best held-out eval loss was 0.0848 while 85 of its 94 probe grasp attempts went to one table station, whether or not a block was there. The train-eval gaps of the broken v1 (−0.023) and the improved v2b (−0.025) are indistinguishable (Loss Lied, Twin Gaps).
- The retrained policy learned that hovering means closing. Six failure modes appeared in it, and most trace to demonstrations that never showed an unconverged hover or a failure state (Hover Means Close, Gut Versus Data).
- Naively merged correction snippets did not rescue the grasp, and removing them did not either. v3 without the DAgger episodes produced a qualifying hold in two of five near-miss reps, under a bar of three. BRIDGE's 44% to 0% to 76% ladder said removing a naive mix should help, and on this bench it didn't (Naive Mixing, Last Run).
- My demonstrations were one motion performed 252 times. Episode-length coefficient of variation was 0.18, tempo 0.14 and jerk 0.10, and the top two principal components carried 54% of the trajectory variance in each source, 48% merged (Read Our Mail).
- The grasp failure is still unexplained. I have three candidate mechanisms and didn't test any of them to a result. The fine-tunes defaulted to absolute joint targets against a delta-trained prior. Every commanded close lived in the bottom 4.5% of the gripper's normalised range, with the model's raw outputs pinned at −1. And expert-only training has a documented quality gap (Folklore Numbers, Bottom Millimetre).
- Human-in-the-loop RL ran end to end on this rig, and the policy gamed its reward in seconds. An 892,000-parameter policy found the weak spot in the reward classifier, and I walked away from the approach on cost (Reward Hacking, Walking Away).
- Serving π0.5 from the GPU box was slower than serving it on the Mac. Two RTX 3090s over Tailscale measured 900 to 1,390 ms per chunk, against 294 ms on the Mac (294 Milliseconds).
- Where it ended: the best policy is still v2b at horizon 20, which carried the block into a loaded fence five times in six. The two later retrains aimed at a firmer grasp hold did not beat it on record. v3_no_dagger held in two of five near-miss reps. The relative-actions retrain finished at an eval loss of 0.0548 (not comparable across action spaces) and got one 59-second probe episode, 38 close commands and not one hold, in a session cut short by a servo-bus fault, so its battery never ran. The one-block ladder is the next step (Last Run).
The Task
I spent a full day on task design before I recorded a single episode, and I still stand by every decision in this section. They matter because of what they cause later, in Clever Hans.
The trick that makes this project doable is to record a short primitive, not the whole task. You don't record "build a wall". You record "pick one block from the pile and place it on the wall" and run it until there's a wall. A 60-second long-horizon task becomes a 12-second primitive run four times, and short tasks are what imitation learning is good at.
You also get every wall state in the dataset for free, in proportion. Episode one starts with an empty fence and a full pile, episode two with one block placed, and so on. Within a wall I only reset the arm. The pile shrinks because it will at deployment too.
Short primitives also matter because multi-step success is roughly multiplicative, and the table that shows it is the most sobering one I've read. ALOHA Unleashed, with 3,198 demonstrations of a kitchen task, saw success through successive stages fall from 95% to 65% to 25%1. Behaviour cloning gets much worse as the task gets longer, so you keep each run short and let repetition build the structure.
The scene is four pink wooden blocks (Flying Tiger's finest, about 75×25×15 mm) scattered flat on the right, a taped yellow corner fence on the left, and one lamp. Almost everything in that sentence was a deliberate choice.
The blocks lie flat and scattered rather than in a heap, because picking from a heap is singulation from clutter. That's a hard research problem, and a bad fit for a position-controlled gripper with zero force sensing. A tilted block has no flat face, and a wedged block stalls the gripper servo, the one with the documented burnout history. Random positions are free diversity. Random stacking is a different research problem that I didn't want by accident.
The blocks are all one colour because language-grounded colour selection is the documented 0 to 10% failure mode of every policy benchmarked on this arm2. With an all-pink scene, the task never asks the model to choose a colour.
The fence is there because block tasks usually fail at placement, not grasping, and the literature is blunt about it. In simulation benchmarks of real-robot policies, stacking converts grasps into task success far less often than placing into something that catches the release3. With ten demos, MimicGen's stack-two sits at 26% while stack-three collapses to 0.7%4. The fence moves the precision out of the learned controller and into the physical world, so a release good to within a centimetre still ends up flush. It's one of the oldest tricks in robotics: let a fixture do the precise part.
I kept the pile zone opposite it tight, about 12 to 15 centimetres square, because a small workspace needs less data (a community data point I never re-verified says 50 episodes over 30 cm failed where 75 over 10 cm worked).
One more fact about the arm sits behind all this: the setup never records force. I didn't want to believe it, so I checked the source. The Feetech STS3215 has four operating modes (position, velocity, PWM, step), and none is current or torque control. The servo has load and current registers, but nothing in LeRobot's robot classes reads them, so the observation is joint positions and nothing else. Force never gets into the dataset, so a policy trained on this arm can't learn to modulate it. π0.5 can't "grip gently" here, because neither the demonstrations nor the action space has the concept.
That's the real reason the blocks are rigid wood, and why my gripper rule was to find one close that holds and reuse it. If nothing can be crushed, it doesn't matter that the arm can't feel force. The risk left is the motor, not the block. Wood doesn't compress, so closing too far is a stall, and stalls are what kill these grippers5. At 15 grams the blocks also avoid the lightweight-object problem. AhaRobot's team found their featherweight cup "prone to being knocked over", which dropped performance at the grasp step6, and a knocked-over object puts the scene into a state no demonstration contains.
The Rig
The rig is two identical arms from one kit, plus a Mac and a GPU box. The kit is a build of LeRobot's SO-101. One is the follower, the arm that grips things. The other is the leader, an identical skeleton with a handle instead of a gripper. You puppet the leader with your hand and the follower copies it joint for joint, which is how demonstrations get recorded.
If you want the wider context for why cheap arms like this exist and why imitation learning took over robotics, I wrote a whole primer in Robotics from Scratch. This post is what happened when I actually did it.
Once built, the two arms only differ at the wrist. Both are clamped to the table so neither can shift relative to the cameras mid-session.
Two cameras feed the policy: front, a Razer Kiyo Pro on a tripod watching the workspace, and wrist, on the follower's camera mount. Both run at 640×480 and 30 fps, because π0.5 resizes every frame to 224×224 before the model sees anything. Getting Eyes explains why capturing more isn't worth the cost.
The arm is only precise to about a centimetre, and no policy on it can place better than that. The firmware has a deadband of about 0.9 degrees, and a degree or more of backlash compounds across five serial joints. One degree at the shoulder is over 5 mm at the gripper (Calibration Numbers has the measurements). The gripper also has no force channel at all, which The Task already designed around.
The compute is split across two machines. A MacBook with an M5 Max records, teleoperates and runs every rollout. A Windows box with two RTX 3090s (24 GB each) does the training inside WSL2, and I reach it over Tailscale. The Build Log covers how a box of parts became this rig, from motor IDs and calibration spans to a four-hour servo fault and a camera that wasn't there.
I ran four fine-tunes on it, all starting from lerobot/pi05_base with the recipe in Training Recipe and these differences:
| Run | Dataset | Episodes / frames | Steps | eval_split | Image transforms | Relative actions |
|---|---|---|---|---|---|---|
pi05_wall (v1) | jenga_wall_20260810_155119 | 149 / 65,810 | 20,000 | 0.05 | off | no |
pi05_wall_v2 (v2b) | jenga_wall_v2b | 293 / 137,073 | 15,000 | 0.05 | on | no |
pi05_wall_v3_no_dagger | jenga_wall_v3_no_dagger | 264 / 118,322 | 20,000 | 0.045 | on | no |
pi05_wall_v3_rel | jenga_wall_v3_rel | 264 / 118,322 | 20,000 | 0.045 | on | yes |
Data Collection
Before recording anything, I checked I could do the task myself from the camera feeds alone, then froze the cameras, the scene and the recording procedure. My runbook called this rehearsal L0: do the task through teleop while watching only the camera feeds in Rerun, never the real arm. It's an upper bound. The policy will see two 224×224 images and nothing else. If a human with a lifetime of visual priors and a real-time brain can't do the task from those two images, a dataset won't teach a network to.
The test mostly catches missing depth. If you catch yourself glancing at the real workspace to judge how far the gripper is from the block, the cameras aren't carrying that information. Better to move a camera now than debug a mystery failure three days later.
L0 passed at half past midnight, three nights before recording: ten clean pick-and-places with my eyes only on the feeds. The idea isn't mine. LeRobot's own data-gathering tips say "you should be able to do the task yourself by only looking at the camera images"7, and my runbook just made it a formal gate with a name. Driving a robot through a screen while your hands hold a second robot is a strange little skill, and fun once it clicks.
From L0 onward the camera views had to stay fixed, so I taped and witness-marked both cameras right then. They aren't just my camera angles any more. They're everything the policy will ever see.
The frozen views weren't pretty, and that was fine, because for observations consistency beats accuracy. The wrist image was soft, because webcams focus at arm's length and the blocks sit centimetres away, and the bench lamp gave it a heavy warm tint. Neither matters as long as it stays the same. π0.5 sees 224×224, where the soft focus disappears, and a colour cast is just part of the distribution as long as the same lamp is on in the same place at recording and rollout. The standard is "good enough and frozen".
The masking tape along the frame edges marks the working area as the front camera sees it, and it shows me straight away if a camera drifts.
The Lock List
Everything else from The Task got locked too. This is the actual lock list from my recording doc, and the thing I'm most pleased with from that week:
| Locked | Value |
|---|---|
| Scene | 4 pink blocks scattered right, fence left |
| Build order | Fixed slots, closed end first, each block flush beside the last |
| Selection rule | Closest block first |
| Grasp | Top-down, centred, constant wrist angle |
| Block pose | One orientation everywhere, no rotations |
| Motion shape | Same arc every episode, lift, travel, descend |
| Home pose | Taped mark, returned to every reset |
| Lighting | Same lamp, same position, blinds shut so every session looks like night |
| Frame contents | Task only; hands, phones and mugs stay out |
| Cameras, fixture, clamps | Witness-marked, untouched |
One thing I couldn't lock was the front camera's autofocus and auto-exposure. I couldn't find a way to turn them off on the Kiyo at the time, so they went on the list as a watch-item: keep an eye on the front feed for focus hunting.
One Fifty
The real recording session produced 150 episodes in one sitting: roughly 38 walls of four blocks, in under two hours with breaks, from mid-afternoon with the blinds shut and the lamp on. The quality gates all passed except for two blemishes. I deleted one and kept the other.
The one I deleted was episode 149, where something got into the frame in the last few seconds, which left 149 episodes and 65,810 frames (66,268 before).
The one I kept was a field of glittery speckle across the work surface that built up gradually over two hours, so the data has it at every density and it would be there at rollout too. The model needs rollout to look like the dataset, not for either to look clean. (My explanation for where the sparkle came from got corrected in Script Measured.)
Evaluation Protocol
Every result in this post comes from the real arm, never from eval loss. I still log and report eval loss, and Loss Lied explains why it never counts as the score.
Before any rollout I wrote down the success criteria as a ladder, from L1 (picks up a block) to L5 (the full four-block wall), so I couldn't rationalise the scoring while watching a rollout. I recorded for the full task and evaluated up the ladder.
The rest of the protocol is a short list of rules, each explained in the results where it came from:
- Eval loss comes from a held-out tail of episodes (eight for v1), and I fixed a noise rule for reading it before the first reading. From v2b on, the tail is designed per source (Loss Lied, The Counterattack).
- A coherence gate on the real arm at the 2,000-step checkpoint, before spending the rest of the run (Loss Lied).
- Rollouts go through LeRobot's episodic strategy and get recorded to a rollout dataset, so I can rewatch every probe session (Designed Probes). Real-Time Chunking runs at the stock execution horizon of 10 until Chunk Seams and 20 after.
- Night conditions matched to the recordings, except for the daylight probes that taught me why (Script Measured).
- Batteries of 5 to 25 episodes, with the outcome branches written down before anything runs, staged from pencil dots or seeded cards. Only the first pick of an episode is scored (Designed Probes, Five Experiments).
- Grasps are scored by a wrist-verified hold rule that I repaired mid-project, so every absolute count from before the repair is a ceiling (Five Experiments).
- Pre-committed bars for the v3 gate: near-miss first-commit grasps at three of five, one-block completions at four of ten (The Fork).
Then there's the noise. I have 3 to 14 reps per condition, from one operator in one scene, so every number in this post is a direction, not a rate.
Training Recipe
All four runs fine-tuned lerobot/pi05_base on the two 3090s with one recipe. Two parts of it are there because the first launches failed. --rename_map maps my front and wrist cameras onto the openpi names π0.5-base was pretrained on, because the run stops if the camera sets don't overlap. And the run can only fetch PaliGemma's tokenizer and config once you've accepted the licence gate on Google's repo8. The full story of those launches is in Three Launches.
This is the recipe, with only the flags that taught me something:
NCCL_P2P_DISABLE=1 uv run accelerate launch \
--multi_gpu --num_processes=2 --mixed_precision=bf16 \
.venv/bin/lerobot-train \
--dataset.repo_id=shavvimal/jenga_wall \
--dataset.eval_split=0.05 \
--rename_map='{"observation.images.front": "observation.images.base_0_rgb", "observation.images.wrist": "observation.images.left_wrist_0_rgb"}' \
--policy.path=lerobot/pi05_base \
--policy.dtype=bfloat16 \
--policy.gradient_checkpointing=true \
--policy.train_expert_only=true \
--batch_size=4 --steps=20000 --save_freq=2000train_expert_only=true is the flag that matters most. It freezes the pretrained vision-language model and trains only the roughly 300-million-parameter flow-matching action expert9, because full fine-tuning doesn't fit in 48 GB of consumer VRAM. This flag comes back in Not the Architecture and again in Folklore Numbers.
The other two flags are there so the run starts at all: NCCL_P2P_DISABLE=1 because WSL2 has no GPU peer-to-peer and NCCL hangs without it, and dtype=bfloat16 because the weights default to float32, which runs a 24 GB card out of memory instantly.
The rest of the recipe, on the model side and the training side:
| Setting | Value |
|---|---|
| Action chunk | 50 actions at 30 Hz |
| Flow-matching steps | 10 |
| Image input | 224×224 |
| Normalisation | quantiles |
| Precision | bf16 |
| Batch | 8 (4 per GPU) |
| Schedule | cosine, 15k to 20k steps |
| Peak learning rate | 2.5e-5 |
| Weights trained | action expert only |
Three of those rows matter later in this post. The batch is small. The schedule is a cosine that floors at a tenth of the peak rate, and I never tested a longer run. And I train about 300 million of 3.3 billion weights.
Loss Lied
The first policy passed every check you can run on a loss curve, then failed the only check that mattered. Its best eval loss was 0.0848 on episodes it never trained on, and the curve fell the way the slides promise. The next morning the arm reached for a block that wasn't there, and nothing in the training logs predicted it.
The full 20,000 steps would take about eleven hours. At the 2000-step checkpoint, just over an hour in, my runbook had a gate: put the checkpoint on the real arm before spending the other 18,000 steps. That meant fetching it to the Mac first. The loss curve checks the model against the dataset, but only the arm checks the dataset against the task. At this point you're looking for coherence, not success. Does it move toward the block, descend near it, and do roughly the same thing twice? If it's coherent, let the run finish. If it flails, the problem is the dataset, and no number of steps will fix it.
Rollout needs --inference.type=rtc, Real-Time Chunking. Inference takes a few hundred milliseconds and the control loop wants a command every 33, so RTC generates the next action chunk while the current one executes.
With nothing but 65,810 frames of my own hands and an hour of gradient descent, the arm turned toward the pile, moved to a block, went down and picked it up.
It was grin-at-the-bench fun, and I want that on record as well as the fact, because later it'll just be a milestone. After five days of tape labels, voltage paranoia and fps forensics, I got to watch a machine do a thing I had done but never programmed.
There's no coordinate, no waypoint and no if statement anywhere in the chain. I moved my hands 150 times, and now the arm could do the motion without me. That grasp is the clip at the top of this post.
That same night, before bed, I also wrote down the caution: one grasp is one sample. I'd fixed the scoring rules in advance because one thrilling success is the easiest result to over-read. The gate's question was "is it coherent," and the answer I recorded was the boring one: the dataset teaches the task, let it finish. I wrote the excited version down next to it anyway.
The run finished clean, 20,000 steps in 11 hours 39 minutes with a final train loss of 0.064, and by mid-morning the arm had shown those numbers didn't mean much.
The eval curve flattened, and one uptick nearly fooled me. Loss on the held-out episodes fell steeply to step 5,000, crept down to about 0.085 by step 11,000, then sat in a 0.085 to 0.087 band to the end. At step 6,000 an uptick from 0.091 to 0.094 looked like the start of overfitting. I stuck to the rule I'd committed to in advance: one uptick on an eight-episode eval set is noise, and the next reading decides. Step 7,000 came in at a new best.
Small eval sets are noisy, so you need the rule before the reading. Otherwise one wiggle kills an eleven-hour run at 30%. The best checkpointed eval was step 16,000 at 0.0848, against the final 20,000 at 0.0862. I called the difference noise and said the arm would break the tie.
Dream Machines reached the same conclusion independently on a manufacturing task: their loss couldn't rank good checkpoints against bad ones either. Their validation error explained about a fifth of the variance in rig success across checkpoints10. Mine missed something bigger.
The curve that certified the script
Validation loss on the 8 held-out episodes, every 1,000 steps of the 20,000-step fine-tune. The curve fell, plateaued and picked a winner - but the held-out episodes came from the same script as the training set, so a policy that never looked could still ace them. Y-axis cropped to 0.08-0.12: loss curves do not baseline at zero.
The arm did break the tie, though not the way I meant: it picked fluently at a spot where there was no block. On the morning of day six I rolled out both finalist checkpoints, several attempts each. The arm drives confidently to roughly the same region every time, bottom-right of the pile zone, descends, closes the gripper, and comes up empty. It doesn't flail or freeze. It does a complete, fluent, well-executed pick at a spot where no block is. Sometimes the reach bends the right way, when a block happens to sit near its preferred spot.
And then the roadblock from the top of this post: after a block had been placed in the fence, the arm reached for the empty gap where that block used to be.
That reach is the giveaway, because degraded vision can't produce it. If the model's vision had just got worse, from the daylight or a moved camera, it would miss blocks noisily. It wouldn't go for one specific absent block at the exact position it used to be in.
Only one kind of system does that: one replaying a memorised script of pick positions. The script is indexed by how far through the wall the episode is, and it executes position N whether or not a block is there. It was a horrible hypothesis and also a clean one. The model wasn't using the cameras to choose where to pick.
Designed Probes
Three experiments, with outcome branches written down before anything ran, confirmed the hypothesis and located it. In the one-block probe the block moved across the whole zone and the grasp stayed at the same table coordinate. On the four-block battery the arm kept closing on a displaced block's old position with the block in plain view centimetres away. Everything downstream of a correct target worked. The failure was in one function: choosing where to pick.
First I had to hold back, because a fix you pick before confirming the diagnosis is just a guess. I was tempted to jump straight to recording corrective data. But more diverse data, DAgger and lighting control fix three different problems, so the first job was to find out which problem I had.
Even before any experiment, a structured set of questions did better than the frustration in my head. Four of them (conditions, tracking, failure phase, repeatability) turned "it doesn't pick up the block" into "confident descend-and-close at a wall-state-indexed position, under shifted lighting, repeatably".
My hunch at the bench was that order mattered. It was half wrong, and the wrong half was the useful one. It went like this: the model was trained on a fixed slot order, so picking the wrong block puts the table into a state the demos never contained, and everything after that is confusion.
The part that held: after any deviation, every later state really is out-of-distribution. That's compounding error, the oldest failure in behaviour cloning11, and it's why I only score the first pick of each episode (the rule is in Evaluation Protocol). After the first mistake you're measuring the knock-on effects, not the original fault.
The part that didn't: a fresh four-block scatter with an empty fence is as in-distribution as it gets, because every training wall started that way. Out-of-distribution drift can't explain the very first reach going to the wrong place. Something upstream of order was broken.
So I designed three experiments, all in night conditions with first-pick-only scoring. The one-block probe was the one that decides it: one block at five positions across the zone, so there's nothing to select. If the reach follows the block, vision works and selection is what's scripted. If it goes to the same table coordinate regardless, the pick itself is scripted.
The other two probes tested whether selection was consistent and whether the wall conditioned the pick. A first-pick census across five fresh scatters would separate "same location regardless" from "a consistent selection rule, just not the taught one". A wall-state probe, an identical scatter with the fence empty versus pre-loaded, would test whether the wall was conditioning the pick directly.
The rollout tool's episodic strategy loads the model once instead of paying a 90-second load per attempt, which cut a three-minute cycle to thirty seconds. It also records every probe session to a rollout dataset I can rewatch.
I still have the raw bench log from the battery, typos and all, and I'd rather quote it than polish it:
- 7: moved the block concerned-left : still moved bottom right. And grasped at
air. Did not follow block. Contineyues trying t9o grasp in that line bottom
right area.
- 8: Far left: goers to bottom right again. Ignored block in far left. Similar
to above,. Ocnties grasping at air in bottom right.
- 9: Far edge, top right. Further up than it used to be. Cibtrubes grasping in
bottomm right. Same error as above. Grasped for gfull rollout.
The chart shows the same thing as data: wherever the block was, every grasp attempt stayed in the habitual cluster. There's one row per probe episode, labelled with the block's actual staged position, and every attempt sits in the shaded cluster regardless.
The block moved; the reach didn’t
Every grasp attempt in the four one-block probe episodes, one row per episode. The row label says where the single block actually sat; the dots are the measured arm pose each time the gripper closed; the tick is the episode's mean. The block crossed the whole zone - the mean attempt pose moved 5.9 pan units, less than the within-episode jitter.
The four-block battery showed that the policy sometimes worked and sometimes didn't, which is why you probe with repeats. The same checkpoint produced a clean two-block episode and a full minute of grasping at air, hours apart. Only repeated trials per condition separate "the policy can't" from "the policy sometimes doesn't". Behaviour is a distribution, one rollout is one sample from it, and the first grasp I celebrated was one such sample.
There were clean successful picks: two episodes placed two blocks each. Whenever a block was actually in hand, placement into the fence was reliable, which is what the fence is for.
There was also a recurring fake opening move toward the top-left. An episode would open with the arm moving to the top-left and opening its gripper above nothing, then panning right to do its actual pick. That's a leftover of the closest-first opening I taught, done from habit rather than from what it saw.
Once a block moved, the arm kept grasping at where it used to be. Whenever the gripper clipped a block and moved it a few centimetres, the arm grasped at the block's old position for the rest of the rollout. It closed on empty air again and again, with the block in plain view. Episode 4 puts numbers on it: 17 grasp attempts across fifty seconds, seven of them consecutive inside an 8.4-unit neighbourhood at a rate of one every two seconds.
Once, after a long spell of empty grasping, the arm abruptly snapped to the displaced block and picked it cleanly, which suggests the visual pathway exists in some form. But when the scene was disturbed, the default was to keep going blind at a remembered coordinate.
Naming the error modes separated what broke from what worked. Pick targeting was broken: a zone-scripted patrol with weak visual capture. Re-localisation after a disturbance was absent.
Recognition that a grasp failed was absent too, and this one is on me. Because I discarded every fumble, the dataset had 149 perfect episodes and zero examples of noticing a miss or re-approaching a moved block. The policy grasps air without reacting because failure is outside its training distribution. If you remove every imperfect episode, you also remove every example of recovering.
Meanwhile placement was reliable, wall-state progression across multi-block episodes worked, grasp mechanics were clean whenever the target was real, and the home-and-reset loop was solid.
One caveat, since I keep checking my own diagnostics: a few one-block positions sat outside the pick zone, so those runs overstate the result slightly. The pile zone is defined as the right-side taped area (see The Task), so the far-left and centre-left placements in the one-block probe were partly outside the task's own scene definition. That makes them an out-of-distribution scene rather than just an unusual pick. The within-zone evidence is enough without them: empty-gap persistence, air-grasps at spots no block had ever occupied that run, and a displaced block ignored from centimetres away.
Script Measured
The script is in the data, and I could measure it to a fraction of a joint unit. The frames added a second, smaller cause on top, and a night re-probe showed how big it was.
I didn't want to rely on eyeballing, so the afternoon went to three parallel investigations over the primary data: the training parquet, the probe-rollout parquet, and the actual video frames. Every impression I put a number on either turned into a mechanism or turned out to be wrong.
The training set had the confound, and I measured it: the wall state alone predicts most of where I picked. Detecting the pick moment in all 149 demos is easy, because a gripper closing onto a block has a crisp signature in the gripper channel. The arm's pose at that instant gives 149 pick poses. I regressed them on nothing but the episode's position within its wall (first block, second, third, fourth), and that explains 51 to 66 percent of the variance in the reach joints: R² of 0.51 for shoulder pan, 0.64 for shoulder lift, 0.66 for elbow. The four pick indices sit at four distinct arm configurations.
Worse, my random scatter between walls wasn't random. Across roughly 38 walls, 84% of first picks landed in a lateral band with a standard deviation of 2.8 joint units, against 13.1 for picks overall. The fresh scatter between walls was my one source of diversity, and habit had flattened it. Closest-block-first, plus a human who scatters the same way every time, put the first block in nearly the same place wall after wall.
I believed I was scattering randomly, and the measurement says I wasn't. My data practically dictated the script.
The rollout replays the training mode, and I measured that too. All 94 grasp attempts across the ten probe episodes fall into two clusters. Nine sit at pan +39 and are harmless: the gripper re-closing at the fence after successful placements.
The other 85, including every attempt in the one-block probe, cluster at pan −41.0 ± 6.6, almost exactly on the training mode. The densest pick cluster in the training set sits at pan −41.6, so the cluster's mean sits 0.6 joint units from the training mode. In the one-block probe the block moved across the zone while the per-episode mean attempt pose moved 5.9 units, less than the jitter within a single episode.
The servo tracking error at attempt moments was 2 to 4 units, so the arm grasps where the policy points.
Here are the two clusters, plotted against the training distribution they were copied from:
The policy grasps the training mode, to within half a unit
Top: shoulder-pan at the pick moment for all 149 training demos, coloured by within-wall pick index. Bottom: the measured arm pose at every grasp attempt of the trained policy across the ten probe episodes. The dashed line is the training set's densest pick cluster; the policy grasps at it to within half a servo unit.
Putting the frames side by side showed I'd been wrong about the lighting. I had judged the morning's lighting as "similar enough, I checked the images", which is embarrassing to type. Training frames are night-lamp: hard directional light, deep cast shadows, saturated pink. Probe frames are daylight-flat: washed out, nearly shadowless, the block pale against a near-white surface.
The geometry was the same and the photometry was completely different. At the habitual grasp station the wrist camera saw pure featureless white, an image with no corrective signal in it even in principle.
My check was honest, but it was a human glancing at a live feed and comparing it against a memory, and memory isn't a reference image. You need both images side by side.
The frames also corrected my story about the sparkle field. In the probe frames it was gone. It's mostly how the surface renders under the scripted camera settings (a 6-millisecond manual exposure under one hard lamp), which amplifies the real pink paint dust the blocks shed, and ambient daylight removes it. My recording-night explanation, "paint dust spread by arm sweeps", was only partly right.
So I had to revise the diagnosis: two causes stack. The data offered a shortcut, and I measured that. The rollout also starved whatever vision the model had, because I trained it under one kind of lighting and showed it another. What I couldn't tell from the daylight probes alone was the ratio between the two.
A night re-probe settled it: daylight made things worse, but it wasn't the cause. I ran it that night with the blinds shut, the lamp on, the sparkle field checked against dataset frames, and one clean pink block sitting in the open. The policy gripped in the same bottom-right area and ignored the block. The script is what drives the policy.
Clever Hans
Once I had the measurements, the mechanism turned out to be ordinary. The training loss rewards the cheapest predictor of the expert's actions, and my dataset made the wall state a very cheap one. The likeliest account is that gradient descent starved the expensive predictor, which makes the failure almost inevitable rather than mysterious.
The training objective never asks the model to look. All it says is: given this input bundle, predict the expert's next 50 actions with minimal error. Nothing in it mentions looking, or blocks, or understanding. The images are one input channel among several, and the loss doesn't care which channel a correct prediction comes from. "Use vision" was something I hoped for, never something I enforced.
The front image has the causal answer, which is where the block actually is. But thanks to the lock list, the non-causal channels were excellent predictors too. The wall state alone recovers half to two-thirds of the pick-pose variance (the measured R²), and for first picks a near-constant is enough (the measured 84% in one band).
Now think like the optimiser: ignoring the pixels already gets a low loss. A predictor that outputs the average demonstrated pose for the current wall state gets there without looking. Using the pixels only buys the residual, and that's expensive to extract, because a 75 mm block covers a handful of pixels at 224×224 and pixels-to-pose is a hard function.
Gradient descent doesn't split the difference between a cheap feature and an expensive one. The cheap one wins and the expensive one starves12. Early in training the easy feature drops the loss fast, so the gradients concentrate there. Once the loss is low, the signal that would have built the vision pathway shrinks to almost nothing.
My best guess is that the visual pathway starves before it forms. It's a guess, because my probes measured behaviour, not the network's insides. If it's right, training ends with a lookup table and a vision system that mostly goes unused.
That also explains the thing that stung most, which is that the eval curve couldn't see it. The eight held-out episodes came from the same process, with the same slot order, the same closest-first rule and the same scattering habits. The context-to-pose correlation holds in them just the same, so the lookup table scores well on them.
A validation split is the same experiment with the answers hidden. It measures generalisation to new samples from the same distribution, and you only expose a shortcut by breaking the distribution.
The one-block probe broke it. Formally it was an intervention: it physically cut the correlation between context and block position, and the policy's behaviour showed which variable it had actually bound to, which was context. The falling eval loss had been certifying the script, not the looking.
There's a hundred-year-old name for this. Clever Hans was the horse that "did arithmetic" by reading his questioner's involuntary posture. He was right whenever the asker knew the answer and was reduced to guessing when they didn't, 89% versus 6% in Pfungst's 1907 experiments13. Posture was a cheaper feature than arithmetic, and it worked on the whole training distribution. My wall state played the part of the questioner's face, and the one-block probe was the first test where nobody in the room knew the answer.
The modern literature calls this shortcut learning14. In imitation learning specifically it's causal confusion: de Haan, Jayaraman and Levine showed that access to more information can yield worse imitation, because policies latch onto spurious correlates of the expert's actions15. The history-leaning variant is known as the copycat problem16.
What I can add from a bench in my living room is the measurement: both the spurious correlate and how strongly the policy bound to it, with numbers, in Script Measured.
Now look again at the lock list from Data Collection, the thing I was most pleased with that week, with the diagnosis in hand.
| Locked | What it taught the network |
|---|---|
| Fixed slot order | Place is inferable from the wall. By design; this one I wanted |
| Closest block first | Pick is inferable from the wall. By accident; this one I didn't |
| One scatter per wall | 38 layouts, thinly disguised repeats of one habit |
| Constant grasp, arc, home | Motion needs no perception to reproduce |
| One lamp, night only | Photometric overfit; daylight is another planet |
| Frame contains only task | Nothing in view but the correlations |
Every constant in that table was a confound I handed to the network. The fence was designed so that "where does the next block go" could be inferred from the wall state. That same property leaked into picking, the one function where inference wasn't supposed to be possible. Imitation learning can't tell your disciplines apart from the task.
The same channel was deliberate for placing and accidental for picking. So I got a policy that places competently with the fence's help and picks without looking.
Not the Architecture
This is a data failure, not a model-size failure. π0.5 fine-tuned on 149 episodes is still behaviour cloning, my recipe froze everything that might have protected it, and the pretraining supplied the competence that still works. A colleague challenged me on this, fairly: isn't this an ACT problem, a small-model problem? π0.5 is a multi-billion-parameter pretrained generalist, so surely it was built not to do this.
The colleague is half right, because the pretraining does help. A from-scratch ACT on these 149 episodes would do worse, with no prior and everything memorised. The one SO-101 benchmark that measures recovery behaviour puts ACT at 6.45% versus π0.5's 30.77%2. You can also see the pretraining in what still works on my bench. The clean grasp mechanics, the reliable placement and the smooth trajectories are largely inherited motor competence.
The other half is wrong in three ways, plus one thing to clear up first. The π0.5 that anyone fine-tunes at home isn't the full π0.5 system from the paper. The LeRobot port ships the flow-matching action head, and the subtask-prediction and reinforcement-learning components were never released upstream17.
First, fine-tuning π0.5 is still behaviour cloning. Flow matching is supervised regression onto the expert's actions (noise-corrupted target, predicted velocity, mean-squared error). Causal confusion comes from the objective when the inputs are correlated, not from the network being small. Pretraining changes the starting point, not the incentive.
Second, my recipe made it more BC-like than the name suggests. To fit in 48 GB, train_expert_only froze the pretrained VLM, vision tower and language model, so the only part that learned was the 300-million-parameter action expert. It trained on 149 episodes and was free to key on whatever predicted the targets. In effect it was a large behaviour-cloning head on frozen features, with the generalist's protection switched off on purpose.
Third, I'm not borrowing this argument from other people's papers. I reproduced and measured it on this exact model, in matched conditions, with a physical intervention.
The irony is that my lock list stripped out the diversity π0.5 is built on. Its paper is about open-world generalisation through co-training on massively heterogeneous data9, and diversity at scale is Physical Intelligence's answer to exactly this kind of failure. A 149-episode fine-tune on one scene with a fixed script is the regime their diversity is meant to get out of, and my lock list put the model straight back into it.
I gave π0.5 a distribution where the best move was to stop looking, and that's what it did.
I rejected three fixes before starting the real one. A better camera was out, because new optics would orphan all 149 episodes. Re-recording in a perfect lightbox was out, because one perfect condition is still one condition. And prompt engineering at rollout was out, because the dataset has one task string, so prompt-conditioned behaviour was never trained.
The most tempting non-fix was to narrow the world to fit the model. Night-matching the lighting and staging blocks where the script likes them would make demos work today. But that hides the failure instead of fixing it, and only new data widens the model to fit the world. Both are legitimate things to do. Confusing them is how a demo-ready policy gets mistaken for a general one.
The Counterattack
The fix was to make the shortcut stop working in the data, and by midnight the new data was recorded, merged and training. The diagnosis had landed that afternoon, and the fix follows directly from the mechanism in Clever Hans: if context stops predicting the pick, looking becomes the cheapest way to reduce the loss. Diversity here means decorrelation rather than coverage18. Twenty high-entropy episodes attack the R², while two hundred more habitual ones would feed it.
So I took my hands out of the loop, because the thing I'd measured as the problem was my own hands. 84% of first picks fell in one band while I sincerely believed I was scattering randomly. A script generated 36 randomised four-block layouts plus six one-block probe positions, non-overlapping and seeded so they're reproducible. I staged each wall from a card instead of from habit.
I kept the picking order fixed on purpose. The design rule is to randomise what the policy must perceive and keep constant what it must execute. Averaging happens when one input maps to many outputs, so a random picking order would teach mush. Closest-block-first over a random scatter keeps the correct pick a deterministic function of the scene, one answer per input, while the card entropy makes that answer unpredictable from context.
The recording session gave me 116 clean entropy episodes across two sittings, nearly triple the 30 to 40 I'd planned, with everything else as per the original lock list. The day also had a camera incident, a stray dataset and two merge traps. I caught all of them before they did damage, and they're in Data Hygiene Catches.
The second half of the fix is DAgger11, because plain behaviour cloning never sees its own mistakes. It trains only on the expert's states. Every frame of my 149 demos is a scene where my hands had already done everything right so far. At rollout the policy is driving, and it visits states no demonstration ever contained (arm parked over empty table, gripper closed on air). It has no supervision there, so errors compound.
Ross, Gordon and Bagnell's fix is to collect data from the learner's own state distribution and have the expert label it. You run the policy, let it get itself into trouble, then take over and demonstrate the right continuation from there. At the bench it's simpler than the maths. The dataset can only contain "what to do when you've grasped air at the wrong station" if the policy grasps air at the wrong station while an expert is holding the leader.
DAgger also directly undoes my own curation. Discarding every fumble deleted every recovery from the demos, and DAgger is how you put recovery back in without letting the fumbles back in.
I ran two rounds of fifteen corrections each, against the trained policy in night conditions. The rule that matters is to let the policy commit to the wrong reach before you grab the leader, because the whole value of a correction is that it starts from the failure state. These corrections came from v1 and went into the merge as recorded, which Naive Mixing comes back to.
The final merge came to 140,672 frames and 294 episodes, and just over half of v2 is anti-script material. The sum is exact: 65,810 plus 23,790 plus 32,321 plus 9,560 plus 9,191.
The retrain launched with two changes from v1, each traceable to a measurement:
| Change | Earned by |
|---|---|
| Image augmentation on | v1's photometric brittleness; daylight collapsed it |
| 15,000 steps, not 20,000 | The measured plateau after 11,000 |
Two hours in, I found that the entire eval set was correction episodes. Three suspiciously flat eval readings, 0.150, 0.151, 0.150, made me ask what the eval set actually was. The splitter holds out the last five percent of episodes, and the merge order had put DAgger round two at the end.
That was worse than unrepresentative, because it wasted the one chance to make eval loss see the script failure. The v1 post-mortem had shown that eval blindness comes from the eval data sharing the training distribution's confounds. With entropy episodes in the eval tail, eval loss would be sensitive to the script failure for the first time. The eval set is a design decision, not something you get from concatenation order.
So I carved a val tail from each source (seven originals, five entropy, three corrections) and re-merged with the mixed tail last. The re-carve also caught one bad 120-second episode, and cutting it took the set to 293 episodes. That corrected merge is v2b.
Twin Gaps
Before the v2b run finished, I pre-registered the test: the v2 checkpoint would face the identical one-block probe battery, scored first-pick-only. The baseline was 85 of 94 grasps at the −41 station and a 5.9-unit attempt spread across a block that moved anywhere it liked. If the fix worked, those two numbers would move and the reach would start following the block.
If the script survived, the escalation ladder was already written down: more DAgger rounds, then unfreezing the VLM's fusion layers, then re-recording with a tighter front camera.
The run finished clean, 15,000 steps in 9 hours 26 minutes. I pulled both runs' full metric histories down as CSVs and analysed them the same way as the probe data.
Three findings matter, and one of them backs up with data a claim I'd only argued from theory.
First, the new eval curve fell, and unlike v1's, the fall means something. The mixed eval set (seven original episodes, five entropy, three corrections, designed rather than inherited) went from 0.1345 at step 1,000 through 0.1057 at 5,000 to a minimum of 0.1006 at step 12,000, and closed at 0.1021. These numbers only fall if the model improves on scatters where context predicts nothing. The best checkpoint by eval was 012000. A couple of early bounces were bigger than v1's wobbles, which is what you'd expect from a harder, more mixed eval set.
Second, v2b's training loss runs well above v1's, which I've started calling the augmentation tax. At matched steps on the smoothed curves, it's 0.088 against 0.078 at step 5,000 and 0.077 against 0.067 at 15,000. That's about 13 percent, widening to 15. The gap keeps growing as training goes on, and a day earlier I'd have read that comparison as a regression.
With the diagnosis in hand, the higher loss is the fix working. Colour jitter makes photometric memorisation unprofitable, and the entropy scatters make positional memorisation unprofitable. v1's smooth grind toward 0.064 over two and a half passes of the same data was the shortcut being learned.
The augmentation tax
Both runs on one chart: smoothed train loss (lines, centred ±200-step rolling mean sampled every 500 steps) with each run's eval checkpoints overlaid (rings). v2b - colour jitter, entropy scatters, DAgger corrections, and a mixed eval set - trains 13-16% worse at matched steps and the gap widens, because the augmentation made the positional shortcut unlearnable. Y-axis cropped to 0.06-0.14: loss curves do not baseline at zero.
Third, the train-eval gap is the same for v1, the policy that grasped empty air, and for v2b. The day before, in Clever Hans, I had argued from mechanism alone that v1's eval was structurally blind to the script failure, because the held-out episodes shared the confound. The histories now show it with a number: v1's final train-eval gap was −0.023, and v2b's was −0.025.
Those are indistinguishable, and one of the two runs produced a policy that grasps empty air. I applied the standard overfitting check, watch the train-eval gap, properly to v1. It showed nothing, and the policy was broken anyway. Gap monitoring detects sample-level memorisation. It can't see distribution-level confounds, and I've now measured that. Only the composition of the eval set, or an intervention on the real robot, can see those.
Twin gaps
Train-eval gap at each eval boundary for both runs: smoothed train loss minus eval loss, so negative means the train curve sits below eval - the ordinary direction, in both runs. v1's gap certified a policy that grasps empty air; v2b's near-identical gap sits on a policy trained against the shortcut. The number cannot tell them apart.
v2b didn't overfit, and the histories show that clearly too. Its gap is mild and stable, and the run completed 0.92 epochs, so the model never saw any frame twice, which rules out classic memorisation by construction. The eval flattening after 12,000 is learning-rate and capacity saturation, not overfitting.
More steps wouldn't have helped under this schedule. The slope over the last five eval points is +0.00006 ± 0.0009 per thousand steps, statistically zero. The caveat is that the cosine had already decayed the learning rate to almost nothing by then, and a longer run with a stretched schedule is a different experiment that I didn't run. The run-health numbers are in Training Run Health. The infrastructure, at least, wasn't the problem any more.
None of these curves can tell me whether any of it changed what the arm does. A falling eval is necessary but not sufficient, which is what day six taught me. The answer still had to come from the probe battery I pre-registered above, now run on checkpoint 012000. I ran it the next morning.
Veto First
v2b now looks before it commits. It aborts over empty table instead of closing blind, and under the morning's detector it found a real block in every episode. Fourteen episodes went into a fresh rollout dataset, with mixed wall states, pre-loaded fences and one to three pile blocks at varied positions. Thirteen ran to the 60-second ceiling and one was a clean 14-second early-out.
I rebuilt the analysis pipeline and re-ran it on day six's probe dataset first, so one detector scores both sides of this before/after comparison. The rebuilt detector finds 137 v1 close events (132 of them on the pick side, which is what the chart below counts) where the original found 94, so only same-detector comparisons are valid.
The gripper stopped closing on nothing
Every pick-side gripper close in the two probe batteries, split by what the close actually was: a full close on empty air, an aborted close (descend, check the wrist view, refuse, move on), or a close with the block in the jaws. Both batteries were re-scored with the same rebuilt wrist-gate detector, so the bars are measured with a matched instrument.
Two numbers aren't in the chart. The longest blind-grasp streak fell from eighteen consecutive empty closes to six. And where v1 got at least one real block in eight of its ten episodes, v2b went fourteen for fourteen.
The aborts are the row that changes the picture. Seventy-three times, v2b descended somewhere, part-closed the gripper, reopened it and left, which v1 barely ever did. To understand them, I pulled the wrist-camera frame at every close event and measured how much pink block was actually in view.
v2b's aborts happen at under 1% pink. It looks down, sees nothing, and doesn't commit. My bench note from the first rollout turned out to be right: "it opens the gripper in the wrong place, descends, but doesn't just close, I assume it sees there's no block, then ascends again". That's the fix working. The entropy data made context worthless as a predictor, and the close decision now uses the pixels.
The veto fires on nothing; the misses have the block in view
How much pink the wrist camera saw at the moment of every close, by close type, v1 vs v2b - both batteries scored with the same rebuilt detector. Aborts pile at ~0% pink in both runs: in v2b that is the veto working (nothing in the jaws, so refuse). v1's full closes on air were blind - median 0.2% pink; v2b's happen with the block in frame - median 14.3%. The vertical spread within each row is dodge only; the pink fraction is the horizontal position.
The same frames show that v2b's remaining air-closes are near-misses, not blind grasps. v1's closes on empty table happened at 0.2% median pink, with nothing in view. v2b's remaining air-closes happen at 14.3% median pink. The block is right there in the frame, and the gripper misses it by a centimetre or two.
So the script is going away, and three new failure modes have taken its place. The first is biased search. Vision can veto a close, but it doesn't choose where to go. The policy still proposes grasp locations from the old bottom-right prior and patrols it (descend, check, abort, shift one station over). That patrol is where the long time-to-first-grasp tail comes from: median 11 seconds, with worst cases of 25, 42 and 60.
It's no mystery where the prior comes from. v1's confounded walls are still the largest single block of v2b's training data, just under half its frames, so the old spatial density still dominates where the policy proposes to go. Context no longer decides the close, but the habit of where to search is still there.
The second is the terminal near-miss. When v2b commits, the last three centimetres are coarse, because nothing in the data ever taught fine correction. Discarding every fumble guaranteed the demos only contain single clean approaches. The DAgger corrections demonstrated coarse redirection to the right block, never millimetric centring.
The third, provisionally, is that placement quality has slipped. Several transports ended with a block tilted against the fence rather than flat inside it, and placement had been v1's one clear strength. It's a small sample, and it gets its own probe before I call it real.
Overall, the causal-confusion fix did what the theory said it would. The confound stopped being predictive, and the policy started looking. What's left is a competence gap rather than a binding gap.
Search is inefficient and the final approach is imprecise, but every episode now finds a real block. Grasps land across the whole zone instead of at one station (pan spread −33 ± 12, the full width of the zone, where v1 parked at its one station). Recovery from a miss takes seconds rather than the rest of the rollout.
The next battery is already designed, with one experiment per failure mode. The first is my favourite test of the whole project: stage no blocks at all and run the policy over an empty table. With nothing to see, every descent it chooses shows the pure motor prior, so I get the residual script directly. And if the policy never commits a close over an empty zone, that proves the veto at the same time. It takes five minutes and needs no staging precision.
The rest of the battery follows the failure modes. The in-zone one-block probe gets rescored with graded metrics, time-to-acquire and path length, because a binary follows-or-doesn't score saturates on a half-cured policy. Then near-miss anatomy from the wrist frames: a consistent offset direction means a systematic error I can target, and pure scatter means a precision floor. A stale-target check with a mid-approach nudge and placement reps against v1's record complete the battery.
The fixes wait for those probes, but I can already see roughly what they'll be. Rebalance the dataset so the v1 majority stops being the prior. And add DAgger corrections in the search phase and at the moment of commitment, the two places the clean demonstrations never covered.
This is the only lesson I wrote down at the bench, and it's what I'd tell myself from seven days earlier. The model learned the task I rehearsed, not the task I intended. Eval loss can't tell you the difference, because held-out episodes inherit every confound of the process that generated them.
Every discipline that made my demonstrations consistent (and the evidence still says consistency was right) also made the world predictable enough to memorise. So the job, for me and maybe for you, is to keep the demonstrator consistent while making the scene unpredictable. Then the only strategy that survives training is one that looks.
The arm looks now, but only to check a spot it has already chosen. It doesn't look first.
Five Experiments
The first finding from the second battery was about my instrument, not the policy. The grasp detector was wrong in both directions, and I had to repair it. The battery itself was built to measure why the arm looks but doesn't look first, and it ran the same afternoon. The probe list had become a bench runsheet: a fixed 25-episode ledger, one launch command, and every staging position marked with pencil dots small enough to vanish at the model's 224×224 input resolution.
Night conditions held throughout, with the fence pre-loaded with two blocks so wall state stayed constant across every slot. Each experiment carried its decision rule from Veto First, written before anything rolled.
| Episodes | Experiment | The question |
|---|---|---|
| 0 and 1 | Empty zone, no blocks anywhere | What is the pure motor prior, and does the abort gate hold? |
| 2 to 11 | One block at five positions, twice each | Graded search: time to acquire, path cost, position dependence |
| 12 to 16 | Near miss, block at one fixed spot, five reps | Is the miss a consistent bias or scatter? |
| 17 to 19 | Nudge mid-approach, three reps | Does the ghost loop survive? |
| 20 to 24 | Placement, five reps | Did placement regress, or did I make the test harder? |
That was twenty-five episodes and 32,612 frames in one sitting.
The first thing the analysis found was a problem with the measuring instrument. Up to this point the grasp detector was a settle band: if the gripper channel settled between 8 and 16 units, roughly a block's width between the jaws, that counted as holding a block.
It's wrong in both directions. Jaw tips stalled against the table settle mid-band and score as a grasp, with visibly empty jaws in the wrist frame. A real grasp that then slips drives the channel below the band. So the battery's one perfect transport scored zero block-closes under the raw rule. A rule that credits the table and misses the battery's only success had to go.
The repaired definition requires residence in the band for at least 1.5 seconds plus a commanded close of 2.2 or less during the hold, checked against the wrist frame. Real grasps in this battery dwell 3.7 seconds or longer, so there's plenty of margin.
The repair also applies backwards, and it downgrades the morning's absolute counts. The morning's count of 37 "held" closes is an overcount and I've retired it. The before/after numbers in Veto First still stand, but only as same-ruler comparisons, which is what they were built to be. The 75%-to-27% shift in blind closes and the jump in aborts compare like with like, because one detector scored both datasets. But every absolute engagement count from the morning, including the fourteen-for-fourteen episode coverage, is now a ceiling until it's re-scored under the repaired rule.
A rig-health check found nothing wrong (the pan servo tracked at all 271 close events with a median error of 1.25 units, with no focus hunting or photometric drift), so whatever fails from here on is the policy.
The Confession
The empty-zone probe proved one of my own predictions wrong: what's left of the prior looks like the dataset's marginal, not v1's old station. This was my favourite probe of the project, and it told me more than I'd hoped. With nothing on the table, every descent was a sample from the prior, as designed. There were twenty-three of them, distributed at pan −34.7 ± 11.6.
In Veto First I wrote that v1's confounded walls, still the largest bloc of the training mix, were the strongest influence on the proposal distribution, and I expected this probe to show the old station. That's the claim the battery corrected. The descents are statistically indistinguishable from the training set's own pick distribution. A Kolmogorov-Smirnov test can't separate the two (p = 0.13).
v1's station cluster sits apart from both, at half the training spread. The patrol order isn't a fixed route either, because the two episodes' probe sequences are uncorrelated (ρ = 0.15).
An empty scene photographs the prior, and the prior is the training set
Shoulder-pan angle at every descent, in three samples measured the same way. v1's rollout parked its pick-side closes in a narrow band around -41. The training set's own pick poses are far wider. v2b's 23 descents over a zone staged with no blocks land on top of that training distribution, not on the old station: with nothing to see, the policy draws from its prior instead of replaying a script. The vertical spread within each row is dodge only; the pan angle is the horizontal position.
When the pixels offer nothing, the policy samples where picks happened in training, across the whole merged distribution, entropy scatters included. That's just what behaviour cloning does. It's a subtler and healthier leftover than the one I predicted. It also takes an item off the fix menu in Veto First before I'd started it, because there's no v1 majority left to rebalance away.
The rest of the battery's good news is quick to report. The ghost loop is gone. On day six the arm kept blindly going back to a remembered coordinate. Now, after a mid-approach nudge, 80% of the 41 subsequent closes executed at the block's new position. The first re-aim came within 2 to 5 seconds, and two of the three nudge episodes never revisited the ghost at all. v1 never managed that on the same test, not once.
The vision veto held over the empty zone, where 53% of closes were look-down-and-refuse aborts. And the near-miss battery caught the visual servo working once, end to end. That trace has its own figure in the next section.
So the machinery for looking exists at every stage. The rest of the battery answers the obvious question: if the machinery exists, why did the one-block battery score zero out of ten?
The Taxonomy
The battery produced six failure modes, each with a number attached. The bench's three provisional modes are still in there. The terminal near-miss and the placement slip survive intact, and biased search turned into the corrected prior story above. The battery adds what I couldn't see at the bench. The chart below is the whole verdict at a glance: what the retrain fixed on top and what still fails underneath. The details follow.
Four capabilities confirmed present, six failure modes still standing
Everything the 25-episode battery settled, split into what it confirmed v2b fixed and what is still failing. The ten claims share no axis, because each was earned at its own denominator: a dot per attempt, filled for the ones that went the right way. Open a row for the evidence behind its number.
Confirmed fixed4
Still failing6
The dominant mode is the premature ballistic commit, and its miss is a bias, which is the fixable kind. All four failed near-misses plateaued about 300 pixels off in the wrist view, three to four centimetres, and closed anyway. Three of them fired from a parking spot between pan −18.5 and −20.9, twenty units fence-side of the block. The misses share a direction, down-left, with a circular concentration of R = 0.91. That's a bias, not a precision floor.
The other sign is that the final pose still comes from the prior even when the target never moves. Terminal pan scatter at a fixed block (±11.3, sample) matches the whole-zone spread. The score at the policy's own comfort spot was one grasp in five.
Here are the same five approaches as wrist-view offset traces, right-aligned on the moment the jaws shut. Four plateau and one dives:
Four closes fired from a 300-pixel plateau, one servoed to zero
Distance from the block to the grasp point in the wrist camera over the final approach, for all five reps with the block staged at the same spot (pan -41.6). Each trace is the sampled offset the report lists for that episode, plotted right-aligned so every line ends at the instant the jaws shut. Four of them plateau between 250 and 400 px and close anyway. One closes the gap to zero and grasps, which is how we know the visual servo exists at all. Episode 16's block only entered the wrist view half a second before its close, so it has three samples rather than five.
The diving trace shows the servo works end to end. It's episode 15, going from 339 pixels to zero in its final two seconds, and it's the clean grasp in the montage. In the other four traces the policy just didn't use it.
The second mode is grasp non-consolidation: when a close does land on the block, it mostly fails to hold. Pinches of 0.1 to 0.8 seconds reopen in place and bulldoze the block around the zone. One episode closed on the same block seven times. Another pushed its block 165 pixels into the back wall.
The one-block battery's zero out of ten breaks down into three episodes that never engaged and six that pinched or pushed. The tenth did the whole task and dropped its block tilted against the fence at the last moment. Even at the comfort spot, the median time to a secured grasp was 17.9 seconds.
The third mode is a position cliff: the bottom-left staging spot got zero true contacts in 120 seconds of trying. The pan sweep crossed the block's azimuth repeatedly, and the policy vetoed twelve descents with the block in plain view. I have an untested suspect that would take five minutes to check. Under the locked 6-millisecond exposure, blocks in that corner render dark maroon rather than pink, and a colour-keyed gate would miss them. Swapping a bright block into that corner is already on the next runsheet.
The fourth mode is a leaky abort gate. Over a zone that definitely contained nothing, 28% of closes were still full blind closes. They included two three-second phantom grasp-and-holds (one of which carried its imaginary block at full elevation), plus one spurious return-to-home.
At the nudge test's stale coordinate the ghost veto fired only about half the time. That's a three-of-six split, too thin to trust, so the next battery has a five-rep nudge arm to firm it up. The blind-close motor prior is still there underneath the veto, and it shows up when the pixels stay empty long enough.
The fifth mode is occupancy-blind release, and its numbers come with a confound I built myself. Placement, flagged provisionally in Veto First, now has data. The release kinematics are identical to v1's and the demos', so the skill didn't regress.
But the demos encode four different release poses, one per wall state, and v2b never produced the right one for a two-block fence. Zero of five drops landed flat in a free slot, the release dwell halved against the demos, and two blocks were expelled while the arm was still moving.
The investigator that scored it also flagged the design flaw. v1 earned its reliable-placement reputation against an empty fence, and my pre-loaded fence made this battery strictly harder than anything v1 was ever scored on. I'll run an empty-fence rep before I call this a regression.
The sixth mode is a tic: after four of the five placements, the arm headed back toward the pile and air-closed on the way. It's the same leak as the fourth mode. There it shows up during search when a zone stays empty, and here it shows up mid-transit after a release. It's one dataset gap showing up in two task phases.
Hover Means Close
One sentence covers all six modes: v2b re-perceives, but it does not re-execute. The targeting inputs are visual now. The station is gone, the ghost loop is broken, and the aborts are gated on the wrist view.
The grasp execution policy (when to stop servoing and close, how deep to go, whether to hold) still comes from demonstrations in which every approach had already converged by the time the hand hovered. So the model learned that hovering near the pile means closing. Convergence was a correlate rather than a precondition, because no demo ever showed one without the other.
The dataset doesn't label any of the new states the policy reaches on its own: the unconverged hover, the post-pinch scene, the nudged block, the occupied fence. My discard-every-fumble rule guaranteed none of them appear. The first DAgger round doesn't cover them either, because its corrections were coarse redirections toward the right block, recorded against v1's failure states, and v1's failure states no longer exist. So the six modes mostly come from one property of the dataset. Gut Versus Data reassigns three of them to other causes.
Not Alone
Other people have hit this, and I found three kinds of evidence that same evening. The exact symptom, a π0.5 fine-tune that reaches the object and then keeps pushing it with small actions instead of grasping, is an open, unresolved issue on Physical Intelligence's own repository. It was reported on a different robot and a different action space19. Whatever this is, it isn't only my rig.
Penn's field evaluation of π0 explains why, from outside my bench: the model "does not have a metrically accurate method to determine the distance between the gripper and the surrounding environment", it is memoryless, and it has no sense of touch20. That's the commit-before-converge failure, described by people who have never seen this arm. The policy can't tell "3 centimetres short" from "there", so it closes on the schedule the demonstrations taught.
The ACT-on-SO-101 write-up I've already leaned on had the same miss one model class earlier and cured it with data, 25 episodes per position bin, ending at 90% success in-distribution and 75% out21.
Two mechanistic notes from the openpi codebase show why the close behaves differently from the reach. Gripper actions are absolute while arm actions are deltas22, so the close is a different kind of signal, learned on its own statistics. And the close decision is made inside a 50-step action chunk planned against vision that can be over a second stale23. Both fit a policy that commits from a hover and stops updating.
The same fact rules out one folk remedy. A hard pause before the close in a demonstration doesn't give the model room to think. It becomes part of the chunk the model imitates. So demonstrations should be slow and smooth, never stop-and-go.
The Fork
I chose one more aimed behaviour-cloning round over jumping to reinforcement learning, but only behind a written gate. The question was why keep feeding data to behaviour cloning at all, and why not jump to reinforcement learning now.
The case for jumping is that the loss never contained the word success. Behaviour cloning matches actions. A close that misses by 3 centimetres and a close that grasps produce identical training signal whenever the joints agree with the demo, and nothing in any gradient of v1 or v2b ever punished an unconverged close. No amount of clean data changes that incentive.
LeRobot ships HIL-SERL, human-in-the-loop reinforcement learning where a reward classifier learns success from the cameras and the leader arm becomes the intervention interface24. Its documented grasp success is around 70% on this class of arm25.
But LeRobot's HIL-SERL doesn't fine-tune π0.5 at all. It trains a separate small policy for the grasp skill (A Smaller Student has the details), so "moving to HIL-SERL" is really a policy swap inside a hybrid. Maybe that's where this ends up anyway.
So it's a bigger move than "just use RL" suggests, and three facts argue for one more properly aimed behaviour-cloning round first. DAgger has never really been tried against this policy, because round 1's corrections answered v1's failure states and those states no longer exist. The measured miss is a bias rather than scatter, and the servo demonstrably works. And two nearly free experiments were still unrun.
The RTC execution horizon defaults to 10, and every probe so far ran at 10.
The plan was a horizon A/B first, then DAgger round 2. That meant eighty corrections across twenty seeded walls, aimed for the first time at states this policy actually visits. HIL-SERL would come in only if the gate below failed. A Smaller Student overturns that plan the next morning.
I wrote the gate down that night, before any retrain launched. I did it for the same reason I had a rule ready for the step-6,000 uptick in Loss Lied. After the v3 retrain, the reduced battery runs again, with the same detectors and the same ledger.
The bar is first-commit grasps on the near-miss test at three of five or better, and one-block completions at four of ten or better, against that day's one of five and zero of ten. If either number misses, the grasp skill goes to HIL-SERL with no more debate.
I set the bar at a desk, from six forensics reports, hours before any arm moved. The person watching the rollout doesn't get a vote.
Chunk Seams
A config parameter I'd dismissed as a non-fix turned out to be behind the second failure mode. The horizon A/B ran that same evening, three launches back to back: horizon 5 against horizon 20, plus a third at the stock 10 with the fence emptied. To explain the result I first need to explain how inference works here.
π0.5 doesn't emit one action at a time. It emits chunks of fifty actions at 30 Hz, about 1.7 seconds of motion. Generating the next chunk takes 633 milliseconds on my Apple-silicon rig, loop overhead included, as the rig-health sweep clocked it. That's nineteen control steps.
The arm can't freeze while it waits. The control loop wants a command every 33 milliseconds whether or not the model has one ready. So nineteen actions of the current chunk execute while the next one is being computed, and the new chunk arrives planned against a camera frame that is nineteen steps stale. The handover from one chunk to the next is the seam.
Physical Intelligence's answer to the handover problem is Real-Time Chunking26, and its core trick is borrowed from image diffusion. RTC treats generating the next chunk as an inpainting problem, which is what image models do to fill a hole so that its edges agree with the pixels around it. Part of the new chunk is declared already known, because it must agree with the plan the arm is executing right now, and the model inpaints the rest around that constraint.
execution_horizon is the length of the known region. The first steps of the new chunk are locked to the old plan, the next few are pulled toward it but allowed to drift, and everything after that is planned fresh.
And inference_delay is measured, not set. It's the count of old-chunk actions that physically execute during generation, and on this rig it's about nineteen. The paper sizes its frozen region to that delay by construction, while the LeRobot knob is a fixed number you choose.
If the horizon is below the delay, the new chunk is free to contradict things that already happened, the steps between h and nineteen. The seam then hands execution to a plan whose recent past never happened. If you draw the mask at 5, 10 and 20 against the measured delay, you can see the problem before any robot moves:
The horizon has to cover the delay
How Real-Time Chunking splices a new action chunk onto the one the arm is already executing. The mask pins the first execution_horizon steps of the new chunk to the plan in progress, blends, then lets the model plan freely. The number that decides everything is the inference delay: 633 ms of generation at 30 Hz is 19 steps that execute while the new chunk is still being made. Geometry computed from the config (chunk_size 50, delay 19 measured, horizons 5, 10 and 20), not sampled from a run.
where the 19 comes from
The arm cannot freeze while the next chunk is generated, so about 19 actions of the old chunk execute during those 633 ms. The new chunk arrives planned from a camera frame that is already 19 steps stale.
the mask over the new chunk, by horizon
h = 10 9 of the 19 executing steps unprotected. Pinned to 10 of the 19 executing steps, leaving 9 unprotected at every seam. This is the setting every rollout in this post has run.
So the stock default of 10 leaves nine of my nineteen executing steps unprotected. The default covers the delay on a rig that generates a chunk in under a third of a second, which is a reasonable rig to design for. Mine takes 633 milliseconds.
In practice the A/B worked as a latency-mismatch detector. execution_horizon has to be at least inference_delay, and mine was set to half of it, in every rollout this post has described.
One number in that arithmetic is still open. All of it uses the 633 milliseconds as measured on the arm at the time. 294 Milliseconds later clocked the model's compute at under half that off the arm, which would put the delay near nine steps and the stock horizon just over it. I don't know which of the two numbers the mask should use, and it makes the horizon-20 result in the next section harder to explain by the mask alone. I haven't resolved that.
Gut Versus Data
Horizon 20 finished the task five times out of six on a loaded fence, and horizon 5 produced no real grasps at all. This is the five in six from the top of the post, on v2b's checkpoint 012000 (the Hub repo holds the final 015000). My gut at the bench had said the opposite, that stock 10 felt best, and I had to separate two variables to see why my gut was wrong.
My gut had a reason. The best raw numbers came at horizon 10, when the de-confound battery grasped five of five at a median 7.7 seconds. But that battery had the empty fence. Once you separate the two variables, the result flips.
The controlled comparison used a loaded fence, the block at the near-miss battery's spot, sixty-second episodes, and one wrist-verified grasp gate scoring both arms. That evening's gate used a commanded-close threshold of 2.5, looser than the 2.2 the taxonomy settled on, applied equally to both arms. Horizon 5's one detector-credited event is an edge-poke on the wrist frame. It had no transports, only churn. Here are the results, with the confound underneath:
Horizon 20 completes the task; horizon 5 churns
The controlled comparison at the loaded fence: block staged at spot S, 60-second episodes, scored by the strict grasp gate (band-dwell at least 1.5 s, commanded close at most 2.5, wrist-verified). A dot per episode, filled for the ones that went the right way, because the three arms were not run at the same denominator. Horizon 5 and horizon 20 ran the same evening; the horizon 10 row is the loaded-fence baseline carried over from the earlier one-block and near-miss batteries, so treat it as context rather than a third arm.
An outlined dot is a grasp the detector credited but the wrist frame does not support: horizon 5 has exactly one, an edge-poke, which is why its honest score is zero.
the confoundboth arms at horizon 10
The best raw numbers of the whole day did arrive at horizon 10. They came from the de-confound battery, which had an empty fence.
Emptying the fence collapsed hover at the parking spot from 0.36 to 0.11 of frames and took grasps from 1 of 5 to 5 of 5. The fence, not the horizon, was what the bench gut was reading.
Horizon 20 succeeded slowly, and the empty-fence run's 7.7 seconds is mostly down to the fence, not the horizon. Horizon 20's grasps came at a 35-second median (horizon 5's one detector-timed event, at 40.5 seconds, was the edge-poke). Two of its five transports went into the already-loaded fence, which counts as finishing here because the task was to carry the block there. Within the controlled condition it isn't close, and horizon 5 is the worst setting I've tested in this project.
The mechanism follows from failure mode two: the pinch-reopen churn was partly a chunking artefact all along. At a short horizon, a close begun in one chunk gets overwritten by the next chunk. That chunk was planned from a hover-ish observation and is free, past its pin, to open the gripper again.
Horizon 20 doesn't stop the churn. The pinch count went from 34 to 38. What it does is let a close survive the churn. The close-and-hold now lasts longer than the replanning that used to overwrite it, which is the difference between bulldozing a block and carrying it.
Dream Machines found the same lever independently. On their rig, re-inferring immediately and blending the first 20 actions of each new chunk toward the old one took one frozen checkpoint from 76% to 93%, against 53% with RTC switched off10. They're different knobs on the same seam. On both rigs, a setting that never touched the weights decided how often the same checkpoint finished the task.
The demonstrations are still mostly to blame for the pinches, as Hover Means Close said. The demos are why the policy commits early, and the counts say the jaws still fire far too often. The horizon decides whether a committed close lasts long enough to consolidate.
The empty-fence battery also settled three taxonomy questions. First, the corner is a coverage hole, not a lighting problem. The bright block sat at the bottom-left spot untouched for a full sixty seconds, which rules out the photometric suspect. Only corrections recorded there will fill the hole.
Second, occupancy explained the drops-on-top but not the rest: even into an empty fence, only one of five placements landed flat. The short-dwell flick release is a problem in its own right, and DAgger has to teach it deliberately.
The third answer changes the story: the fence blocks may be what pulls the hover. With the fence emptied, hover time at the pan −20 parking spot dropped from 0.36 of frames to 0.11. That gives a candidate cause for the near-misses' down-left bias. The two pink blocks sitting in the fence were plausibly acting as visual distractors, pulling the hover fence-ward.
All of this goes straight into the plan. Every rollout from here runs at horizon 20, the DAgger session included, because corrections have to come from the behaviour distribution that will actually deploy. The longer commitment carries one risk: staler vision could slow the re-aim that the nudge test showed. The gate battery's nudge arm will measure that at the new setting.
A Smaller Student
The next morning I shelved the gate anyway. I parked DAgger round 2, the v3 retrain and the gate's battery, and went straight to HIL-SERL on the grasp primitive, as a cold decision made in writing with no fresh rollout behind it.
HIL-SERL doesn't train π0.5 at all. It trains a small blank policy from scratch, which I learned by reading the implementation before running it27. The learner is a blank SAC policy, a gaussian_actor of about 892,000 trainable parameters against π0.5's 3.3 billion. It acts in end-effector space at 10 Hz: Δx, Δy, Δz plus a three-way discrete gripper action (open, stay, close).
π0.5 is never in the loop. I keep it as the full-task policy for a later hybrid, where it does approach, transport and placement while the small network learns the one skill the big one keeps fumbling.
Why should 892,000 parameters trained from scratch fix what 3.3 billion fine-tuned ones couldn't? Because of the objective. A missed close finally produces a different gradient from a grasp. And the policy's own pinches, misses and recoveries become training data, which are the exact states my discard-every-fumble rule spent a week deleting.
Five parts make up the loop. Fifteen teleoperated demonstrations seed a replay buffer that stays roughly half of every SAC batch forever. A reward classifier watches cropped camera frames and ends episodes with reward 1.
An actor process on the Mac drives the arm, and a learner on the GPU box runs gradient steps continuously, connected to the actor by gRPC. The fifth part is me, hands over the keyboard, ready to take over whenever the policy needs it.
It's the keyboard because the leader-arm intervention mode is documented but not implemented, and my gamepad is invisible to macOS. So I flew the arm with arrow keys at about 9 centimetres per second. The fifteen demos came to 3,316 frames.
I designed the reward classifier's dataset rather than improvising it, because the classifier learns "success" from whatever I label, and I already had a taxonomy of the ways this system fools me. In twelve success episodes I held the block up and moved it around the zone after each success mark, so positive frames cover the space.
Three of those were deliberately awkward. One hovers over the block for three seconds before closing, planting negative frames seconds before a success. Two hold the grasped block low, just above the table.
Then came eight deliberate failures I call spoofers, one for each way the measured failure modes could fake a success: the ballistic close on empty air, the pinch-and-drop, the table-press, the bulldoze, and the phantom carry, which is closing on nothing and moving around the zone as if holding a block.
The recording that counted came to 19 usable episodes, 7,832 frames, 3,184 positive frames and seven clean spoofer negatives. The classifier trained 5,000 steps in about four minutes on one 3090, at roughly 99% accuracy.
Reward Hacking
Then it passed a bench smoke test, with the episode order deliberately inverted. Both known spoofers ran first for a full minute with zero positive frames, then two real grasps both auto-terminated with reward 1, which I verified from the parquet. So as far as I'd tested it, the classifier was right.
The first real RL episodes showed reward hacking. Every one ended with reward 1.0 inside 2.5 to 4 seconds, and the jaws never touched a block. The classifier that had passed its spoofer test hours earlier was now handing out success for free.
I tested the suspects in order rather than on vibes. Apple-silicon numerical garbage wasn't it, because CPU and MPS probabilities were identical to three decimal places across sixteen frames. The reproduction test found the real mechanism.
The reproduction test showed that a static scene fires at random. I saved frames to disk and replayed static scenes. A block sitting under the hover with open jaws produced sixty seconds of clean zeros. Then the same static scene fired a single frame at 52.4 seconds, without the jaws having moved, because camera noise crossed the 0.7 threshold. Another fired 0.3 seconds into a post-drop episode.
So the cause was two things together. There's a borderline-probability zone when the block is directly under the jaws, and the rule let one single frame over threshold end an episode with reward 1. The policy didn't need to grasp. It only needed to hover where the classifier's probability was already high and wait for a noisy frame.
You can run the exploit yourself, and the debounce that defeats it:
One noisy frame ends an episode with reward 1
The reward classifier's probability over the 60-second reproduction that caught the exploit: a block on the table under open, motionless jaws. The logged events are real (clean zeros for 52 seconds, then one frame of camera noise crossing the 0.7 threshold at 52.4 s; genuine holds sustain p > 0.95); the curve between them is reconstructed for shape. Switch the scene and the termination rule, and the strip fades where the episode would have ended. Drawn at 4 samples per second for legibility; the rig's control loop runs at 10 Hz, so the three-frame debounce is about 0.3 seconds of dwell there.
Terminated at 52.5 s with reward 1. The jaws never moved; camera noise crossed the threshold for a single frame, and one frame is all the stock rule needs. Loitering pays.
The fix came in three parts: a debounce, hard negatives, and offline validation before I trusted the classifier again. I never moved the threshold, because the debounce and the data were enough.
With the debounce patch, it now takes three consecutive over-threshold frames to terminate. Real holds keep probabilities above 0.95 for many frames and noise spikes last one, so the difference between them is how long the signal lasts.
The hard negatives taught the classifier the difference, rather than just suppressing a region, and the offline validation showed it. The reproduction episodes went back in as about 1,970 hard negatives, which I merged into a second classifier dataset and retrained on. After that, failure-state frames dropped to probabilities at or below 0.0003. True positives got stronger, between 0.85 and 0.9996, where the first classifier's weakest true positive had sat at 0.59.
The reproduction test also ruled out correctness on Apple silicon as a cause of the original failure modes, and the same reach-then-fail-to-grasp signature is reported on CUDA rigs19, so the real problem was the objective rather than the hardware, with latency still a suspect.
With the retrained classifier installed, I started tutoring the policy, and I saw the next hole live on the arm: my own classifier recipe rewarded holding the block on the table. Two of the twelve success episodes had taught that holding the block low counts as success. That's defensible for a generic hold and wrong for a lift task, because it rewards gripping the block while it still rests on the table.
I watched the policy pinch without lifting, collect its reward, and do it more, and I heard myself say it out loud at the bench: it gets a 1, so it does more of it. It was textbook reward hacking, running at 10 Hz two feet away from me. I put the fix on the queue: grip-on-table negatives, classifier version three.
To be fair to the policy, it's not the only one in the flat doing this. Luffy worked out a while ago that sitting gets a 1, where a 1 is a piece of Babybel, and he has been sitting and staring ever since. The king of reward hacking:
The bigger thing I took from the day is that the first RL session worked more as an adversarial audit of the reward function than as a training run, with the policy as the auditor. As soon as the system had a real reward in it, a minutes-old 892,000-parameter network found that reward's weak spot in seconds.
Walking Away
Over the weekend I made the second cold decision of this project: I abandoned HIL-SERL. It didn't fail a gate, because the gate never ran. I just walked away from it.
No measurement forced that decision, and I want to be precise about that. The machinery worked. The actor and learner were talking across the tailnet, the classifier held up under adversarial testing, and the stack was stable after seven patches.
What decided it was the cost, and two bench days had made the cost very clear. Every training minute needs a human at the keyboard flying an end-effector at nine centimetres a second, and the reward function was already heading into its third design iteration. The specialist at the end of it would only ever do grasp-and-lift, and handing off to and from π0.5 is an integration nobody has shipped. So I went back to behaviour cloning, and the RL lane became a runbook instead of a working directory.
If that reads like the same mistake the gate was built to prevent, made again in the opposite direction, then partly, yes. I'd rather leave that open than pretend I have a verdict.
The same weekend produced one more experiment, a subtractive one. v3_no_dagger is the original demonstrations plus the entropy scatters, with all 29 DAgger correction episodes removed. The hypothesis was that the corrections had been poisoning the dataset rather than patching it.
The run went clean: 264 episodes, 118,322 frames, a twelve-episode eval tail and 20,000 steps, in twelve hours with zero crashes.
That run produced the best eval number this project had: 0.0842 at step 18,000. It fell to 0.0842 from 0.1229 earlier in the same run and was still improving, unlike v1's plateau at 11,000. v2b's best had been 0.1006, and v1's 0.0848. That number comes back in Folklore Numbers.
Read Our Mail
While the run cooled I watched a talk by David Watkins28 of the RAI Institute on what they learned building a data factory around Stanford's UMI, the handheld gripper that collects robot demonstrations without a robot29.
The core of the talk was a data autopsy, and careful demonstrations lost to messy ones. RAI's own careful dataset did much worse than Stanford's, so they compared the two on four axes: episode length, jerk, trajectory shape and a principal-component projection. On every one, their care showed up as uniformity. His conclusion: "instead of being very careful and deliberate, it was almost better to have this diverse set of different explorations."
That goes directly against my lock list. So I ran their autopsy on my own data the same night.
My data came out uniform. Every episode was the same motion at the same speed. Episode-length coefficient of variation was 0.18, tempo variation across episodes 0.14, and jerk variation 0.10.
The number that stung most showed the entropy round hadn't changed the trajectories at all. The top two principal components explain 54.2% of the original demonstrations' trajectory variance, and 53.7% of the entropy round's. Statistically, that's the same homogeneity.
The entropy round randomised what the policy has to perceive and deliberately kept constant what it has to execute. So it fixed the perception distribution and left the trajectory distribution untouched. The design rule did what it said, and going by the talk that's only half a fix.
Then I ran the same script on the reference, Stanford's public cup-in-the-wild dataset30, which opened the talk. It ran on its 1,447 episodes the same way it ran on my 264. The task and embodiment differ, but the spread measures are unit-free.
Stanford's data is far more varied than mine on every axis but one. Their episode-length spread is nearly double mine. Their tempo spread is eight times mine, their smoothness spread thirty-five times, and their episodes range from 0.37 to 16.6 times mean tempo where mine sit between 0.55 and 1.4. The one axis where the two datasets are close is trajectory-shape concentration. Theirs sits at 43.6%, which is around what the new gate in One Block asks for.
Stanford's data has 8× my tempo spread and 35× my smoothness spread
The autopsy instruments run twice with the same code: on my 264-episode jenga dataset (30 Hz joint state, mean episode 14.9 s) and on UMI's public cup-in-the-wild dataset (1,447 episodes, 10 Hz end-effector state, mean 48.3 s). Different task and embodiment, but coefficients of variation are unit-free spread measures and the trajectory PCA uses the same 40-waypoint resampling on both. The tempo histograms are real per-episode data: each episode's mean speed divided by its dataset's mean, identical bins both sides. Bar pairs are scaled per row; read the multiplier.
my tempo distribution264 episodes · tempo range 0.55×-1.4×
UMI tempo distribution1447 episodes · tempo range 0.37×-16.58×
92% of my episodes sit within ±25% of the dataset’s mean tempo; UMI runs from 0.37× to 16.6×. And the one axis where the two are close is the tell: UMI’s trajectory PCA lands at 43.6%, which is what the one-block gate’s <45% bar is asking my next dataset to reach.
UMI’s Hugging Face export ships observation state only, so the gripper-interval instruments are excluded as artefacts; the four shown are computed from state on both sides. UMI’s cup task and handheld collection are not this task - the claim is about spread, not difficulty.
The lock list optimised for clean demos when what I was short of was demo coverage. Coverage turns out to be three separate budgets: positions (fixed in v2), trajectories (never attempted) and recoveries (almost absent, at 0.09 events per episode by the autopsy's count).
My discard-every-fumble rule deleted every recovery along with the fumbles. That means every failure state v2b later reached had no demonstration for it, because of my own rule.
Watkins made the same point from the other direction. Wild data holds things curated datasets never contain, because those are built from "things that we as researchers are so biased to try to collect". My discard rule is that bias, written down as a rule.
The audit turned "maybe we need diversity" into something I could measure. Its first run scored positions and approach directions diverse, and length, tempo, jerk and trajectory shape all homogeneous. The demo corpus is one motion, performed 252 times (the 264 episodes less the twelve-episode eval tail).
I noted three more points from the talk. The first is that these policies are memoryless, so the state of the task has to be readable from the pixels right now. His demo was a pouring policy that works with a translucent bottle and fails with an opaque one, because the policy can't know it's done.
My phantom carries and blind closes are the same thing. If the current observation can't answer a decision the policy needs to make, the demos are teaching it to guess.
The second was force. Handheld collection hides contact forces, and teleoperation captures them through a kind of cheat. Pushing on the world doesn't move the recorded pose. A teleoperator overdrives the inputs, the training targets are commanded positions, and so forceful interaction gets captured for free in the gap between commanded and achieved. Without it, he said, hidden contact "starts to alias the output of your policy".
My rig depends on that cheat, and nothing makes π0.5 pay attention to it. A commanded close of up to 2.2 with the jaws stalled on wood is the force signal, and my own grasp detector uses that same gap. The 0.1-to-0.8-second pinches are what grasping looks like when the policy ignores force.
The third was that the demonstrator is a distribution, so one operator's habits end up in the data as if they were physics, like my 84% of first picks from one band of the pile (Script Measured).
Naive Mixing
One part of the talk came with a paper attached, BRIDGE31 from the RAI Institute, and it made a prediction about my next result before I'd run it. Its workflow is mine with one important difference.
The difference is that BRIDGE never naively merges the base and support data. Like me, they collect a base dataset, train, evaluate to find the holes, and collect targeted support data aimed at those phases. Then they train the base policy, freeze it, train a separate support head, and train a lightweight router that switches between the heads at inference, with no manual phase labels.
On their NIST pulley task, naive mixing erased the base policy's success and the router raised it. The base policy alone scored 44%. Base plus support data, naively mixed, scored 0%. Routed, it scored 76%, against a teleop-only ceiling of 84%.
Across all three of their tasks, naive mixing was worse than not collecting the support data at all. Their failure analysis says mixing incompatible supervision distorts the learned action distribution even before the final contact phase. One task developed a systematic lateral transport bias purely from having two action modes superimposed.
Structurally, my 29 correction snippets are the naive-mix condition. Compare that lateral bias with v2b's near-miss signature: a consistent down-left bias with a circular concentration of R = 0.91. My snippets begin mid-failure, in states the demonstrations never visit, and they prescribe actions that the demonstrations contradict at visually similar states.
Worse, the merge that folded them in stripped the intervention flag as a schema cleanup. So I'd deleted the router's training signal on purpose and then spent a week wondering about poisoning.
The paper also warns that proprioceptive state can become a shortcut, but I didn't use its fix, or the state-dropout version I could have applied on the data side, because the shortcut I measured was visual, not proprioceptive.
So that weekend v3_no_dagger stopped being a hunch and became this paper's ablation row on a $500 arm. I wrote the prediction down first: take the unrouted corrections out and the base policy should recover.
There's a constructive lesson in their numbers too. The support data's content was never the problem, since 76% beats 44%. On their numbers, recovery data only hurts when it's unrouted.
Dream Machines got the opposite result independently: 240 intervention rollouts, merged without any router, took a policy trained on one hour of data from 28% to 88%10. Their corrections came from the policy they then retrained, on its own failure states.
Mine differed on both counts. They were 29 snippets aimed at v1's failure states, and the dataset they were merged into changed in the same round, as the entropy scatters arrived. So they corrected a policy that no longer existed by the time anything trained on them. With one result on my side, I can't say which of those differences explains the gap.
Folklore Numbers
The battery came back well short of its bar. v3_no_dagger had the project's best eval number and still failed its probes the way v2b had, so removing the corrections did not rescue the grasp (Last Run has the re-scored numbers). On this bench the ablation went against BRIDGE's prediction. The subtraction ruled the corrections out as the main cause, and the eval loss told me nothing at all.
That left the question under the whole project: why don't the published SO-101 numbers, 56 to 90% success, reproduce on the full wall task here? I found five answers in community writeups, the issue trackers and my own configs.
The first finding changes how I read everything else: task class, not execution, explains most of the gap. Every community number at or above 85% is a single object with a tolerant target or a fixed position. On the same hardware and recipes, multi-object sequential tasks run 40 to 55%, precision insertion 45 to 80%, and selection among distractors collapses to about 10%. The benchmark that measured this on this exact arm class puts π0.5's average across four tasks at 56%, with grasp instability in 91 to 100% of failed trials for every model family tested2.
My task stacks all three hard factors, and I found no documented case of multi-object flush placement with pure behaviour cloning on this class of arm. I had been benchmarking myself against numbers earned on a different difficulty class.
The published numbers were earned on a different difficulty class
Every community SO-101 result the research pass could pin down, grouped by which hard factor the task carries. Bars are reported success rates; where a row shows a range, the bar draws its midpoint (the 90/75 row draws in-distribution). Sources: Sherry Chen (Hugging Face blog), a Medium bottle-grasp writeup, the SmolVLA paper, the Hiroshima SO-101 benchmark, FiberTune, and ggando; expand a row for the method behind its number.
One object, tolerant target
no hard factorSmolVLA paper: cube into boxπ0 · 50 episodes100%
Single cube, pick-and-place. The same paper, same rig, reports 40% the moment one stacking step is added.
Hiroshima: pen transferπ0.5 · 100 episodes95%
Single pen, hand-to-container transfer; the friendliest task of the four-task benchmark whose overall π0.5 average is 56%.
Sherry Chen: block into containerACT · 150 episodes90 / 75 OOD
25 episodes per position bin, deliberate orientation variation, grasp-the-middle discipline: the recipe this post borrowed for its own v2 round.
Bottle grasp writeupπ0.5, delta actions · 100 episodes~87%
Single bottle at a FIXED position, grasp and lift. The strongest π0.5-on-SO-101 number found, and the one writeup that trained with relative (delta) actions. Medium post, no stable URL.
FiberTune: randomised blockπ0.5 · 90 episodes72.7%
Single block at randomised positions.
ggando: cube into bowlSmolVLA · 75 episodes60-80%
Cube into bowl inside a 10 cm workspace. Documents the grasping-at-air symptom, cured by shrinking the workspace and enforcing one grasp strategy.
Sequencing enters
+ multi-object, orderingHiroshima: packingπ0.5 · 100 episodes55%
Multi-object sequential packing, same rig as the 95% pen row.
SmolVLA paper: stackingπ0 · 50 episodes40%
The 100% cube task, with one stacking step added.
Precision enters
+ tight toleranceHiroshima: insertionπ0.5 · 100 episodes65%
Precision insertion; the class runs 45-80% across writeups.
Selection enters
+ choose among distractorsHiroshima: selective sortingπ0.5 · 100 episodes10%
Language-grounded selection with distractors: 0-10% across every policy tested. The benchmark also reports grasp instability in 91-100% of failed trials for all four model families.
My task: selection (pick from a pile) + sequencing (next free slot) + precision (flush placement). No documented pure-BC result exists on this class of arm for that combination. The bars above were never the right benchmark.
Two smaller corrections came with it: "my model is overfitting" was the wrong frame, and the community cure for my symptom was data, not architecture. The head-to-head comparisons I found put the 3-billion-parameter models above ACT on matched small datasets, and the SO-101 benchmark's own summary is that "stronger pretrained VLA policies generally outperform the imitation learning baseline"2. And the community writeup that documents my exact grasping-at-air symptom cured it with demo density per approach configuration32.
The second finding is mechanical, and the maintainers acknowledge it: the pretrained base model learned delta arm actions with an absolute gripper, while LeRobot fine-tunes default to absolute joint positions. The pull request that added the fix describes the absence as "causing a distribution mismatch when fine-tuning from OpenPI pretrained weights"33. All three of my training runs trained absolute, so every rollout in this post so far came from a model working against its own pretrained prior about what an action means.
Dream Machines ran this exact switch independently and found nothing: relative joint targets landed at 73% against absolute's 76%, inside their noise10. But their failures were in a handover between two arms. That null result doesn't test whether a convention mismatch costs anything in the last centimetre of a grasp, which is where mine fails.
The third finding is that normalisation squashes the gripper's close commands into a sliver of its range. I confirmed it in my own statistics file within the hour. Normalisation maps each action channel's 1st-to-99th percentile range onto a fixed interval, and my gripper's statistics are lopsided: quantiles of 0.80 and 31.6, while every commanded close in the dataset lives between 0.75 and 2.2. That squashes the whole range of grasps, the difference between a pinch, a hold and a squeeze, into the bottom 4.5% of the channel's normalised travel.
A loss that weights error uniformly across the channel's travel can barely see that. Three separate issues on the openpi tracker point to this mechanism for policies that reach fine and grasp badly34.
Finding four is a ceiling I can't lift from here. Expert-only fine-tuning is the only mode two 24 GB cards can run under this stack's data parallelism, and it has a documented quality gap against full fine-tuning. In one reported case it almost failed on a task full fine-tuning handled, despite showing lower training loss35. My 48 GB is two cards that each hold a full replica, so a full fine-tune never fit on my bench. The only way round it is a rented cloud GPU.
Time is the other untested ceiling. My runs were cosine schedules of 15,000 to 20,000 steps at an effective batch of 8, roughly 160,000 samples, and a longer schedule is one lever this project never tested.
Finding five is good news that looks like bad news: the 3-to-4-centimetre consistent miss is not a resolution floor. The resolution-limited regime starts at sub-centimetre precision36, and my miss is systematic, which means it has a cause I can fix. The pixel arithmetic is still tight, though. After letterboxing to the model's 224-pixel input, the 25-millimetre block spans about 9 to 14 pixels in the front view, roughly one vision-transformer patch.
One contradiction I've left open. The community's standard fix for premature closes is a shorter execution horizon37, and my controlled A/B in Gut Versus Data measured the opposite. Both can be true, because short horizons cause a different failure, the chunk-overwrite churn, on a rig that then measured nineteen steps of inference delay. Neither knob is the real fix. The real fix is supervision that ties closing to contact rather than to time.
One more autopsy number says uniform tempo feeds that time-based close. With tempo variation at 0.14, close timing is nearly deterministic from chunk position alone, so the model never needs to check convergence to know when to close. Tempo diversity would break the correlation between time and the close.
The plan that came out of this is ordered by information per hour, and each step is falsifiable. Diagnostics come before any retrain. Then I retrain on v3's own dataset with relative actions and zero new data, which tests the mismatch finding in isolation.
After that comes a task ladder: prove the stack on the community-validated difficulty class, one block and a tolerant target, before climbing back to the wall. It's a cheap positive control that separates "my pipeline is broken" from "my task is hard". Then comes the big re-record under new data discipline. Two things are explicitly not on the list: a third camera, whose input slot on this model is semantically a second wrist on a bimanual rig, and the state-dropout tricks from Naive Mixing.
Bottom Millimetre
Before the relative-actions retrain launched, the diagnostics showed the normalisation finding directly at the model's output. I loaded the failed v3 checkpoint and fed it six frames from its own training data, four where the demonstrator had commanded a close and two commanding open. Then I read the raw gripper outputs before un-normalisation, which are the numbers the model actually computes in.
The close outputs sat at the bottom of the normalised range, and the opens overshot the top. On the close frames the outputs read −0.987, −0.949, −0.929 and −0.964, with chunk minima down to −1.035. On the opens they ran from +0.49 to +1.29, with a maximum of +1.80.
The trained model is pinned against the bottom of its range. Think of normalisation as a volume knob calibrated to the dataset's typical range, with the 1st percentile mapped to −1 and the 99th to +1. Closes sit at and slightly beyond −1, and opens have to overshoot past +1 because real opens exceed the 99th-percentile calibration point. So every close depth is packed into the very bottom of the range, and the model outputs them all at about −1.
Every close the dataset contains lives in 4.5% of the channel
The scale the model actually computes gripper actions on. Quantile normalisation maps the channel's 1st percentile (raw 0.8) to −1 and its 99th (raw 31.6) to +1; every commanded close in the dataset (raw 0.75-2.2) is squashed into the bottom 4.5% of that rail's travel. The probe dots are the trained v3 checkpoint's real pre-unnormalisation outputs from six of its own training frames: four commanded closes and two opens, plus each side's chunk extreme (−1.035 and +1.80). Drag the raw command, or use the presets, to see where it lands.
↑ every close in the dataset (raw 0.75-2.2) → the bottom 4.5% of the rail
the trained model, probed on its own frames
Raw 1.60 → -0.948. Inside the close vocabulary: pinch, hold and squeeze (raw 0.75-2.2) all land between -1.003 and -0.909, the bottom 4.5% of the rail's travel. A loss that weights error uniformly across the rail barely sees mistakes in here.
That night I also checked the relative-actions transform three independent ways before I let it near a GPU, because enabling the flag against statistics computed in absolute space fails silently, with no runtime validation.
The retrain went live that afternoon with a launcher diff of one added line. I'd confirmed the gripper normalisation symptom at the model output, but I deliberately left that fix out, because with two fixes in one run I couldn't attribute the result to either.
294 Milliseconds
That evening I also built a remote CUDA inference engine, and the Mac it was meant to rescue turned out to be faster. I built it because of the 633 milliseconds of chunk generation that makes vision stale in Chunk Seams. With the RL lane closed, the two 3090s sat idle, so the obvious thing to try without new data was to serve π0.5 from CUDA across the tailnet and keep the arm on the Mac.
The stock remote-inference stack can't do real-time chunking or record episodes, so I built a remote RTC engine inside the rollout stack that swaps only the prepare-predict-postprocess block for a gRPC call.
The infrastructure worked, and the idea behind it didn't. The build passed 84 tests, made correct round-trips over the real tailnet, and its warmup exclusion worked as designed. But steady-state remote prediction took 900 to 1,390 milliseconds.
Compute was the biggest piece of that. Isolating the components accounted for about 750 milliseconds: 8 of round-trip, about 140 of payload transfer and about 600 of compute on the 3090. The compute cost is plausibly the WSL2 virtualisation tax on ten denoising steps' worth of small kernel launches. The rest of the remote latency was server-side overhead the isolation didn't capture.
Then the control settled it. The identical benchmark on the Mac's own silicon runs at 294 milliseconds. The M5 Max beats the WSL2-mediated 3090 by a factor of two before the wire cost is even added.
The remote path loses before the wire is even counted
Per-chunk π0.5 inference latency on the same checkpoint. Component isolation puts the remote path at about 748 ms (8 ms round-trip, ~140 ms payload transfer, ~600 ms compute on the 3090 through WSL2); the observed steady state is 900-1390 ms, the balance being server-side overhead the isolation didn't capture. The local Apple-silicon control runs the identical benchmark at 294 ms. The hollow bar is the 633 ms the rig-health sweep measured on the arm with the control loop included, a different measurement from the two benchmarked bars and the number the build was justified by.
This is also the number the warning at the top of the post said would be questioned. Off the arm, the same model computes a chunk in 294 milliseconds, against the 633 the rig-health sweep clocked on the arm with the loop included. The gap between those two numbers is the open question in Chunk Seams, and the remote path lost to both of them.
My case for the build rested on the 633, and a six-line benchmark that I could have run first cut the gap it was meant to close.
One Block
The first rung of the task ladder got its own runbook the same day, and it answers the autopsy point by point. The dataset is one-block: a single pink block appears in the pick zone, and the arm picks it and places it into the next free slot of a corner fence pre-loaded with zero to three bricks. It's about 200 episodes over two sittings.
The design removes the two hard factors that pull the community numbers down. With one block in the zone there's nothing to select. And the fence's preload count means the pixels always show which slot comes next, so every wall state gets covered without the policy ever executing a sequence.
Every episode starts the way the rollout will. I put the block into the scene from outside the camera frame during the unrecorded reset, because that's the exact gesture the rollout will use. The first recorded frame has to look like a block that just appeared, not one that was always there.
The lock list changes but doesn't go away. Grasp technique, task string and scene stay locked, and the path is unlocked. A seeded card per episode now varies block position across five bins with continuous jitter, fence preload, tempo, arc height and approach azimuth. I do each one smoothly, never stop-and-go, because chunked models learn pauses as policy.
Twenty-five episodes put recoveries back in, as demonstrations. Each has a recovery badge: nudge the block a few centimetres mid-approach with a stick, or half-close and re-grasp, then complete cleanly. They're recorded as full episodes from standard starts. That gives continuous expert trajectories through failure states rather than correction snippets from an alien state distribution, which is BRIDGE's constructive lesson from Naive Mixing.
My old rule for the first 150 demonstrations, discard every fumble, is now reversed: a wobble you recover from is data. I only discard an episode if it can't end in a correct place, not because it wasn't pretty.
The gate from The Fork is back too, and this time it sits in front of the training run, which is where this month's evidence says it belongs. Before any model trains on this dataset, the autopsy script scores it against pre-committed bars. Training doesn't launch until the data passes, whatever I feel like at the bench.
The bars are set well above the old data. Length variation must exceed 0.25, against the old data's 0.18. Tempo must exceed 0.20 against 0.14, and jerk 0.20 against 0.10. Principal-component concentration must fall below 45%, against the merged corpus's 48% (the 54.2% and 53.7% quoted in Read Our Mail are per-source runs of the same instrument). Recovery content must exceed 0.1 events per episode, against the old data's 0.09, which its own report glosses as "little recovery content".
Last Run
By this point the project had its working result: v2b at horizon 20 carrying a block into a loaded fence five times in six. The relative-actions retrain was an experiment to push the grasp hold further, and this is how far it got.
The retrain finished on 18 August after 12 hours 14 minutes, zero crashes and 1.42 epochs. Checkpoint 018000 went to the Hub as shavvimal/pi05_wall_v3_rel.
Its eval loss can't be compared with earlier runs. It read 0.0801 at step 1,000, 0.0585 at 10,000, a best of 0.0548 at 18,000 and 0.0562 at the finish. Those numbers aren't comparable to v3_no_dagger's 0.0842, because the regression target changed from absolute positions to deltas. And Loss Lied already showed how little this project's eval numbers say about the arm.
The battery was pre-registered as the same four launches v3_no_dagger faced, all at execution horizon 20: near-miss ×5, one-block ×10, nudge ×3 (cut back from the five-rep arm The Taxonomy had asked for) and empty-fence placement ×5. The bars were 3/5 and 4/10.
Relative actions narrowed the serving options. Local RTC supports them by re-anchoring the leftover chunk prefix onto the new state, but the sync engine and the remote server from 294 Milliseconds both reject them.
The battery launch failed on a servo-bus fault, with the whole follower bus silent while the USB serial side enumerated fine.
Two minutes later the bus answered again, and at 12:10 the cache got its only v3_rel episode: no grasp, in one sample. It ran 59.4 seconds with 38 commanded closes. The longest band residence with a commanded close at or below 2.2 lasted 0.2 seconds, against the 1.5 the repaired detector demands. The gripper never read below 0.9, and the pan swept from −55 to +39. Nothing more was recorded after that.
For comparison, I re-scored v3_no_dagger's near-miss battery of 17 August under the same repaired rule, and it fell short too. That battery was five episodes at the same spot. Commanded closes per episode ran 17, 25, 20, 33 and 11, and the longest qualifying holds were 0.5, 2.5, 5.4, 0.6 and 0.2 seconds.
Two of five contained a hold that cleared 1.5 seconds at some point in the minute. That's a looser score than the shelved gate's first-commit bar of three, and still short of it, and one episode was an early-out at 31 seconds. Those are the numbers behind the shortfall in Folklore Numbers.
The four models and the five wall datasets behind them are public on the Hub38, the three HIL-SERL datasets are still private, and Assets has the ledger.
So the relative-actions model has one rollout on record, and one episode is one sample. The battery that would answer the question is written down with its bars, and it hasn't run. The project's result is still v2b's five in six.
Takeaways
These lessons answer the three questions from the top of the post: how to record data the model can't shortcut (1 to 3), which signals to trust (4), and how to find out what is going wrong (5 and 6).
-
Randomising the scene fixes perception but doesn't diversify the demonstrated trajectories. If you randomise what the policy must perceive and keep constant what it must execute, the trajectory distribution stays untouched. My entropy round changed where the blocks sat and left every trajectory the same motion (Veto First, Read Our Mail).
-
Keep the recoveries. My discard-every-fumble rule left the dataset with no example of noticing a miss, so every state the retrained policy later failed in was one my own rule had left unlabelled (Designed Probes, Hover Means Close).
-
Teach the gripper to close on contact, not on time. Demonstrations that are already converged when the hand hovers teach the model that hovering means closing, and uniform tempo lets it time the close from chunk position alone (Hover Means Close, Folklore Numbers). Whether messier demonstrations are the cure is still open (Read Our Mail).
-
Don't trust eval loss to catch a shortcut the held-out episodes share too. Break the distribution on purpose, with an intervention on the arm or an eval tail built from a different generating process, before trusting a curve (Twin Gaps).
-
Find out what is going wrong with designed probes, not by watching rollouts. Each probe breaks one suspected correlate, like the one-block probe, which moved the block while the wall state stayed put (Designed Probes, Veto First, Five Experiments).
-
Treat the serving settings as an experiment variable, like the weights. A chunk-handover setting that never touched the weights took the same checkpoint from zero transports in six to five in six, and the A/B worked as a detector for a latency mismatch I hadn't known was there (Gut Versus Data).
-
Don't expect corrections collected for an earlier policy to help its replacement. Mine did nothing measurable here, merged or removed, which is not what BRIDGE's ladder predicted (Naive Mixing, Folklore Numbers). BRIDGE routes such corrections, and the alternative is to take them from the policy you then retrain. I tested neither.
-
Check the pretrained prior's action conventions before the first run: delta or absolute targets, and where the gripper's closes land in the normalised range (Bottom Millimetre). My one relative-actions rollout settles nothing about the first half.
Limitations
The obvious limitation is scale. Everything here is one task, one operator and one scene, with 3 to 14 reps per condition in batteries of up to 25 episodes and no confidence intervals. The five in six is one of those conditions, measured once, with one block at one spot and a looser grasp gate than the one the taxonomy later settled on.
The recipe never changed either. Every run was expert-only, never a full fine-tune or LoRA, and I didn't touch batch size, learning rate or schedule, so there's no hyperparameter ladder here. The rig has its own quirks too: two cameras at 640×480, and inference on Apple silicon with a latency of its own.
Some numbers come with caveats. The grasp detector was repaired mid-project, so absolute engagement counts from before Five Experiments are ceilings. The 294 ms benchmark is off-arm compute only, while the 19-step seam cadence was measured on the arm with the whole loop's overhead. So the staleness window the chunk-seam arithmetic assumed may be over-estimated by about half, and I don't know which number is right. The v3_rel result is one episode, and I abandoned the HIL-SERL lane before it produced a single checkpoint.
Besides the relative-actions battery that never ran, three questions stayed open. Would the gripper normalisation fix on its own change the close, given that I parked it on purpose to keep the relative-actions result attributable?
The other two are about a ceiling and a cause that nothing here has tested. Would a full fine-tune on a rented GPU close the expert-only gap? And is the bottom-left coverage hole only a data hole, now that the bright-block test ruled out the photometric explanation and nothing has filled the corner since?
What's Next
The next rung is the one-block dataset, and the order matters. Nothing trains on its 200 seeded episodes until the autopsy script passes the data against the bars in One Block. If a sitting fails a variation bar, the fix is a bolder tempo and arc contrast in the next sitting, not a re-record.
Recoveries go in as the 25 nudge and regrasp episodes, each a full episode from a standard start, rather than as snippets merged after the fact. Each is one continuous expert trajectory through a failure state, with no intervention flag for a merge to strip. It isn't the same as correcting the policy's own failures, and that difference becomes testable once a one-block policy exists to fail.
The ladder is also a positive control, in the difficulty class where the community numbers reach 85% and above (Folklore Numbers). A policy that fails there says my pipeline is broken, and one that succeeds says my task is hard.
The actual wall, with its pile and its flush placements, is at the top of the ladder. The next rung tests whether single-block transport holds across positions and fence states.
The Build Log
Appendix. The findings, takeaways and next steps all end above this line. Everything below is my bench log, dumped in full: how a box of parts became the rig, and every setup problem I hit on the way. It's here for anyone building an SO-101 or fine-tuning π0.5 with LeRobot themselves.
I kept all of it because the rig is the dataset, and most of what went wrong later was already visible here, in a servo fault nobody logs and a camera set up for the wrong country.
Twelve Motors
The kit from The Rig arrived as parts for two arms, not as arms. On the bench I had twelve servos in labelled boxes, a pile of grey PLA+ printed parts, two control boards, two power supplies, four table clamps and a webcam. The obvious move is to start screwing things together. I asked a different question first: do the motors need configuring before they go into the printed parts?
They do, and I got that from the order of the official guide39 rather than from any sentence in it. The headings run Configure the motors, then Clean Parts, then Joint 1. Motor configuration is the first half of assembly, and you don't touch a printed part until it's done.
The reason is that every servo ships with the same factory ID. To give them distinct IDs you talk to each motor individually, which means only one motor on the bus at a time. If two are connected when the write lands, both accept it and you get duplicates. The IDs live in each motor's EEPROM, so the assignment is permanent, and once the motors are buried in printed parts and daisy-chained, isolating one means tearing the arm down. People have disassembled finished arms over this.
The board in that photo was the other early surprise. I had two identical control boards and asked which one was the leader. That was the wrong question, because nothing about leader versus follower is stored on the board. It comes down to three things I control: which supply gets plugged in, which port I write down, and what I pass on the command line.
So I picked a board, put masking tape on it, and wrote LEADER. That piece of tape is a safety interlock, and the two power supplies below show why.
One detail on that board taught me something about documentation. The jumper block is labelled A UART ~ SERVO and B USB ~ SERVO, printed in silkscreen right next to the jumpers it describes. LeRobot's docs just say "set the jumpers to B" and leave you wondering what A was for, but the board itself tells you. A routes an external UART header to the servos for driving the arm from a Raspberry Pi, and B routes the USB port, which is what LeRobot uses. Documentation printed on the thing it documents can't go out of date.
There are two jumpers because serial is two wires, and one jumper on A with one on B leaves the link half-connected. Commands go out, replies never come back, and the tool times out in a way that looks just like a loose cable. The board's two 3-pin servo sockets both read D V G. The identical labels show they're wired in parallel, so there's no in and no out and either socket works. In the finished arm one takes the cable from the previous motor and the other passes to the next.
One phrase in the docs needs decoding: "connect MotorBus to your computer via USB and power" means both the USB-C cable and the separate barrel-jack supply. USB alone runs the board's serial chip and does nothing for the servo bus. That sets up a trap I only spotted later. lerobot-find-port works fine with the power disconnected, because the serial chip is USB-powered, so it succeeding tells you nothing about whether the barrel jack is in. The very next tool needs the servo to answer. If you go straight from one to the other you get a comms error, and the unplugged barrel jack is the last thing you'll think to check.
Errors Have Direction
The kit ships two supplies: 12 V for the follower and 5 V for the leader. My first instinct was that this had to be backwards. The leader is the arm you drag around by hand, surely it needs more drive?
It needs less, and that's by design. The follower has to hold a payload against gravity, so it gets 12 V motors and aggressive gearing. The leader only has to hold its own weight while staying easy to push around, so torque there is a liability. A stiff leader is exhausting to demonstrate with, and every demonstration you record is worse for it. The two arms are tuned in opposite directions, and that's where the voltage difference comes from.
In fact the leader's servos never produce torque at all. The first thing LeRobot's leader class does on connect is disable_torque(). Those six motors are only used as position sensors.
Then there's the number printed on the leader's servos: 7.4V, driven from a 5 V brick. I stared at that mismatch for a while, convinced one of the numbers had to be wrong. Neither is. A servo's printed voltage is a nominal figure, not a demand. It's the design centre where the torque and speed specs were measured (7.4 is two lithium cells in series).
Run below it and the motor is weaker and slower, and perfectly happy. Run above it and you're into thermal limits the part wasn't built for. Under-volting is safe and over-volting destroys the part. So when you're unsure which brick you're holding, use the smaller one.
Plugging the 12 V brick into the leader board puts 12 V across six 7.4 V servos simultaneously. LeRobot has no voltage awareness anywhere in its codebase, and by the time any code runs the damage is already done. On these kits the two barrel connectors are physically identical. The tape on the jacks is the entire safety system.
That callout applies to more than voltage. Coming from software, I assumed a system complains before it breaks. This one doesn't. There are three ways to destroy the kit and the software can't see any of them: the wrong supply (instant, silent), hot-plugging servo cables (the 3-pin connector guarantees nothing about which pin touches first, so signal can make contact before ground and push current through the logic side), and stalling a motor.
On that last one I read the source rather than the docs, and found that the follower's configure() writes protection limits inside a single branch: if motor == "gripper". That is the only branch. The other five joints run at factory torque limits with no software protection at all, and the servo's own overload protection fires only after two seconds blocked above 80% of stall torque40.
The community has cooked gripper servos this way, and the maintainer's diagnosis on the canonical issue is blunt about squeezing too hard through the leader5. With no feedback from the software, you have to rely on habits instead. That's the real adjustment in moving from code to hardware.
There was one more trap, a domestic one. The kit ships from the US and I'm in the UK, and that's two separate problems. Plug shape is trivial, a few pounds of adapters. Mains voltage is not. A 120 V-only supply on 230 V British mains goes bang, and no plug adapter protects you, because adapters change the shape of the pins and nothing else.
The 5 V brick's small print read 100-250V, universal input, so adapters were enough. That one line was the difference between three pounds of adapters and a step-down transformer living on the bench forever. The rest of the label is worth reading closely too. The 0.5A on my own 12 V block's input side is a draw, not a requirement, roughly 115 VA of headroom for a supply delivering 60 W, and the mains lead's 250V is an insulation rating.
A power supply has two independent compatibility questions, can it survive my mains and is it right for my load, and they're printed in different places on the same label. Clearing one feels like clearing both, and it isn't.
Tape and Ratios
Setting the IDs starts with finding the ports, and lerobot-find-port does less than it looks like. It can't identify the board, because a USB serial adapter doesn't announce what it's attached to. What it does is list every serial port, ask you to unplug the cable, and report which entry vanished.
Watching it run makes this obvious. It printed about 170 ports, nearly all of them macOS pseudo-terminals, and the one that mattered was /dev/tty.usbmodem5AE60833381, identified purely by disappearing. I wrote it on the tape under the word LEADER. It's a name you can trust, because macOS derives it from the board's USB serial number, not from which socket it's in. It survives reboots and, as Country Settings shows, a complete re-plumbing through a hub.
Then the loop. lerobot-setup-motors walks backwards down the arm, prompting for one motor at a time:
Connect the controller board to the 'gripper' motor only and press enter.
'gripper' motor id set to 6
Connect the controller board to the 'wrist_roll' motor only and press enter.
'wrist_roll' motor id set to 5
...
'shoulder_pan' motor id set to 1Six motors in eleven minutes, without a single error. The routine that kept it safe was: power out, unplug the 3-pin at the board end, plug in the next motor, power in, press Enter, write the new ID on the motor's tape. Pulling the barrel jack before every swap costs four seconds and avoids the documented board-killer of hot-swapping an inductive load. Unplugging only at the board end means each motor keeps its own cable forever, which matters later when some connectors end up buried inside printed parts.
While the leader's six motors went through the loop I made the one mistake of the day, and I'm including it because catching it was the most useful lesson of the build. I had told myself the leader's motors were all geared the same and so interchangeable, grab any box. That's wrong, and wrong in the dangerous direction.
The follower's six motors are uniform, 1/345 gearing. The leader is the mixed one: one 1/345, two 1/191 and three 1/147, matched to how much self-weight each joint carries. I caught it before anything was assembled, by reading the actual paragraph in the docs instead of trusting the summary in my head. If a 1/147 goes where the shoulder lift belongs, the leader can't hold its own arm up, and I'd have found that out at teleop, three days and one full assembly later.
And there's a second trap inside that one, in how the labels are printed. The leader's 1/147 and 1/191 motors have the plain red-and-yellow Feetech label. The leader's single 1/345, the one motor I only have one of, has the black-and-yellow enhanced-style label, the same style as the six follower motors. So if you sort by label style, which is the natural thing to do, you misfile the scarcest part.
The reliable discriminator is the part number: STS3215-12V is a follower motor, plain STS3215 is a leader motor.
The mixed gearing has one upside. Because the leader's ratios are printed on the motor bodies, every leader motor can be checked twice at assembly: the ID on its tape and the printed ratio must both agree with the joint it's going into. The follower doesn't have that. Its six motors are physically indistinguishable, so the tape is the only record of the invisible EEPROM state, and a mix-up would stay hidden until far too late.
Redundancy costs a bit up front and pays off when you need to catch an error.
Which joint gets which ID is never actually stated as a rule anywhere. The docs say "the first motor", "motor 3", inside each joint's steps and assume you'll infer the rest. Since the cost of inferring wrong was a teardown, I checked whether three independent parts of the system agreed: the joint sections name motors one through six in order, the axis table maps shoulder pan to 1 through gripper at 6, and the names setup-motors prompted with went shoulder_pan equals 1 down to gripper equals 6. All three describe the same anatomical order, base up, so ID N goes in joint N.
It's a boring answer, but I checked it rather than assumed it, and checking cost five minutes against a mistake I couldn't undo. When an error can't be undone, find independent descriptions of the same fact and confirm they agree.
The follower's ID loop offered one tempting shortcut, and it's worth recording why it was wrong. Setting IDs is only an EEPROM write, no torque involved, and the vendor says 5 V is acceptable for the follower arm generally, so why not run the follower's loop on the already-validated 5 V brick and postpone checking my own 12 V block?
Two reasons. The docs plainly say to use the follower's supply, and the board's own silkscreen reads DC 9~12.6V, which makes 12 V mid-range and 5 V below spec. The leader goes off-spec on purpose, because protecting its 7.4 V servos matters more than the board's comfort. The follower has no such conflict. Substituting would have moved away from the one configuration where every component agrees, just to dodge a check I'd have to do before teleop anyway.
Putting off a required check only moves it to a point where more is assembled and a failure costs more to diagnose. And when the plain instruction and the clever reading disagree, and the clever reading only buys convenience, go with the plain instruction.
One last question that day: should I configure the follower's motors now too, and then build both arms side by side? I split it. I batched the configuration, because the rig was set up and I had the routine down, so the follower's six cost eleven more minutes instead of a full context reload in three days.
But I built the arms one after the other, because the first arm is where you learn. I was going to get something wrong on the leader, a horn seated backwards or a screw driven one turn too far, and building both in parallel means making that mistake twice and finding it twice. My test for what to batch: if attempt two gets cheaper because of attempt one, don't run them at once.
Horns and Vocabulary
Assembly was the first place where following instructions and understanding them came apart. Every joint section starts with the same line: "Install both motor horns. Secure the top horn with a M3×6mm screw. No screws are required for the bottom horn." That's a complete instruction if you know what a motor horn is, and meaningless if you don't. I didn't.
A horn is the coupling between the motor and the thing it moves. The servo's output is a small brass shaft with fine teeth cut into it, called splines, and the horn is a disc whose centre bore has matching teeth. It grips the fragile shaft and gives you a wide flat face with screw holes so an arm segment can bolt on. A drill chuck, basically.
There are two per joint because each joint is braced on both sides like a hinge. The splined horn on the driven shaft transmits torque, and a smooth-bore horn sits on a passive pivot opposite, so the joint doesn't hang every gram of arm off one small brass shaft.
My first guess at which horn got the screw was backwards, and the correction didn't come from the manual. The servo has a threaded hole in the centre of its output shaft, and the passive pivot has no thread at all. The screw can only go in one place, so the splined bore goes on the output shaft and gets screwed, and the smooth bore goes on the passive side with a thumb push. I didn't need to look that one up, I just needed to look at the part.
Where the docs simply stop is hub orientation. Horns have a raised hub on one face and a flat other face, and across all ten mentions of "horn" in the SO-101 and SO-100 guides, not one says which way round the hub goes. I stopped searching and dry-fitted both ways before any screw went in. The correct orientation sits flush with no rocking and lets the M3 reach its thread. The wrong one gaps, rocks, or bottoms the screw head out early, and the difference is obvious once you've felt both.
When documentation says nothing about something physical, a five-second reversible experiment beats guessing, searching and asking. The part in your hand answers faster than any source, and it can't be out of date.
The rest of the mechanical build came down to a few routines. Horns go on before the motor goes into its housing, because afterwards the faces are enclosed and you'd be pushing a disc onto a shaft you can't reach. Cables get connected and routed as each motor seats, because the channels get covered by the next segment bolted over them. Screws into PLA+ go snug, then stop. The printed parts strip their threads if you keep turning, and a stripped motor mount means reprinting a part I can't reprint.
And then joint 5 breaks the pattern three ways at once: one horn instead of two, motor before horn instead of after, two screws instead of four. After four joints of repetition my hands were on autopilot, and that's exactly when the deviation arrives. I'd love to know if that's deliberate.
Three Manuals
The wrist camera mount is where the documentation ran out. I searched the whole official SO-101 assembly guide for the word "camera" and got zero hits. The guide takes you to the gripper and stops. Wrist cameras are all over LeRobot's policy docs as dataset keys and observation channels, but nothing anywhere tells you how to physically attach one.
The gap makes sense once you see who makes what. The arm is HuggingFace's, the mount is PartaBot's printed part, and the board's mounting spec is Waveshare's. Nobody is negligent, because each documents what they made. But a real build crosses all three, and the joins between them are where you get stuck. Nothing tells you which manual to open next, so you work it out from what the part is.
The camera mount answer turned out to be M3 hardware, longer than the kit's M3×6. I got that from PartaBot's build video, because it isn't written down anywhere. The board mount did have a published answer, just not from HuggingFace: Waveshare documents four 2.5 mm corner holes. And that number has a trap in it.
A 2.5 mm hole and an M2.5 screw sound like a match and aren't. An M2.5 screw is 2.5 mm across its threads, so a 2.5 mm hole gives you interference, not clearance. M2 is the right screw. Nominal sizes describe the screw, and the hole has to be bigger than the thing passing through it. It's obvious once stated, and easy to miss when you're matching two identical numbers at midnight.
Everything in the kit came with matched hardware, so "what size screw" never came up until the first part outside the kit's scope. As soon as you add anything the kit didn't anticipate, you have to solve the sizing problem the kit had been solving for you. It's worth knowing, because a wrist camera is the first thing most people add.
I nearly missed daisy-chaining the motors, because of where the docs put it. The wiring instruction (each motor's 3-pin cable to the next, motor 1 down to the board) is at the end of the motor-configuration section, 160 lines before you need it. Reading top to bottom, it comes across as a footnote to ID-setting rather than an assembly step. Without the chain, calibration talks to nothing. The intended routine, which the joint videos show even though the written steps don't, is to connect and route each cable as its motor seats, before the next segment gets bolted over the channel.
That webcam matters more than it looks. π0.5 is a vision-language-action model, so it takes images, not just joint angles. Every demonstration carries the camera streams, and the cameras' positions become part of what the model learns the task is. Move one between recording and inference and the policy is looking at a world it never saw.
The four table clamps are there for the same reason. The arms must not move relative to the table and the cameras, because a follower that shifts a centimetre mid-session makes every episode after the shift disagree with every episode before it about where things are. In robot learning, the setup is the dataset.
The second arm showed that building them one after the other was the right call. Joints 1 through 5 are identical on both arms, and the build only diverges at the last step, where the follower gets a claw and the leader gets a handle with a trigger. Everything I'd worked out slowly on the leader carried over: how hard to drive a screw before PLA+ complains, which way each part faces, when to route each cable. The second arm was fast because the first was slow.
Calibration Numbers
Calibration is the first point in the build where you get a number to check, after two days of tape, torque and eyeballs. Torque drops so the arm goes limp, you move every joint to the middle of its travel and press Enter, then you sweep each joint end to end while a live min/max table updates.
That middle matters more than it looks. It becomes the centre of the encoder's range, and if you set it near an extreme you permanently lose half that joint's usable travel. Each servo reports position as a count from 0 to 4095, so one full turn is 4096 steps, about 0.088 degrees per count.
The raw numbers look alarming at first. The follower's shoulder lift ran 628 to 3053 and the leader's ran 647 to 3163, hundreds of counts apart at both ends on supposedly identical arms. The reason is the splines. Each horn seats on its shaft at one of a fixed set of rotational positions, and nothing controls which one you happened to use. So the same physical pose lands on a different raw count on each arm. The homing offset is there to absorb this.
What assembly luck can't change is how far a joint travels, because both arms have the same elbow, the same shoulder, the same printed parts. So the span is the useful number, and the percentage difference between the two arms' spans, call it Δ, is the diagnostic.
| Joint | Follower span | Leader span | Δ |
|---|---|---|---|
shoulder_pan | 2708 | 2727 | 0.7% |
shoulder_lift | 2425 | 2516 | 3.6% |
elbow_flex | 2226 | 2215 | 0.5% |
wrist_flex | 2389 | 2444 | 2.3% |
wrist_roll | full turn | full turn | n/a |
gripper | 1518 | 1205 | 26.0% |
The part I like is that Δ checks my sweep, not the hardware. Two near-identical machines should produce near-identical spans. If they don't, the likely cause is that I swept one arm less thoroughly than the other and stopped short of a mechanical limit. Under five percent on every arm joint means both arms got swept to their real stops.
The gripper's 26% shows the check works, because it's the one joint where the two arms really are different mechanisms: the follower has a claw and the leader has a trigger. A 26% on the elbow would have sent me back to recalibrate. On the gripper it's the correct answer.
When a procedure has no ground truth, you can run it on two things that should agree and compare them. It cost nothing here, because both arms needed calibrating anyway.
Two calibration habits matter more than the sweep itself. The first is that the --robot.id you pass is permanent. It names the calibration file, and every later command (teleop, record, train, rollout) must pass the same id or it runs against a different arm geometry. That's a leading cause of "the policy worked in training and does nothing on the robot".
The second is that I backed up the calibration JSONs straight away. Once recording starts, a recalibration produces numbers that don't match the old ones, and every episode recorded before it quietly stops matching the arm. If the cache ever dies mid-project, you restore, you don't recalibrate.
While I had the numbers in hand, I also wrote down three figures that limit how precise any policy on this hardware can be. The encoder resolves 0.088 degrees per tick, but the firmware has a deadband of roughly ten ticks, call it 0.9 degrees, and a command smaller than that produces no motion at all. The geartrain adds about a degree of backlash, rising to two and a half under load, and it compounds across five serial joints. One degree of error at the shoulder is over 5 mm at the gripper. So whatever task this arm learns has to work with centimetre-scale placement.
I measured one apparent defect against those numbers and then left it alone on purpose. The calibration showed the follower's claw under-commanding by about 14 degrees at full trigger squeeze, and I nearly chased it with a recalibration. Then I put a real block in the jaws. 80% of gripper travel closes well past a 25 mm block, and it picked and placed fine in teleop. The object decides what "enough" is. Chasing 100% would have cost bench time for nothing.
I also went into the source to see where these numbers actually get used, and found I'd had it wrong. I'd assumed the spans act as a scale factor that maps leader range onto follower range. That's true in LeRobot's percentage modes, but this setup runs degrees mode, where the span is used once, to find the midpoint that becomes zero. So a sloppy sweep does different damage in each: percentage modes rescale motion, and degrees mode shifts the zero point.
Either way, nothing errors. You get a robot that moves confidently to slightly wrong positions, and every demonstration recorded afterwards inherits the error. It's wrong silently rather than broken loudly, which is the same kind of failure as everything from Loss Lied onwards.
Four Hours
In the first teleop, five joints tracked beautifully and wrist_flex did nothing. On a later run it slammed to a hard stop with the arm in the air. Then, with a safety cap applied, it walked steadily 181 degrees away from where it was commanded and froze. And then the whole follower bus went silent: a broadcast ping returned zero motors while the USB port still enumerated fine.
Four different symptoms over four hours of debugging, one cause, and the fix was "unplug it at the wall."
I worked out afterwards what had actually happened. I'd started the very first teleop with the two arms in mismatched poses, so the follower snapped to the leader's position at full torque and the wrist slammed into its hard stop. The servo latched an overload fault, which is the protection from its datasheet that unloads the output after two seconds blocked. That latch survived process restarts, reconnects and three full recalibrations, because LeRobot never toggles torque inside its loop, so nothing ever reset it.
Every attempt after that stalled the joint again and drew stall current. Eventually the cheap 12 V supply latched itself off from sustained overcurrent, the way cheap barrel supplies do, and it stayed off until I cut the mains side. A wall-side power cycle cleared both latches at once. The joint has tracked perfectly ever since.
The uncomfortable part is why it took four hours, and the reason was that I had too much data. I had calibration ranges, midpoints, span deltas and normalisation modes, and I found real oddities in all of them. Each was true and none was the cause. Calibration produces numbers, so I reasoned about calibration three times. Each time I got better agreement between the arms and fixed nothing.
Meanwhile the actual state was in a hardware register nobody logs. If you only look where you have data, you'll find something, but it won't be the cause.
Reading the source that night, I found something unsettling: LeRobot can't observe servo faults at all. Register 65, Status, is a read-only hardware-error bitmask with bits for voltage, temperature, current and overload. It's defined in LeRobot's own control table and nothing in the codebase ever reads it.
It goes deeper. Sync writes go out with no status packet requested, so there's no way to see a motor rejecting a goal. The per-motor error byte that does come back on reads gets discarded inside the Feetech SDK before LeRobot ever sees it. A servo can sit in a latched fault, ignoring commands, while the software reads its position, writes new goals and reports everything fine.
What finally cracked it was one piece of reasoning that I'd keep from the whole night. A servo's control loop is self-consistent, because the same encoder feeds both the position it reports and its internal controller. So no host-side error (calibration, normalisation, sign conventions) can ever make the reported position diverge from the commanded goal, because those errors move both together.
My log showed the setpoint holding five degrees above present while present marched 181 degrees the other way. Software can't do that. I had the log in front of me for an hour before I drew the conclusion. Once I did, the calibration theories were all ruled out and only physical causes were left.
There are two corrections from that night that I'd want any SO-101 builder to have. First, --robot.max_relative_target looks like a safety limiter and it isn't one. Clamping the setpoint to present-plus-N clamps the position error, which clamps the proportional term, which caps the corrective torque the servo can apply. It makes the arm less able to hold itself up, and it slows a runaway down so you can watch it without preventing anything. The real stop button is the barrel jack.
Second, audit your own diagnostics. I wrote four little probe scripts that night and three were actively harmful. One called connect(), which runs configure(), which writes EEPROM and re-enables torque on the servo under test. One held the joint at a commanded target for a second per step, which is the exact stall event that had latched it. One measured with torque off, so it could never have seen this class of fault. A tool you write to observe a system can still change that system.
Getting Eyes
The camera step turned out to be mostly decisions, and all three were worth making before plugging anything in.
The names came first, because LeRobot camera names aren't labels. They become dataset keys like observation.images.front, baked into every recorded episode and every trained policy, and they must be identical across teleop, record and rollout. I caught my own docs using wrist/scene in one place and front/wrist in another. Now that's a one-line edit. After recording, it's a dataset whose keys don't match the commands that use it. They're front and wrist for good.
Indices can't be trusted the same way. On macOS there's no udev to pin a camera to a path, and the integer index is an accident of enumeration that reshuffles on any replug. So I identify cameras by procedure. Every session starts with lerobot-find-cameras, which saves a test frame per index, and you match by image content. Front sees the workspace, wrist sees the gripper, and the third camera sees me, because the MacBook's own webcam enumerates right alongside the robot's.
Resolution is the decision that looks wrong on the bench and only makes sense later, at training time. Both cameras can do 1080p, but everything runs at 640×480 at 30 fps. π0.5 resizes every frame to 224×224 before the model sees anything, so capturing above 640×480 adds zero accuracy and costs twice: USB bandwidth on the Mac while recording, and video-decode time in the training dataloader, which is the bottleneck that makes a second GPU stop helping.
Then the front camera, a Razer Kiyo Pro, simply wasn't there. Only two cameras were found: the wrist cam and the MacBook's. My software instinct said missing device, so missing driver, and that was wrong twice over.
There are no macOS drivers for the Kiyo and none are needed, because it's a UVC camera. USB Video Class is the standard protocol that makes webcams plug-and-play. That's a guarantee you can debug with: if the OS can see the device electrically, it will work. So "not working" means "not seen", which means the problem is physical.
There's a command for "can the OS see it electrically", and ioreg -p IOUSB showed no Razer device at all. It wasn't misbehaving, it was absent, with nothing enumerated for a driver to bind to. I swapped the USB-C adapter and it appeared instantly.
That ten-minute fix gave me a triage order that I've used for every "not detected" since. Check the USB bus with ioreg, then the OS camera list with system_profiler, then the application layer with lerobot-find-cameras. Each layer has a different kind of fix, and the fix is always at the deepest layer that's silent. The wrist_flex night in Four Hours was the same lesson at ten times the cost. The difference this time was that I asked which layer was silent before theorising inside the layer I happened to be looking at.
Country Settings
The Kiyo wasn't done with me. With enumeration fixed, it delivered 18 real frames per second while claiming 30. That became the week's one hard blocker, and I wrote it down with its own pass condition rather than working around it. A 30 fps dataset fed by an 18 fps camera silently duplicates four of every ten frames into every episode forever, and nothing in lerobot-record will warn you.
Fixing it took three layers, each a different lesson behind the same symptom.
Layer one was USB bandwidth. The Kiyo had negotiated USB 2 through a cable without SuperSpeed pairs. The nasty part is that at USB 2 its control channel works perfectly. It enumerates, accepts resolution and fps settings and passes every property check, while its video stream delivers almost nothing. My camera probe had been setting modes and reading them back, which is pure control-channel traffic, and printing OK over a dead data path. A validator that doesn't exercise the path validates nothing.
I rewrote the probe to pull actual frames, and added a proper 5 Gbps cable and hub to the shopping list. ioreg's Device Speed field is the one-line check: 2 means USB 2, 3 means SuperSpeed, and the Kiyo needed a 3.
Layer two was a lens cap. With the new hub the probe read 12 fps, and I nearly went back to debugging USB. Then I grabbed one frame and looked at it, and it was pure black. A camera staring into darkness winds its shutter out to maximum and the frame rate follows. A covered camera and a broken camera produce identical metrics, and only the picture tells them apart.
Layer three is the one I'd tell at a party. Cap off, scene bright, good image, still 12 fps. I compiled a small open-source tool called uvc-util and started reading the camera's actual UVC control registers over IOKit, using the standard control plane every webcam carries. And there it was: power-line-frequency = 1, which is 50 Hz anti-flicker mode, the UK setting.
The camera times its exposures to the mains cycle so indoor lighting doesn't strobe, and it enforces that by capping capture at 25 fps. Auto-exposure in indoor light then dragged that down to 12. Nothing was broken. The camera was correctly configured for a country, in a way that happened to be wrong for a robot. Setting 60 Hz timing plus manual 6 ms exposure gave 28.8 fps sustained with no banding under the bench lamp. It also froze brightness across episodes, so that's one auto-variable removed rather than worked around.
uvc-util -I 1 -g power-line-frequency # → 1 (50 Hz) the smoking gun
uvc-util -I 1 -s power-line-frequency=2 # 60 Hz timing → 25 fps cap lifted
uvc-util -I 1 -s auto-exposure-mode=1 # manual exposure
uvc-util -I 1 -s exposure-time-abs=60 # 6 ms shutterTwo follow-ups. The settings are volatile and reset on every unplug, so the fix became a session-start script rather than something I have to remember. My own probe also turned out to under-report on this camera, because cycling through six mode requests in six seconds leaves it mid-renegotiation. I gave the probe a sustained mode that measures the way teleop actually uses the camera, which is open once and stream. That's the second diagnostic of the project that needed diagnosing.
The parcel that fixed the cables also put the whole bench onto one Mac port, and the final ioreg tree showed how USB actually works. Every USB 3 hub is two hubs in one shell, a SuperSpeed hub and a USB 2 hub on separate wire pairs, and each device attaches to whichever it can negotiate. The Kiyo had the 5 Gbps branch to itself. The wrist cam and both serial boards shared the 480 Mbps branch and needed a tenth of it. The serial boards enumerate at Speed 1, twelve megabits, which is correct for a 1 Mbaud UART.
That also explained the earlier mystery of boards that "didn't work through the hub". They'd been plugged into female USB-C sockets that were power-passthrough inputs, electrically connected to nothing a data device can talk to. Nothing errors and nothing logs, and the device is simply absent. Both boards came through the rewiring with their /dev/tty.usbmodem paths unchanged, because of the serial-number naming I set up on day one.
One more fault from that stretch is worth recording, because of when it happened and what caught it. A routine reconnect died with Failed to write 'Lock' on id_=5. That was the identical error, on the identical motor, as a one-off from the previous night's calibration, where I'd almost not bothered writing the note "retry, don't panic, but if it recurs on the same ID, suspect that link's connector". It recurred straight after the wrist-camera mounting session, which was half an hour of screwing and cabling around the part of the arm where motor 5's wires run. Moving the arm around cleared it. It was a pinched or half-seated 3-pin connector, disturbed during the mounting, making contact or not depending on pose.
The note is the real lesson here. Nobody remembers an error string and a motor ID across two midnights, and that one line meant the second occurrence was a pattern instead of a fresh mystery. A log is a diagnostic you set up in advance, not just a record. The rest was normal debugging. The last thing you touched is the first suspect, and physical work disturbs more than the part you were working on. And a fault that comes and goes with the arm's posture can only be mechanical. The rule in the log now is that on a third strike on motor 5 the cable gets swapped, not debugged.
Toll Booths
Getting a dataset from the Mac to gradients on the GPU box involves a series of small unmarked obstacles, and I hit every one. The rsync straight out of my own runbook failed twice before moving a byte. macOS ships openrsync, which doesn't know GNU's --info=progress2, and rsync won't create missing parent directories on the remote.
Then the SSH connection itself hung. Two easily-missed lines above the timeout explained that Tailscale wanted a browser click to re-verify the connection. The gotcha is that every failed attempt creates a fresh auth URL, so you have to approve the URL from the attempt you're keeping alive. It's a security check that shows up as a hang, two layers below the tool you're running. Same as every camera failure that week: find the silent layer first.
Two authentication facts made the rest painless. A Hugging Face login is a 37-byte token file, so scp logs a machine in with no browser and no prompt. Weights and Biases needed a key that only a human with a browser could fetch, so I had to decide on the order. I started training without the dashboard rather than leave two RTX 3090s idle waiting for it, betting that restarting is nearly free early in a run. That bet got tested sooner than I expected.
Three Launches
I'd verified the training command (the full block and its flags are in Training Recipe) against the LeRobot source four days earlier. But what executes is the GPU box's checkout, and four days is enough for drift, so my pre-flight included grepping the deployed repo for the exact flags the command depends on. They were there. The command still failed twice, and each failure taught me something real about π0.5.
Launch one died at feature validation. The dataset has cameras named front and wrist. π0.5-base was pretrained on openpi's names, base_0_rgb, left_wrist_0_rgb, right_wrist_0_rgb, and disjoint camera sets are a hard stop. The fix is a --rename_map flag, but the useful part is two facts I verified in the source before trusting it.
First, the missing third camera is supported. π0.5 fills empty slots with blank images and a zeroed attention mask, because it was pretrained knowing that robots have different numbers of cameras. So before you invent a shim, check whether your model already handles the gaps in your hardware.
Second, the rename is saved into the checkpoint's own preprocessing pipeline, so the fix lives in the artefact rather than in my memory. That's the same principle that put the Kiyo settings in a session-start script. (One catch I found the hard way later: the rollout tool's validator ignores the saved pipeline and needs the same flag passed again. The docs are only a hypothesis until the machine agrees.)
Launch two got through the model download and died on a 403. π0.5's language tower is PaliGemma. The fine-tuned weights sit ungated on the Hub, but the code fetches the tokenizer and config from Google's original repo, which is licence-gated8. It turns out "forbidden" has layers just like "not detected" did. You can be not logged in, not have accepted the gate, or have a fine-grained token missing the gated-repo scope, and each layer has a one-line check. That night I needed the second and third fixed, and both are browser clicks that no automation can do.
Launch three ran. The failures cost thirteen minutes in total, most of it PaliGemma download time. Before the first checkpoint exists, a training run is the cheapest thing on the bench to throw away. Avoiding a relaunch would have cost more than relaunching did.
The wandb key arrived mid-warmup and forced one more restart, which cost ninety seconds of progress. That's why the whole thing ran as a script inside tmux with output mirrored to a log file through tee. When a launch dies, the pane can die with it, but the log survives. Every diagnosis that night came from that file.
One trap inside the doc's own command only shows up some of the time. $(which lerobot-train) expands in the outer shell, so it works when typed into an activated venv and fails inside a fresh tmux pane where no venv is on PATH. Explicit .venv/bin/ paths work in both.
Assets
Everything this post measured came out of the datasets and checkpoints below. The five jenga_wall datasets and the four models are public. The three HIL-SERL datasets are still private at the time of writing. Each model is one checkpoint with no optimiser state, so it can be run but not resumed.
Dataset (shavvimal/) | Episodes | Frames | Role |
|---|---|---|---|
jenga_wall_20260810_155119 | 149 | 65,810 | v1 training set, the day-five recording session |
jenga_wall_v2 | 294 | 140,672 | v1 plus entropy scatters plus DAgger r1 and r2 corrections |
jenga_wall_v2b | 293 | 137,073 | v2 training set: jenga_wall_v2 minus one bad episode |
jenga_wall_v3_no_dagger | 264 | 118,322 | v3 training set: v2b with the 29 DAgger episodes removed |
jenga_wall_v3_rel | 264 | 118,322 | the same episodes, in the repo the relative-actions run pointed at |
hilserl_grasp_demos_cropped_resized | 15 | 3,316 | HIL-SERL grasp demos, 10 fps, 128×128 |
hilserl_grasp_reward_cropped_resized | 19 | 7,832 | reward-classifier training set, round one |
hilserl_grasp_reward_v2 | 23 | 9,802 | reward-classifier training set, round two |
Model (shavvimal/) | Trained on | Steps | Checkpoint on the Hub |
|---|---|---|---|
pi05_wall | jenga_wall_20260810_155119 | 20,000 | 020000, the final step |
pi05_wall_v2 | jenga_wall_v2b | 15,000 | 015000, the final step |
pi05_wall_v3_no_dagger | jenga_wall_v3_no_dagger | 20,000 | 018000, hand-picked; the one the RTC server served |
pi05_wall_v3_rel | jenga_wall_v3_rel | 20,000 | 018000, hand-picked |
Every run shared one recipe, with the full command in Training Recipe: accelerate launch across two RTX 3090s in bf16 with NCCL_P2P_DISABLE=1, starting from lerobot/pi05_base with gradient checkpointing and train_expert_only=true. Batch 4 per process (effective 8), learning rate 2.5e-5 with the scheduler's decay set to the total steps, a checkpoint every 2,000 steps and an eval pass every 1,000. The runs differ in three places: eval_split was 0.05 for v1 and v2 and 0.045 for both v3 runs, image transforms were off for v1 and on from v2 onward, and only pi05_wall_v3_rel set --policy.use_relative_actions=true. The patches the rollouts ran on live on the feat/remote-rtc-inference branch of github.com/Shavvimal/lerobot.
Rerun Memory Warning
One reassurance for anyone who runs this rig: Rerun eventually prints a scary Exceeded gRPC proxy server memory limit (1.0 GiB) warning during long sessions. That's the viewer trimming its own scrollback. It's a warning about the dashboard, not the robot.
Camera Crop Trap
One more perception trap from the practice sittings: the macOS Camera app doesn't show what the dataset sees. The Kiyo's sensor is 16:9, and LeRobot records a 4:3 crop of it, so a scene staged to look right in the Camera app can have a block half outside the dataset's actual frame. Only judge the scene through the pipeline's own views, which means Rerun or the saved OpenCV captures. Every intermediary has its own crop, and a check through the wrong one validates the wrong thing.
Hardcoded Username
The recording command failed on its first run, because my docs derived a constant instead of stating it. They derived the Hugging Face username with hf auth whoami piped through awk. The CLI had changed its output format, the split matched nothing, and the record tool refused the malformed dataset name. The actual error was buried under two screens of harmless macOS objc warnings. Diagnosis took one echo. The username never changes, so it's now hardcoded everywhere.
The check also caught something worse. The account is shavvimal, not the shavhugan my email would suggest. Written down unchecked, that assumption would have broken every dataset name.
Checkpoint Gauntlet
Two of the three obstacles between the gate checkpoint and the arm were plumbing. I ran the rsync twice on purpose, because when you copy from a directory a live trainer is still writing to, you only know the copy is complete once a second pass shows nothing changed. The Mac's environment also lacked the pi extra, and π0.5 needs it just to run inference. The trap is that the sync command removes any extra you don't re-list.
Data Hygiene Catches
On the counterattack's recording and merge day I caught six data hazards before any of them did damage. Before recording over the most valuable directory on the machine, I answered "will this overwrite anything?" by reading the source rather than trusting that it would be fine. Every launch auto-appends a timestamp to the dataset name, and a collision raises an error instead of overwriting. Six probe directories from six launches were sitting in the cache to confirm it.
Mid-afternoon the Kiyo had another camera incident (the first two are in Getting Eyes and Country Settings). This time it was a wedged stream, with the device showing healthy in ioreg and the setup script erroring, and a replug fixed it. That's three incidents at three different layers of the same stack.
A sanity check that's becoming a habit excluded one stray six-episode dataset. Its episodes averaged 112 seconds, suspiciously close to the 120-second timer ceiling, which meant the backstop had been firing instead of my keypress. Dividing frames by episodes caught it before it could poison a merge.
One query on the data was enough to throw out one junk DAgger correction. I remembered it only as "the second one, or maybe index 2", and the query returned episode 1, 5.7 seconds, pan travel 0.0, zero gripper holds. That's a frozen arm, and it stands out clearly against fourteen 15-to-31-second sweeps. It was the third time this project that a measurement corrected my memory.
Assembling the v2 training set had two traps, and I caught both in time. DAgger datasets carry an extra intervention column the plain recordings lack, and a merge can't reconcile mismatched schemas. Thirty seconds diffing the feature lists in each dataset's metadata caught what would have been a failed multi-minute operation.
The in-place edit backup is the more dangerous one. A second in-place edit on the same dataset deletes and replaces the first edit's _old backup, so chained edits silently lose the backup of the original. The safe way is to write every operation to a new repo id and never touch the originals at all.
Training Run Health
Both training runs were healthy, and the CSVs caught one detail the docs had wrong: the cosine doesn't decay to zero. Both runs floor at exactly a tenth of the peak rate, 2.5e-6.
Gradient norms stayed between 0.67 and 1.18 with zero spikes, dataloading took under 0.5% of step time, and the colour-jitter cost was unmeasurable against a 1.85-second update. v2b's slower wall clock comes from its bigger eval, about 430 seconds per thousand-step boundary against v1's 200.
Footnotes
-
Tony Zhao et al.: "ALOHA Unleashed: A Simple Recipe for Robot Dexterity", Google DeepMind, CoRL 2024. RandomKitchen, 3,198 demos: success through successive stages 95%, 65%, 25% ↩
-
Yu & Qiu: "Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis", 2026. Recovery rates ACT 6.45%, SmolVLA 3.23%, π0.5 30.77%; π0.5 with 100 demos per task 56.25% overall; language-grounded Selective Color Sorting 0 to 10% across all policies ↩ ↩2 ↩3 ↩4
-
Xuanlin Li et al.: "Evaluating Real-World Robot Manipulation Policies in Simulation" (SimplerEnv), CoRL 2024. Across its manipulation suites, stacking tasks convert successful grasps into task success at markedly lower rates than place-into-container tasks ↩
-
Ajay Mandlekar et al.: "MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations", CoRL 2023. With 10 human demos: Stack Two 26.0%, Stack Three 0.7% ↩
-
LeRobot: "SO-101 follower's servo motor trouble", GitHub issue #2819. Maintainer diagnosis: "your servo has broken, this can happen if it is overloaded for to long... pressing to hard and overloading motor of gripper" ↩ ↩2
-
Yulin Cui et al.: "AhaRobot: A Low-Cost Open-Source Bimanual Mobile Manipulator for Embodied AI", 2025: "the lightweight cup was prone to being knocked over, causing a performance drop at the 'Grasp Cup' step" ↩
-
Hugging Face: "Imitation learning on real robots", LeRobot documentation. Recommends at least 50 episodes with 10 per location, and that "you should be able to do the task yourself by only looking at the camera images" ↩
-
Google: "paligemma-3b-pt-224", Hugging Face. Licence-gated; π0.5 fine-tunes fetch its tokenizer and config from this repo ↩ ↩2
-
Kevin Black et al.: "π0.5: a Vision-Language-Action Model with Open-World Generalization", CoRL 2025, and the Physical Intelligence blog. A PaliGemma-derived VLM backbone with a ~300M-parameter flow-matching action expert, co-trained on heterogeneous data for open-world generalisation ↩ ↩2
-
Dream Machines: "Empirical results fine-tuning π0.5 on a real manufacturing task", September 2026 ↩ ↩2 ↩3 ↩4
-
Stéphane Ross, Geoffrey Gordon, Drew Bagnell: "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning" (DAgger), AISTATS 2011 ↩ ↩2
-
Mohammad Pezeshki et al.: "Gradient Starvation: A Learning Proclivity in Neural Networks", NeurIPS 2021; Harshay Shah et al.: "The Pitfalls of Simplicity Bias in Neural Networks", NeurIPS 2020 ↩
-
Wikipedia: "Clever Hans". In Pfungst's 1907 experiments the horse answered correctly 89% of the time when the questioner knew the answer and was visible, 6% when not ↩
-
Robert Geirhos et al.: "Shortcut Learning in Deep Neural Networks", Nature Machine Intelligence, 2020 ↩
-
Pim de Haan, Dinesh Jayaraman, Sergey Levine: "Causal Confusion in Imitation Learning", NeurIPS 2019: "access to more information can yield worse performance" ↩
-
Chuan Wen et al.: "Fighting Copycat Agents in Behavioral Cloning from Observation Histories", NeurIPS 2020 ↩
-
Hugging Face: "π0.5 policy documentation", LeRobot. The port ships the flow-matching action head under Apache 2.0; the paper's subtask-prediction and RL components were not released upstream ↩
-
Fanqi Lin et al.: "Data Scaling Laws in Imitation Learning for Robotic Manipulation", 2024. Diversity "is far more important than the absolute number of demonstrations"; past a per-environment threshold, additional demos have minimal effect ↩
-
openpi: "[pi0.5] Robot reaches object but fails to grasp, keeps pushing object with small actions", Physical Intelligence GitHub issue #912. Open and unresolved at the time of writing ↩ ↩2
-
Penn PAL Lab: "Evaluating π0 in the Wild", University of Pennsylvania. The policy "does not have a metrically accurate method to determine the distance between the gripper and the surrounding environment"; it is also memoryless and has no tactile sensing ↩
-
Sherry Chen: "Three iterations of ACT on SO-101", Hugging Face blog. Fifty undiverse episodes produced a near-0% policy whose first iteration "pecked at the table near the block, missing the grasp"; recomposing the data (25 episodes per position bin, orientation variation, grasp-the-middle discipline), not multiplying it, moved success to 90% in-distribution and 75% out ↩
-
Physical Intelligence: "openpi training configs", GitHub. The π0-family transforms use delta joint actions with the gripper dimension masked out as absolute ↩
-
Hugging Face: "Real-Time Chunking", LeRobot documentation. Chunked policies plan up to 50 actions per inference and execute a horizon of them, so late-chunk actions run against observations up to 1.7 seconds old ↩
-
Hugging Face: "Train a Robot with Real-World Reinforcement Learning (HIL-SERL)", LeRobot documentation. An actor-learner SAC loop in end-effector space with a learned reward classifier; the leader arm doubles as the human-intervention interface ↩
-
ggando: "HIL-SERL for SO-101: Real-World Grasping from Scratch". Roughly 70% grasp success learned from scratch on an SO-101; related runs report 80% on the different reach-and-grasp task after about 750 episodes ↩
-
Kevin Black, Manuel Y. Galliker, Sergey Levine: "Real-Time Action Chunking with Large Models", Physical Intelligence, June 2025. Frames chunk handover as inpainting: freeze the timesteps that will already have executed by the time the new chunk arrives, and generate the rest around them ↩
-
Jianlan Luo, Charles Xu, Jeffrey Wu, Sergey Levine: "Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning" (HIL-SERL), 2024. Demonstration-seeded SAC with a learned reward classifier and human interventions, reporting near-perfect real-world success on a range of manipulation tasks within one to two and a half hours of training ↩
-
David Watkins: research lead for data collection and foundation models at the RAI Institute (formerly the Boston Dynamics AI Institute), since moved to Toyota Research Institute; co-author with Stefanie Tellex of "What to Tell the Robot"; personal site at davidjosephwatkins.com. No public recording of the talk to link at the time of writing ↩
-
Cheng Chi et al.: "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots" (UMI), RSS 2024 ↩
-
Hugging Face: "lerobot/umi_cup_in_the_wild", the UMI cup-arrangement dataset in LeRobot format: 1,447 episodes, 699,432 frames at 10 Hz. The comparison ran the autopsy instruments on its state channel; that export ships no action column, so the gripper-interval instruments were excluded as artefacts ↩
-
Vidullan Surendran et al.: "Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts" (BRIDGE), RAI Institute, 2026. On their NIST pulley task: base policy 44%, base plus support data naively mixed 0%, routed through state-gated experts 76%, teleop-only ceiling 84% ↩
-
ggando: "SmolVLA on SO-101". Documents the grasping-at-air symptom on a cube-to-bowl task and cures it by shrinking the workspace and enforcing one grasp strategy: demo density per approach configuration, not architecture ↩
-
LeRobot: "pull request #2970", adding
use_relative_actionsfor π0-family fine-tunes; the description notes its absence was "causing a distribution mismatch when fine-tuning from OpenPI pretrained weights" ↩ -
openpi: issues #1012 and #692, Physical Intelligence GitHub. Together with issue #912, the reports converge on gripper quantile normalisation as the mechanism behind fine-tuned policies that reach correctly but fail to grasp ↩
-
LeRobot: "issue #4463", GitHub. Expert-only training "almost failed" on a task that full fine-tuning handled, despite showing lower training loss ↩
-
Yihao Zhang et al.: "Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation", 2025. Places the resolution-limited regime for 224×224 visual inputs at sub-centimetre precision, well below my measured 3-to-4-centimetre systematic miss ↩
-
openpi: "issue #766", Physical Intelligence GitHub. The community's standard remedy for premature gripper closes is a shorter execution horizon ↩
-
Hugging Face: "shavvimal/pi05_wall_v3_rel" and the sibling model repos
pi05_wall,pi05_wall_v2,pi05_wall_v3_no_dagger; one checkpoint each. The five training datasets: jenga_wall_20260810_155119, jenga_wall_v2, jenga_wall_v2b, jenga_wall_v3_no_dagger, jenga_wall_v3_rel ↩ -
Hugging Face: "SO-101 assembly and setup guide", LeRobot documentation ↩
-
Feetech: "STS3215 servo specification" (translated datasheet): overload protection triggers after 2 s blocked above 80% of locked-rotor torque ↩