A signal takes 2.56 s to reach the Moon and come back. In a 2024 University of Bristol study, people steering a robot arm with delays up to 2.6 s did worse on every measure. RoBuff lets you steer a simulated robot arm with your hand through a webcam, and switch on the Moon's delay. A see-through ghost arm then shows where the arm is heading before the delayed view comes back. The jaw holds objects by friction alone. It works with a real hand on a laptop webcam. In 5 of 5 test runs steered by code, it lifted a test cube 78.4 mm.
The ghost arm is a predictive display. DLR, Germany's space agency, used the same idea on its ROTEX robot arm in space in 1993. Sources
The hand tracking, the arm and the physics all run in your browser. The arm is a simulated SO-ARM100, a small open-source robot arm. No real robot is attached. In Moon mode, your commands take 1.28 s to reach the arm. The picture takes another 1.28 s to come back, as on a real Earth-to-Moon link. A see-through ghost arm shows where the arm is heading. I have not measured yet whether the ghost helps people.
MediaPipe is Google's hand-tracking model. It finds 21 points on a hand in each camera frame. It runs on your own computer, so no video is uploaded.
Timed in an automated test, with a photo of a hand as the camera feed. I have also used it with my own hand, but did not time it.
Every hand shakes a little. The app evens out the position of your palm, so the arm does not shake with it. It smooths more when your hand is still and less when you move, so the arm still keeps up.
MuJoCo is a physics engine that works out contact and friction. It simulates the SO-ARM100 arm, so the cube rises only if the jaw's grip really holds it.
three.js is a JavaScript library for drawing 3D graphics in a web page. It draws the arm and the rock where MuJoCo puts them at each step.
Everything runs in the browser · 240 frames per second in an automated test · physics at 500 steps per second
From my automated tests · a hand steered by code and a photo as the camera · all simulated
MediaPipe finds your hand in each webcam frame. Move your hand and the arm follows. Move it nearer or farther to change the reach. Pinch to close the jaw. I have tried it with my own hand on a laptop webcam, and it works. The tests and the picture here used a hand steered by code.
Example values from a test with a photo of a hand
You type what you want. Gemini, Google's AI model, turns your words into a description the app can build. Without Gemini, a simple keyword reader in the app does this instead. The app builds a 3D model. For things like a mug or a plane, Gemini is asked for 3 to 8 coloured simple shapes. Before it appears, 7 checks decide whether the arm could hold it.
These results used the keyword reader · Gemini 3.8 Flash when connected
A hidden copy of the simulation lifts the object with the same arm. It holds the object up, and it passes only if the object is still at least 48 mm up at the end.
A generated object is only useful if the arm can hold it. So the app checks every request first. Then it places the object and runs a simulated lift test. If the lift fails, the object is removed. I tested seven requests in the browser. Five were lifted, one of them after being rebuilt as a smaller model. The checks refused the other two before they were placed.
Adding a new object to the scene took 210.8 to 258.5 ms. Each lift test took 399 to 551 ms. Both were timed in an automated Chrome test on my laptop, with MuJoCo running in the browser on the CPU.
A mug or a plane is hard to make from one shape. So Gemini is asked for 3 to 8 coloured simple shapes, and the app builds the object from them. Sometimes Gemini sends back one plain shape instead. In the physics engine, each part is its own solid, so the jaw grips the real parts, not one box around the whole object.
In one live run, all 5 requests were scaled down to fit the jaw and passed the lift test. Four of them are below.
| Request | Parts | Mass | Lift test |
|---|---|---|---|
| a plane | 7 | 10.45 g | rose 78.3 mm |
| a Tesla | 6 | 196 g | rose 75.8 mm |
| a coffee mug | 5 | 196 g | rose 69.2 mm |
| Minecraft grass block | 5 | 196 g | rose 74.3 mm |
Heavy ones were scaled down to 196 g, just under the 200 g limit. Gemini writes a new part list each time, so the parts and sizes change from run to run. These are toy-size models made of simple shapes, not detailed copies.
You can also give any object a skin, a painted look such as brushed gold. Nano Banana, Google's image model, draws it from your words. A skin changes only the look, never the physics.
Ask
40 mm
I asked for a Bennu-like sample. The app's keyword reader handled it, not Gemini.
Weigh
29 g
18 cm³ at 1.6 g/cm³, the density of the sample brought back from Bennu.
Check
7 checks
Valid shape, volume, jaw angle, size, mass, friction, grip height.
Place
258.5 ms
The physics scene is rebuilt with the rock in it. Timed in an automated Chrome test on my laptop.
Lift test
77.3 mm
A hidden copy of the arm lifts it. A rise of 48 mm or more is a pass.
Pick
82.8 mm
A hand steered by code moves the arm, and the arm holds it for 5.77 s.
After it was placed, the rock rolled 18 mm as it settled. The lift test starts from where it stopped.
A physics engine that works out contact and friction. I run it in the browser so a rock rises only if friction really holds it.
Google's hand-tracking model. It finds 21 points on a hand in each camera frame. It runs on your own computer, so no video is uploaded. In a test that fed it a photo of a hand, it took a median 11 ms per frame.
A small open-source robot arm with 5 joints and a jaw. The model comes from MuJoCo Menagerie, a library of checked robot models. I used it as published, with no changes.
These are automated test tools. Playwright drives a real Chrome browser from a script. Anyone can use it to re-run my tests of the pick, the hand tracking and the object requests. Vitest tests the physics, the filters and the checks without a browser. 381 of 381 unit tests passed.
Google's AI model. It turns a typed request into an object description (kind, size, material) that the checks can test. In a live test, 5 of 5 replies came back in the right format. The median reply took 3867 ms. That test only checked the format. The sizes it returned did not always match the request. It also writes the part lists for objects made of parts, and it answers when you talk to RoBuff.
A second simulation runs your commands with no delay. It draws a see-through arm where the delayed arm is heading. DLR used the same idea on its ROTEX robot arm in space in 1993. Whether it helps people is not measured yet.
Chrome's built-in speech recognition. It turns what you say into text, so you can give commands without typing. Hold V to talk, or press Talk live to talk hands-free. Commands like "stop" or "move left 5 centimetres" are handled in the app. Anything else goes to Gemini when it is connected.
A text-to-speech service. It reads RoBuff's replies out loud. In a live check, 5 of 5 test lines came back as audio, and the first audio arrived after a median 176 ms. Without it, the browser's own voice speaks. Captions are always on.
Google's image model. It paints an object from a few words, such as brushed gold. It uses Nano Banana Pro by default, and Nano Banana 2 Lite is the faster choice. In live runs, a new picture took 12.1 to 12.9 s on Pro and 3.1 to 4.5 s on Lite. Each new picture is paid for, so the app allows up to 25 Pro or 100 Lite pictures a day. The app saves each picture, so the same request is not paid for twice.
| Part | State | What that means |
|---|---|---|
| Hand to arm | Works and tested | Tested with a scripted hand (one steered by code), and with a photo of a hand fed in as the camera. The hand was found in 123 of 123 frames. It also works with my real hand on a laptop webcam. I tried it, but did not time it. The pinch point is not tuned for a live hand yet. |
| Friction pick and lift | Works and tested | A hand steered by code moved the arm through the same steps a webcam hand uses. The test cube rose 78.4 mm and stayed up 5.73 s. 5 of 5 repeats from one start passed. 6 of 9 cube positions worked. |
| Generated objects and lift test | Works and tested | 7 requests in the browser. 5 were lifted. 2 were refused, with the numbers shown. |
| Objects built from parts | Works and tested | In one live run, all 5 requests were built from parts, scaled down to fit the jaw and lifted. Gemini writes a new part list each time, so the results change from run to run. Where parts overlap, their mass is counted twice. |
| Skins | Works and tested | A skin is a painted look, such as brushed gold. It changes only the look, never the mass, friction or shape. Pictures from Nano Banana are AI-made and carry SynthID, Google's invisible mark for AI images. If Nano Banana fails, the app draws a pattern itself and says so, which was tested with a faked failure. The drawing times on the Nano Banana card come from live runs and were not re-checked independently. |
| Gravity menu | Works and tested | Earth, Mars, Moon, Bennu and orbit. A cube dropped 0.10 m landed in 0.354 s on the Moon (theory says 0.351 s) and 0.144 s on Earth (theory says 0.143 s). |
| Robot arm | Simulated | MuJoCo physics on the public SO-ARM100 model. No real robot is connected. |
| Contact at low gravity | Simulated, unreliable | At Bennu gravity, a 5 cm/s push lifted a resting cube 46.9 mm. That is an error in the simulation, not real physics. The app shows a warning when gravity is below 0.1 m/s². |
| Object requests | Works and tested | Gemini 3.8 Flash reads typed requests. In a live test, 5 of 5 replies came back in the right format. The median reply took 3867 ms. That test only checked the format. The sizes returned did not always match the request. Without Gemini, a simple keyword reader in the app takes over. The 7 requests shown above used that keyword reader, not Gemini. |
| Voice | Works, with a fallback | Hold V to talk. Chrome's speech recognition needs the internet. Known commands are handled in the browser, and other lines go to Gemini when it is connected. ElevenLabs speaks each reply, and without it the browser's own voice speaks. Captions are always on. |
| Talk live | Works and tested | Press the Talk live button once and talk hands-free in Chrome. Gemini writes the answer, and RoBuff says it out loud. RoBuff can also act on what you say, such as making an object, changing gravity or moving the arm, and the app checks each action first. A line that goes to Gemini takes about 5 to 7 s to answer. While RoBuff is speaking, it does not listen, so a spoken stop waits until it finishes. Space stops the arm at once. In one test, code typed the lines through a fake microphone. Asked for a tennis ball, Gemini made a solid aluminium ball 67 mm wide. At 234 g it was over the 200 g limit, so it was refused. The app then asked for the lighter 52 mm size the refusal named, and that ball passed the lift test at 195.9 g. In a second run, the lift test dropped it. |
| Voice arm control | Works and tested | Say "move left 5 centimetres" and the arm moves. In an automated test, the gripper moved 49.7 mm for a 50 mm request. After a stop command, it moved less than 1 mm. In Moon mode, spoken moves are delayed like hand moves, and the ghost arm shows them first. Spoken pick-ups and put-downs work only without the Moon delay. The tests fed in text, not a real voice. A test with a real voice is not logged yet. |
| Stop | Built, tested with scripted hands | Pressing Space freezes every joint and the jaw. Only the on-screen Resume button clears it. In two-hand mode, holding both hands open for 0.4 s also stops the arm. In tests it stopped at 0.40 s. A 601-frame replay of a pinch and carry never set it off by mistake. A real-hand test of the stop is not logged yet. |
| Earth-Moon delay | Built, tested with a scripted hand | In Moon mode, commands reached the arm 1.28 s late, and the view came back another 1.28 s late. Measured in the simulation. |
| Ghost arm | Built, tested with a scripted hand | Two simulations run side by side. The ghost guesses the rock is 1.6 g/cm³ with friction 0.8. The delayed arm uses a hidden 2.3 g/cm³ and 0.6. So the ghost's guess can be wrong, as it would be with a real rock. |
| Ghost result | Not measured yet | No trials with a person are recorded yet. The tool to record them is built. I wrote the pass rule before collecting any data. With the ghost on, the median task time must be at most 0.75 of the time without it. It must also succeed at least as often. I will be the only person tested, so the result will not show that it helps other people. |
Objects can be 10 to 70 mm wide and up to 300 g. I set those limits with a safety margin from grasp tests on the simulated arm. When you steer by hand, the limits are tighter. Objects must be 15 to 55 mm wide and at most 200 g. These are estimates. I have not tested them with a live hand yet. The calculator and the refusals use these tighter limits. I have not checked how much a real SO-ARM100 can lift.
Predictive displays come from work at MIT (Sheridan, 1984) and JPL (Bejczy, Kim and Venema, 1990). DLR used one on its ROTEX robot arm in space in 1993. I searched 755 past UB Hacking projects. None put an arm in a physics simulation or added a space signal delay. For Fall 2025 I searched the full project pages. For older years I searched only titles and taglines, so I may have missed some.