Today's humanoid robots are remote-controlled steel giants with sophisticated actuators and control systems that are second to none. They walk, jump and dance, and in many demos that look autonomous, a person with gloves or a VR headset is sitting just out of frame. That way they are not much use yet. For the industry to deliver on its promises, they have to become autonomous. Gripping, fitting, dosing, tightening the grip when something slips, with nobody guiding the hand. Language models had the internet for their leap, a finished archive of text that only needed collecting. For what happens between a fingertip and a workpiece, no such archive exists. The data simply isn't there yet. It has to be created first, and the industry does not agree on how.
This summer showed how much is riding on that question. On August 25, Figure in San Jose announced an app that pays people to film themselves doing housework. According to the company, Index collected 16 million videos from 108 countries in four months, paid out 15 million dollars to the people filming, and processes 30 minutes of new footage every second. For the next twelve months, Figure has announced more than a billion dollars for data and compute. At the same time, NVIDIA is building a physics engine with Google DeepMind and Disney Research so that robots can learn the same things in simulation. China has announced more than 40 state-backed data collection centers where trainers in exoskeletons teleoperate robots. And in Zurich, a robot's control policy is being derived from a video model that has learned how things move when you touch them.
Whichever route a company takes, the data has to come into existence first. The argument is about how. Several routes are running side by side right now, there is capital on each of them, and it is quite possible that one turns out to be a dead end and the billions in it are gone. The routes differ mainly on one question, namely where the one quantity is supposed to come from that every grip needs and no camera records.
That quantity is force. The physiologists Roland Johansson and Göran Westling showed how much it matters back in the 1980s (overview in Nature Reviews Neuroscience). People adjust their grip force to the surface immediately after contact, holding something smooth more firmly than something rough. Numb their fingertips and that adjustment disappears completely, even though they can see their hand the whole time. They either crush the object or drop it.
The eye cannot close the gap because it is too slow. When an object starts to slip, the hand corrects its grip force after roughly 74 milliseconds, as a reflex, straight from the skin. A conscious correction takes about twice as long, and one that goes through vision tends to take longer still. By the time you have seen the glass slipping, it is on the floor. The palm side of the hand holds around 17,000 tactile units for this, and not one of them cares what the grip looks like.
A camera records what a movement looks like, not how much force it took. Rodney Brooks, co-founder of iRobot, has put thousands of robots into factories himself, and he considers the data being collected the wrong kind. Dexterity comes from touch, from resistance and slip at the fingertip, and that is exactly the dimension almost entirely missing from camera images and teleoperation recordings. You can measure that something is missing. A research team at Meta has shown that models which also process force and vibration improve their success rate by 63 percent over purely image-based tactile models.
There are several answers to where this force information should come from, and each one is its own way of producing training data. You can record the force along with everything else. You can leave it out, collect at scale instead and bet that the robot will regulate it by itself at runtime. You can leave the robot out of the collecting altogether. Or you can compute the force. Each route has an advantage, each has a risk, and none is proven.
Recording the force means not only capturing what a grip looks like while the data is being produced, but measuring what it does. A force-torque sensor sits at the robot's wrist, tactile sensors sit in the fingertips, and while a person teleoperates the machine, every newton the gripper applies is logged, along with every moment the part begins to slip. The dataset RH20T does exactly that, with force, torque, image, sound and joint position in sync across roughly 1,100 hours, which makes it the documented exception in the field. The advantage is obvious. This is the only route where the force is actually in the data.
The risk lies in the sensor. Tactile sensors come in many designs, fingertips with a camera behind a soft gel that deforms on contact, films that measure pressure through resistance or capacitance, optical fibers, microphones that hear the vibration of a slip. Every design delivers a different signal. The gel sensor delivers an image of its deformation, the film a series of numbers, the microphone an audio track. A model that has learned on one sensor what “about to slip” looks like does not recognize the same event on the next sensor, because the signal has a different shape, and even two units of the same design differ because of manufacturing tolerances. Images eventually got ImageNet, a dataset everyone builds on. For tactile data there is no standard, which is why nothing adds up. Then there is wear. A gel wears down, within seconds in an abrasion test and over weeks in operation, and then delivers a different signal for the same grip than it did on day one. More robust designs last longer, but they lack the common denominator too. Anyone who takes this route ties their data to a piece of hardware. If it disappears from the market, the data goes with it. That is exactly what happened with the long-popular BioTac sensor. It has been discontinued, and older datasets are aging with it.
That is the argument of the second route. Force does not have to be in the dataset at all, because it arises at the robot anyway. Every motor draws current, and the current gives away the torque. A force-torque sensor at the wrist measures what is acting on the workpiece. Impedance control, which means making the arm deliberately compliant instead of driving it rigidly to a position, has been inserting pegs into holes for decades, by now with tolerances below a tenth of a millimeter. The model learns the plan from video, and the machine regulates the force. So this route collects what can be collected cheaply and at scale, images and movements.
The scale comes from two sources. The first is data factories. AgiBot World from Shanghai comprises around a million trajectories, meaning recorded sequences of movement, just under 3,000 hours from a single hall where people teleoperate robots without a single product ever leaving the building. Rest of World recently counted more than 40 announced state-funded centers of this kind in China. What almost all of those hours contain is images and joint angles. How firm a grip was, whether something gave way or started to slip, is generally not in there. I call that kind of data force-blind.
The second source is people filming themselves. At Index a creator, which is what Figure calls the people filming, wears a camera on their head, tidies the kitchen, folds laundry or packs boxes in a warehouse, and gets paid by the minute. 15 million dollars for 16 million videos works out at less than a dollar per video on average. People in 108 countries produce, for a few dollars, the template for the machine that is supposed to do their work one day, and what Figure is buying is variety. Every new kitchen is one the model has never seen. Data like this is used to train vision-language-action models, VLAs for short, which predict the next movement directly from a camera image and a spoken instruction. With EgoScale, NVIDIA has trained such a model on 20,854 hours of egocentric video, footage from the point of view of a person using their hands, and reports a clean scaling law. More hours, a predictably better model.
This route is measured in hours, and the hour has become cheap. An hour of real teleoperation data now costs a fraction of what it cost two years ago. But the hour is the wrong unit. A data center can record a thousand hours of slicing bread when ten would have done. A 2024 study with more than 40,000 demonstrations found that a robot's performance grows with the variety of environments and objects, and that the amount per environment stops mattering beyond roughly 50 demonstrations. A thousand hours of slicing bread will not make a robot dexterous if ten would have done and the decisive quantity was never recorded in the first place.
What this route has proven is that a robot can grasp things it has never seen. Robust, force-controlled contact work is still to come, and that is where the risk lies. A force-torque sensor at the wrist measures the sum, not the spot on the finger where the force is applied. For a peg in a hole, the sum is enough. For a raw egg, a cable or a zipper, what matters is which finger presses where and how hard, and whether something starts to slip long before the wrist notices any of it. Most of the money is on this route, which is why its risk is the biggest. If it turns out that video cannot replace force, what is left is an archive of billions of clips that cost billions and makes no robot any more dexterous.
The third route does without the robot during collection altogether. The best-known approach is called UMI, an action camera on a handheld gripper that a person uses to demonstrate tasks. The camera records what the gripper sees and how it moves, and a robot with the same gripper later learns from that. The advantage is the price. No robot in the room, no hall, no teleoperation, and anyone can collect anywhere. The risk is the same as with video, only without the robot that could at least regulate the force later. The data is force-blind, and the position the camera computes from the image drifts on long tasks. A further development, TacUMI, builds tactile sensors and force measurement into the gripper, and straight away the hardware gets more expensive and more fragile, throughput drops, and the route inherits every sensor problem from route one.
If you don't want to record force or regulate it afterwards, you can compute it. In simulation, thousands of machines practice at once, every attempt costs electricity instead of parts, and nobody has to pick up a workpiece that fell on the floor. That is the advantage, and it is enormous. The industry has its own term for the distance between what a robot learns in simulation and what is left of it in reality, the sim-to-real gap. The Beijing training center X-Humanoid has measured how large it is for grasping. A robot arm is supposed to set a toppled cup upright. In simulation it succeeds in nine out of ten attempts. Trained on simulation data alone and moved to the real machine, it succeeds in one out of ten. The most common cause was a cup slipping out of the gripper while being turned, because the gripper had not closed firmly enough. In simulation, the same grip had been enough.
The reason is the surface. How rough a cup is, how a cable gives, how much backlash a gearbox has, none of that can yet be computed as precisely as the grip requires. In the test on the real robot, things then happen that never occurred in simulation, and the grip fails because of them. In a comparison of common physics engines on cloth, not a single one got dynamic manipulation exactly right. This is precisely where the investment is going. NVIDIA, together with Google DeepMind and Disney Research, has set up the open-source physics engine Newton, which since version 1.0 includes hydroelastic contact modeling, meaning distributed pressure patches instead of a single contact point. As an ingredient, simulation already works today. The same Beijing measurement shows that simulation data alongside real data raises the hit rate on the real machine. It works as a preliminary stage, as basic training. So far no engine models the last centimeter well enough that you would trust a production line to the result.
Video models take a different computational route. A model that has learned from billions of clips to generate believable videos must also have learned how things move when you touch them, otherwise every clip would look wrong. The Zurich company mimic uses that knowledge. It builds its robot control on the video model from Black Forest Labs and teaches it to output the robot's next movement instead of the next frame. By the company's own account, 30 minutes of footage from the real robot are then enough where 30 hours used to be necessary, in use at Audi among others. The risk of this route is called hallucination. In a recent paper on such models, an object jumps into the gripper without the gripper having touched it first. And the 30 minutes come from the real robot. Even the industry's most elegant route ends at the point where force is actually applied for the first time.
If you record the force, you have the right data and can hardly multiply it, because every sensor delivers its own signal that no other sensor understands, and because it wears out. The risk is putting a great deal of effort into data that only works with hardware that will no longer exist in a few years. If you collect at scale, you have millions of hours and billions of dollars, and the force has to emerge afterwards from the control loop, which is proven for the peg in the hole and not for the raw egg. The risk is a video archive that cost billions and never contained what mattered. If you leave the robot out of the collecting, you collect most cheaply and carry the same risk as the scale route, without the robot that could regulate afterwards. If you compute, you get volume almost for free and lose the last centimeter, where it is decided whether the part is held. The risk is a robot that can do everything in simulation and nothing in the hall. It is also the only route that could make the other three look old overnight, if contact is one day computed as well as walking is today.
All four routes are running side by side today, and none has pulled away from the others. Figure bets on scale and regulates afterwards. NVIDIA bets on scale and on simulation at the same time. The data factories in China bet on hours. Research bets on sensors that have no standard. Nobody has won, nobody has dropped out, and the billions are spread across all of them. That is exactly what makes the situation so uncomfortable for anyone building or investing in this field. You cannot say today which route will make a robot autonomous first, and you cannot say whether the routes will meet at some point or whether three of them end as soon as the fourth works. The industry has millions of hours of practice and does not know whether they are the right ones. The only thing it knows for certain is where the force gets measured in the end, at the first real screw, in the first real hall. Who gets it into their data by then, and whether hours are even what is missing, is the open question of this race.
If you are on one of these routes and have to explain to the outside world why your data is the right data, send me a message. With Lucent I help deep tech teams translate technical substance into a story that customers and investors understand.