Imagine learning to play the violin by watching someone else. You can follow the bow and hear the result. When you pick up the instrument, you also feel the string beneath your finger. Practice brings these observations together as you adjust how you play.
If we want robots to learn from a million hours of human demonstrations, what should those hours contain? We want to preserve practiced movements, measure the contact that accompanies them, and know how those movements relate to the robot’s controls. That question shaped S1, our skill capture glove.
We directly record 20 finger joint angles using joint encoders. We measure pressure at the fingertips and palm, with video from a wrist-mounted global-shutter camera on the same timeline. The demonstrations below show what we can collect, how we check it, and how we designed the glove together with our robot hand.
A million hours of natural human skill
We invited a musician to compose a piece and perform it live while wearing S1. We chose music because it asks a lot of the fingers. A violin passage calls for small adjustments; a finger snap happens in a fraction of a second. We wanted to see whether people could make these practiced movements while wearing the glove. If the device changes how someone grips or makes them slow down, it changes the behavior we want to record.
Film · Capturing practiced skill
We tried 30 demanding tasks, including fast Rubik’s Cube solves, card shuffling, and butterfly-knife tricks. The video below shows how we got on. (Credit to one of our engineering team members for the butterfly-knife tricks.)
Video · Dexterity while wearing S1
For data collection, we are interested in the skills robots will need for useful work. An experienced assembly worker shows us the pace and consistency a job requires. They also show us what to do when a part does not seat as expected. Those corrections belong in the recording too.
A million hours of manipulation, measured accurately
The performance tells us what a person can do while wearing S1. We also need to check what the recording preserves. We start by checking whether the measurements continue when the hand disappears from view. We then compare fingertip positions with an independent measurement, before looking at the contact behind the movement.
Continuity through occlusion
We recorded hand movements inside a container, where the camera could no longer see the fingers. The glove kept measuring. We also recorded movements where an object or the other hand hid only some of the fingers.
We started with MediaPipe Hand Landmarker. To strengthen the vision baseline, we also ran an offline pipeline using a WiLoR hand detector followed by HaMeR reconstruction. We still observed two problems.
Under full occlusion, the vision pipelines sometimes lose the hand estimate entirely. With partial occlusion, estimates can remain available but jump between finger poses. In these recordings, the glove keeps measuring through both, with smoother joint trajectories.
Video · Motion capture through occlusion
Accurate joint motion
A continuous recording still needs an accuracy check. We used OptiTrack motion capture to check the fingertip positions calculated from the glove. We combine the measured joint angles with the link geometry using forward kinematics, then compare the resulting positions with OptiTrack in the glove’s wrist frame.
In this recording, we measured millimeter-level fingertip position accuracy across the index, middle, and ring fingers. This check concerns fingertip positions in the glove’s wrist frame.
Video · OptiTrack validation
Contact
Knowing where a finger is does not tell us how firmly it is pressing. On the violin, a finger can stay almost in place while pressing more firmly against a string. The joint angles tell us where the finger moved; the tactile measurements record changes in pressure at the instrumented surfaces.
We record both alongside video on a shared timeline. In the passage below, we can follow when contact begins and how pressure changes through an adjustment. We also track the objects being handled, including articulated objects, using an object-tracking system developed in-house. In a future post, we’ll share more about this system and our real-to-sim work.
Videos · Contact and object tracking
Can we collect this at scale?
We built an earlier version of the glove seven months ago and deployed it in a couple of factories. It had a rigid shell.
Photos · Earlier factory deployments
Once people started trying it on, we ran into a practical limit: only a subset of workers could wear it. Their hands were different sizes, and the shell left too little room to accommodate them.
That sent us back to the design. If we wanted to collect at scale, more people needed to be able to wear the glove. We changed to the soft shell we use today so it could fit a wider range of hands. Each wearer still fits and calibrates the glove before recording.
Allowing that variation also leaves differences between the contacts made by a person’s hand and those made by the robot. We believe a learned model can help account for this contact gap. We are developing that approach and will share it in a later release.
We check the signals during setup, then review the recorded motion and tactile measurements alongside the video. Each new wearer needs to contribute usable recordings, not simply be able to put on the glove.
We have started mass production. We are preparing to put the same collection hardware in more people’s hands and record a wider range of tasks, objects, and working conditions.
Video · Fitting different hands
Co-designing capture and execution
These requirements shaped the glove and robot hand together. People need to perform naturally, and the movements we record need to correspond to something the robot can command. UMI brings that correspondence into the collection interface. For S1, we build it into the joint arrangement and sensing hardware.
Matched kinematics
We gave the glove and robot hand the same joint arrangement. Each measured glove angle becomes a position target for the corresponding robot joint. We do not need a geometric retargeting solver for these finger movements. The mapping stays the same across tasks.
Using inverse kinematics to match fingertip targets leaves the solver to choose how each joint bends. The individual joint movements in a demonstration may be lost. Our mapping preserves direct control of each joint. The video shows us moving the MCP, PIP, and DIP joints independently, at the knuckle, middle, and tip of each finger. This control matters for tasks such as writing, where a finger may need to hyperextend.
Video · Co-designed kinematics
Aligning tactile measurements
We use the same tactile sensors on the glove and robot hand. The fingertips still have different shapes, and hand size varies between wearers. Even at matching joint angles, a finger may touch a different part of the object or distribute pressure differently.
DexUMI preserves the target robot hand’s joint-to-fingertip mapping. We allow more variation so different people can wear the glove. Our contact-alignment model is intended to account for the resulting differences; joint correspondence alone does not resolve them.
A million hours of what? Human skill recorded with enough detail to learn from, and a clear correspondence to the robot that will do the work. That is what we mean by high-quality dexterous data at scale, and why we design capture and execution together.
What comes next
Recording human skill also sets a demanding target for the robot hand. The natural speed and dexterity of these movements remain difficult for robotic hands to reproduce. That challenge shaped our second-generation hand. We’ll share more soon. Stay tuned.
Interested in collecting with S1? Request access to S1 to hear about availability.