Task · Egocentric data
Egocentric Tool Use Datasets
A tool use dataset captures first-person footage of hands grasping and using tools — drivers, wrenches, gauges, blades, probes — with the tool-in-use spans and tool-action verbs labeled alongside the RGB. Most annotated tool-use corpora are non-commercial, research-only, or toy-tool proxies; HoloAssist is the one near-permissive (CDLA-Permissive-style) exception, so custom capture with your tool set and taxonomy is the route to commercially-clean tool-handling training data.
Quick facts
- Resolution
- 1080p @ 30fps baseline; 2160p for close-range gauging and fine tool work
- Field of view
- ≥120° horizontal — keeps both hands and the tool in frame through the reach and the action
- Mount
- Head-mounted (glasses, cap, or GoPro-style rig); never chest-mount or tripod — the camera must follow gaze onto the tool tip
- Sensors
- RGB (baseline), IMU — head, optional wrist for tool swing/torque motion, Depth (optional, for tool-tip-to-target distance), Gaze (optional, to mark attention on the working point)
- Labels
- Tool-in-use temporal spans (tool present and active vs idle/holstered); Tool class and instance (driver, wrench, gauge, blade, probe); Tool-action verb (drive, torque, gauge, cut, pry, measure)
- Volume
- 40–160 accepted hours per tool set / task program
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
166 hours, 350 pairs. HoloAssist is a large-scale egocentric human-interaction dataset — 166 hours captured by 350 unique instructor-performer pairs on a mixed-reality headset with seven synchronized data streams — built for interactive AI assistants that guide people through real-world physical tasks.
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities
4,321 videos, 8 exo + 4 ego views. Assembly101 is a multi-view procedural-activity dataset of 4,321 videos of people assembling and disassembling 101 take-apart toy vehicles, recorded simultaneously with 8 static and 4 egocentric cameras and annotated with over 1M fine-grained action segments and 18M 3D hand poses.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
What tool use data contains
A tool use dataset captures first-person footage of hands grasping and using tools — drivers, wrenches, gauges, blades, probes — with the tool-in-use spans and tool-action verbs labeled alongside the RGB. Most annotated tool-use corpora are non-commercial, research-only, or toy-tool proxies; HoloAssist is the one near-permissive (CDLA-Permissive-style) exception, so custom capture with your tool set and taxonomy is the route to commercially-clean tool-handling training data.
The capture settings this covers:
- Power-driver fastening: picking up a cordless driver, indexing the bit onto a screw, driving a run of fasteners, then holstering the tool.
- Torque work: seating an open-end or torque wrench on a nut, pulling to spec, and the click-and-release at the end of the pull.
- Gauging and measuring: reading calipers, a bore gauge, or a multimeter probe onto a target surface and lifting it back off.
- Tool changeover: swapping from a driver to a wrench to a pick without setting the workpiece down — the handling in the gap between tools.
- Cutting and trimming: running a blade, snips, or a rotary tool along a line, including the approach, the cut, and the finish stroke.
- Two-person tool handover during a repair — one worker passes a tool and confirms grip before releasing it.
Why robotics and AI labs need tool use data
Apple's EgoDex pretrains dexterous manipulation on large-scale egocentric human video, the first-person tool-in-hand footage a tool-use policy learns from. [1]
EgoLive shows large-scale egocentric human demonstrations of real-world tasks lifting manipulation policies, so consented head-mounted footage of tool use directly improves the robot skill it targets. [2]
EgoScale found dexterous-manipulation performance scaling log-linearly with the volume and diversity of egocentric human data, so covering a broad tool set across many operators and grips is a measurable ROI lever, not busywork. [3]
Capture and delivery spec
Every tool use capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps baseline; 2160p for close-range gauging and fine tool work |
| Frame rate | 30fps baseline; 60fps for fast fastening, cutting, and tool changeovers |
| Field of view | ≥120° horizontal — keeps both hands and the tool in frame through the reach and the action |
| Mount | Head-mounted (glasses, cap, or GoPro-style rig); never chest-mount or tripod — the camera must follow gaze onto the tool tip |
| Sensors | RGB (baseline), IMU — head, optional wrist for tool swing/torque motion, Depth (optional, for tool-tip-to-target distance), Gaze (optional, to mark attention on the working point) |
| Labels | Tool-in-use temporal spans (tool present and active vs idle/holstered); Tool class and instance (driver, wrench, gauge, blade, probe); Tool-action verb (drive, torque, gauge, cut, pry, measure); Target/fastener state before and after the action; Grasp type on the tool handle; Optional 21–25-keypoint 3D pose on the tool-holding hand |
| QA gates | Tool and both hands in frame during the entire action span; No motion blur on the tool-contact frames; Tool clearly identifiable — not fully occluded by the hand at the key moment; FOV and horizon check; Per-clip wearer consent artifact attached |
| Delivery | H.265 clips + per-clip JSON metadata (tool class, in-use spans, action verbs, hand pose), Hugging Face-streamable, with a consent artifact attached to every clip |
| Volume | 40–160 accepted hours per tool set / task program |
Open tool use datasets
The 5 open corpora most relevant to tool use are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| HoloAssist | ~166 h · two-person interactive assembly & repair · HoloLens 2 | Egocentric RGB + eye/hand tracking + instructor dialog | CDLA-Permissive-style research | Conditional | The one near-permissive tool corpus here, but tasks are staged instructor-follower repairs, not autonomous single-operator tool work; confirm the terms before any commercial use. |
| Assembly101 | 101 toy-vehicle (dis)assembly sequences · ego + multi-view exo | RGB ego + exo, plus 3D hand poses | Non-commercial research | No | The tool actions are on toy vehicles with a toy screwdriver, so the torque and cam-out don't match a real driver — and the NC license bars a shipping product. |
| MECCANO | Single toy-motorbike build · screwdriver + wrench actions | RGB + depth + gaze (multimodal) | Research-only | No | One toy build with one tool set; excellent for gaze-conditioned tool-use study, but research-only terms exclude commercial training. |
| Ego-Exo4D | 1,286+ h skilled activities incl. bike repair & cooking · ego + exo | Multi-view RGB + gaze + IMU + narrations | Signed research data license | No | Tool tasks like bike repair are a slice of a mixed-activity corpus, and the signed license restricts commercial training — strong for benchmarking, not for shipping. |
| EgoExoLearn | ~120 h asynchronous ego + exo procedure-following (lab, cooking) | Egocentric + exocentric RGB + gaze + narrations | Research license | No | Built to study following a demonstration, so the tool skills sit inside longer procedures; research license, commercial terms require negotiation. |
Open datasets vs Truelabel custom capture
Every open tool-use corpus is staged, toy, or license-locked. HoloAssist is the most permissive of them — a CDLA-Permissive-style license — but its tasks are staged instructor-follower repairs. Assembly101 and MECCANO film toy tools under non-commercial and research-only terms, and Ego-Exo4D and EgoExoLearn need signed research agreements. None of them gives you commercial rights to footage of your tools.
Toy tools don't carry the right physics. A plastic screwdriver on a toy motorbike has none of the torque, slip, or cam-out signature of a real impact driver on a steel fastener, so a policy trained on toy-tool contact won't transfer to the shop floor. Custom capture films your actual tools on your actual fasteners, materials, and clearances.
Open corpora freeze someone else's tool vocabulary. Custom capture lets you define the tool classes, the tool-in-use span schema, and the tool-action verbs your model predicts, delivered in your episode format instead of remapped from a research benchmark's label set.
Tool-use footage shows hands and often faces, so provenance isn't optional. Custom capture ships a per-clip consent chain and the option of exclusivity, so the same tool-handling footage isn't also sold to the lab you're racing.
Tool use: by the numbers
The figures below are specific to tool use egocentric data and anchor the comparisons above.
- HoloAssist: ~166 h of two-person repair on HoloLens 2 — the one CDLA-Permissive-style tool corpus, and still staged instructor-follower work
- Ego-Exo4D: 1,286+ hours of skilled tool activities (bike repair, cooking, health, music), captured ego + exo across 740+ participants
- EgoExoLearn: ~120 hours of asynchronous ego + exo procedure-following on skilled tool tasks
- "tool use dataset" ~10 searches/mo (DataForSEO, 2026-07-06) — a purely academic, unclaimed SERP
- TrueLabel tool-use programs: tool-in-use spans + tool-action taxonomy at 40–160 accepted hours per tool set, first pilot batch in days
How Truelabel captures tool use data
Truelabel runs tool use programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–160 accepted hours per tool set / task program, delivered as H.265 clips + per-clip JSON metadata (tool class, in-use spans, action verbs, hand pose), Hugging Face-streamable, with a consent artifact attached to every clip. Go deeper via what egocentric data is, egocentric data licensing, industrial egocentric video with tools in frame, imitation learning from human demonstrations, VLA training data, and teleoperation data.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to pretrain dexterous manipulation from first-person human video.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used to lift manipulation policies.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale reports dexterous-manipulation performance scaling with the volume and diversity of egocentric human data.
arXiv ↩ - HoloAssist: an Egocentric Human Interaction Dataset
HoloAssist captures two-person interactive assembly/repair tasks on HoloLens 2 under a CDLA-Permissive-style research license.
Microsoft Research - HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
The HoloAssist paper documents an egocentric human-interaction dataset of real-world assistive assembly and repair tasks.
arXiv - Assembly101: A Large-Scale Multi-View Video Dataset
Assembly101 is a procedural toy-vehicle (dis)assembly dataset with 3D hand poses under a non-commercial research license.
Assembly101 project - MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain
MECCANO is a research-only multimodal (RGB, depth, gaze) egocentric dataset of a single toy-motorbike build with screwdriver and wrench actions.
University of Catania (IPLAB) - Ego-Exo4D project site
Ego-Exo4D is a large multi-view skilled-activity dataset (including bike repair and cooking) released under a signed research data license.
ego-exo4d-data.org - EgoExoLearn: bridging egocentric and exocentric skill learning
EgoExoLearn bridges egocentric and exocentric skill learning with asynchronous procedure-following demonstrations of skilled tool tasks.
OpenGVLab - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
Apple's EgoDex learns dexterous manipulation from large-scale egocentric video with per-hand joint pose paired to RGB.
arXiv
FAQ
What is a tool use dataset?
It is first-person video in which the labels describe the tool, not just the hand: which tool is in frame, the temporal span where it is actively being used, the action verb (drive, torque, gauge, cut), and the target's state before and after. Tool-use data is what grasp-and-use and VLA policies learn tool handling from, and the same first-person footage increasingly feeds humanoid manipulation pretraining.
Which open tool-use corpus is actually commercially usable?
Realistically none, cleanly. HoloAssist is the one near-permissive exception — a CDLA-Permissive-style research license over roughly 166 hours of two-person repair on HoloLens 2 — but the tasks are staged instructor-follower repairs, so confirm the terms before shipping anything trained on it. Assembly101 and MECCANO are non-commercial or research-only, and Ego-Exo4D and EgoExoLearn require signed research agreements.
Can you match our specific tool set and tool taxonomy?
Yes — that is the whole point of custom capture. You give us the tools your robot will pick up — the impact driver, the torque wrench, the calipers, the blade — and we build the tool-class and tool-action taxonomy around them, then QA every clip against it. Open corpora can't: MECCANO and Assembly101 are locked to toy screwdrivers and plastic parts, so their tool set is fixed to the paper.
Do you label tool-in-use spans and hand pose alongside RGB?
Both. The baseline is head-mounted RGB with tool-in-use temporal spans, tool-class and tool-action labels, and the target state before and after. We add per-hand 21–25-joint pose on the tool-holding hand, head and wrist IMU, and depth on request — the same signal stack Apple's EgoDex used to learn dexterous manipulation from first-person video.
Why not just use Assembly101 or MECCANO?
Because they film toys. A toy screwdriver on a plastic motorbike has none of the torque, cam-out, or reaction force of a real driver on a steel fastener, so the tool-contact distribution won't transfer. And both are non-commercial, so even the toy footage is off-limits for a policy you plan to ship.
How much egocentric tool-use data do we actually need?
Enough to cover your tool set across operators and grips, then scale on the curve. EgoScale found dexterous-manipulation performance scaling log-linearly with the volume and diversity of egocentric human data, and Physical Intelligence has shown tool skills transferring from human first-person video into robot policies. Most tool-use programs start with a 40–160 accepted-hour pilot on a target tool set, then scale once the policy responds.
Looking for tool use dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request tool-use egocentric data