Industry · Egocentric data
Egocentric Video Data for Surgical Robotics
A surgical video dataset for egocentric AI is first-person, head-mounted footage of a clinician performing a procedure — the surgeon's own view of hands, instruments, and the operative field. The open egocentric-surgical corpora (EgoSurgery-Phase, EgoExOR) are small and research-only, the laparoscopic sets everyone reaches for are tool-POV rather than head-mounted, and no commercially-licensed egocentric surgical dataset exists as of 2026 — so consented, de-identified custom capture is the only clean route to training data.
Quick facts
- Resolution
- 1080p @ 30fps baseline; stereo 2160p @ 60fps optional
- Field of view
- ≥120° horizontal — the full operative field and both hands in frame
- Mount
- Head-mounted (surgical-loupe or glasses rig); never chest-mount or fixed tripod, which lose the hands and instrument tips
- Sensors
- RGB (baseline), IMU (head), Gaze (optional, attention-to-instrument), Depth (optional, instrument/tissue geometry)
- Labels
- Frame-aligned surgical-phase segments; Instrument presence and tool-exchange spans; Hand-tool interaction and grasp/contact state
- Volume
- 40–120 accepted hours per program
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos
15 hours, 9 surgical phases. EgoSurgery-Phase is the first publicly available real open-surgery egocentric video dataset for surgical phase recognition — 15 hours spanning 9 distinct phases, captured with an egocentric camera attached to the surgeon's head, and released with eye gaze.
EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding
94 minutes, 84,553 frames. EgoExOR is the first operating-room dataset to fuse first- and third-person views — 94 minutes (84,553 frames) of two emulated spine procedures pairing egocentric RGB, gaze, hand tracking and audio with exocentric RGB-D and ultrasound, annotated with 568,235 scene-graph triplets.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
What surgical robotics needs from egocentric data
A surgical video dataset for egocentric AI is first-person, head-mounted footage of a clinician performing a procedure — the surgeon's own view of hands, instruments, and the operative field. The open egocentric-surgical corpora (EgoSurgery-Phase, EgoExOR) are small and research-only, the laparoscopic sets everyone reaches for are tool-POV rather than head-mounted, and no commercially-licensed egocentric surgical dataset exists as of 2026 — so consented, de-identified custom capture is the only clean route to training data.
The capture settings this covers:
- A surgeon's first-person view down onto the operative field — both hands, the active instrument, and the tissue plane held in one frame through an open procedure.
- Dry-lab drills captured head-mounted: suturing, knot-tying, and needle-driver handling on a bench trainer for skill-assessment models.
- Cadaver-lab or veterinary-surgery instrument handling — passing, loading, and exchanging tools across a full task cycle where real-OR access is impossible.
- Phase transitions a recognition model must segment: the hand-offs between exposure, dissection, hemostasis, and closure.
- Bimanual instrument coordination — one hand retracts or steadies while the other cuts, sutures, or cauterizes.
- Robotic-console teleoperation shot from the surgeon's-eye view, pairing operator intent with instrument motion for surgical-copilot policies.
Why surgical robotics needs first-person human video
Apple's EgoDex pretrains dexterous manipulation on large-scale egocentric human video, so a clinician's first-person view of instrument handling sits in the same training substrate a surgical-manipulation policy learns from. [1]
EgoLive shows large-scale egocentric human demonstrations of real-world tasks lifting manipulation policies, the mechanism by which consented head-mounted operating-room footage can train surgical-copilot and instrument-tracking models. [2]
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to the data scarcity a consented surgical corpus is built to close. [3]
Capture and delivery spec
Every surgical robotics capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps baseline; stereo 2160p @ 60fps optional |
| Frame rate | 30fps baseline; 60fps for fast instrument motion and re-grasps |
| Field of view | ≥120° horizontal — the full operative field and both hands in frame |
| Mount | Head-mounted (surgical-loupe or glasses rig); never chest-mount or fixed tripod, which lose the hands and instrument tips |
| Sensors | RGB (baseline), IMU (head), Gaze (optional, attention-to-instrument), Depth (optional, instrument/tissue geometry) |
| Labels | Frame-aligned surgical-phase segments; Instrument presence and tool-exchange spans; Hand-tool interaction and grasp/contact state; Optional gaze-to-instrument attention; De-identification / PHI-redaction pass log |
| QA gates | Hands and instruments in frame above threshold on every action event; Stability and no motion blur on contact and exchange frames; De-identification pass (faces, PII, patient identifiers, screen text); Per-clip consent + facility-release artifact attached |
| Delivery | H.265 clips + per-clip JSON metadata (phase, instrument, contact events), HIPAA/GDPR-grade provenance chain, with a consent artifact attached to every clip |
| Volume | 40–120 accepted hours per program |
Open surgical robotics datasets
The 5 open corpora most relevant to surgical robotics are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EgoSurgery-Phase | ~15 h of real open-surgery egocentric video, phase-labeled | Head-mounted RGB + surgical-phase labels | Research-only | No | The one corpus of real head-mounted open surgery — but small, research-only, and shipped with no consent chain a commercial buyer can rely on. |
| EgoExOR | ~94 min · ~84k frames across two simulated spine procedures | Ego RGB / gaze / hand + exo RGB-D + ultrasound | Research-only | No | Rich multimodal OR capture, but the procedures are simulated (not live-OR) and the terms are research-only — a benchmark, not training fuel. |
| EndoNet / Cholec80 | 80 laparoscopic cholecystectomy videos, tool + phase annotated | Laparoscope tool-POV (NOT head-mounted egocentric) | Research-only | No | The reflex choice for surgical video, but it is a laparoscope camera geometry — it does not transfer to a surgeon's head-mounted first-person view. |
| EgoExoLearn | ~120 h of paired ego/exo procedural-following demonstrations | Ego + exo RGB | Research-only | No | Proves the paired-view procedural format works, but the domain is everyday and lab tasks — not surgery — so contact and tool distributions don't match. |
| Ego4D (health-adjacent slices) | A subset of the 3,670 h corpus (care and clinical-adjacent scenes) | Head-mounted RGB (+ optional modalities) | Ego4D signed data-use agreement | Conditional | Head-mounted and large, but not surgical — and the signed license adds friction most surgical-AI teams cannot clear for a shipping product. |
Open datasets vs Truelabel custom capture
Consent and de-identification is a harder gate than license here. Operating-room footage shows patients, faces, and PHI, so provenance is not optional — custom capture ships a per-clip consent and facility-release chain that EgoSurgery-Phase and the other research corpora never document.
The open egocentric-surgical corpora are tiny and research-only. EgoSurgery-Phase is roughly 15 hours of real open surgery and EgoExOR is under two hours of simulated spine procedures — enough to benchmark phase recognition, nowhere near enough to train a shipping policy, and neither carries commercial rights.
The laparoscopic corpora everyone reaches for aren't egocentric. Cholec80's 80 cholecystectomy videos are a laparoscope tool-POV, not the surgeon's head-mounted first-person view — a different camera geometry that doesn't transfer to head-mounted surgical assistants or skill-assessment models.
Real-OR access is the wall, and simulated capture climbs it. Dry-lab, cadaver-lab, and veterinary-surgery capture to your brief sidesteps the hospital-access and patient-consent bottleneck while still delivering the instrument-handling, phase, and hand-tool interaction signal — adjacent procedural-following corpora like EgoExoLearn prove the paired-view format but not the surgical domain.
Surgical robotics: by the numbers
The figures below are specific to surgical robotics egocentric data and anchor the comparisons above.
- EgoSurgery-Phase: ~15 hours of real open-surgery egocentric video across labeled surgical phases
- EgoExOR: ~94 minutes / ~84k frames of paired ego-exo operating-room capture across two simulated spine procedures
- Cholec80: 80 laparoscopic cholecystectomy videos — tool-POV, not head-mounted egocentric
- Zero commercially-licensed egocentric surgical corpora as of July 2026
- "surgical video dataset" — the tranche's only MEDIUM-competition Ads term (DataForSEO, 2026-07-06)
How Truelabel captures surgical robotics data
Truelabel runs surgical robotics programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–120 accepted hours per program, delivered as H.265 clips + per-clip JSON metadata (phase, instrument, contact events), HIPAA/GDPR-grade provenance chain, with a consent artifact attached to every clip. Go deeper via what egocentric data is, egocentric data licensing, per-clip consent artifacts, data provenance chain, teleoperation data, and VLA training data.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to pretrain dexterous manipulation from first-person human video.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used to lift manipulation policies.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to embodied-AI data scarcity.
arXiv ↩ - EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos
EndoNet / Cholec80 is a dataset of 80 laparoscopic cholecystectomy videos annotated for tools and phases, captured from a laparoscope tool-POV rather than a head-mounted egocentric view.
arXiv - EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
EgoExoLearn provides paired egocentric and exocentric video of procedural-following demonstrations across everyday and lab tasks.
arXiv
FAQ
How is patient and clinician consent and de-identification handled for OR footage?
Every clip carries a consent artifact tied to the wearer and, where patients are in frame, a facility release; a de-identification pass redacts faces, patient identifiers, and on-screen PHI before delivery, and the whole chain is auditable per clip. This is the provenance the research corpora rarely document, and it is the first thing surgical-AI legal teams ask about.
Can you capture in real ORs, or only simulated and dry-lab procedures?
Both, scoped to what access allows. Dry-lab drills, cadaver-lab work, and veterinary surgery need no patient consent and sidestep the hospital-access wall, so most programs start there; live-OR capture is possible with the right facility agreements and patient consent, and the same head-mounted spec and QA gates apply either way.
What annotations ship with surgical egocentric clips?
Frame-aligned surgical-phase segments, instrument presence and tool-exchange spans, hand-tool interaction and grasp/contact state, and optional gaze-to-instrument attention — the label stack a phase-recognition or skill-assessment model consumes directly, delivered in your episode format rather than reverse-engineered from a research paper's schema.
Why not just train on EgoSurgery-Phase, EgoExOR, or Cholec80?
Because none of them is both egocentric and commercially usable. EgoSurgery-Phase is a small research-only corpus, EgoExOR captures simulated procedures under research terms, and Cholec80 is a laparoscope tool-POV rather than a head-mounted view. They are excellent for prototyping and benchmarking; a shipping surgical model needs footage you hold the rights to and that matches your camera geometry.
Do you support HIPAA/GDPR-grade delivery chains?
Yes. Delivery is H.265 plus per-clip JSON with a de-identification log and a provenance manifest, handled under HIPAA/GDPR-grade retention and consent rules you define. Where footage cannot be de-identified to your standard, it doesn't ship.
Why does surgical robotics need first-person human video at all?
Because the field is moving from teleoperation-only demonstrations toward learning dexterity from first-person human video, and head-mounted surgical capture is the way to get that signal for phase recognition, instrument tracking, and surgical-copilot policies without a data rig on every operating table.
Looking for surgical video dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request surgical egocentric data