Robotic manipulation is one of the hardest data collection problems in machine learning. A language model trains on text that already exists in abundance on the internet. A vision model trains on images that can be scraped, filtered, and labeled at scale. A manipulation policy is trained on demonstration trajectories that must be collected physically, one episode at a time, in real environments or in carefully constructed simulations. The data does not exist until someone generates it, and generating it well requires decisions about hardware, collection protocol, scene diversity, and annotation standards that determine whether the resulting model generalizes or merely overfits to the collection setup.
The field is moving fast. Cross-embodiment datasets pooling trajectories across dozens of robot platforms and hundreds of task types have demonstrated that scale and diversity drive generalization in manipulation learning. The annotation standards and quality control processes that turn raw demonstration data into training-ready episodes, however, receive less attention than the collection infrastructure and the model architectures trained on the resulting data.
This blog covers what a production-grade manipulation dataset actually requires: how demonstration data is collected, what annotations need to be captured, and how quality control prevents the failures that only surface at deployment. Physical AI data services and video annotation services are the two capabilities most directly involved in building manipulation datasets that produce policies capable of generalizing beyond the collection environment.
Key Takeaways
- Demonstration data for robotic manipulation must be collected physically or in high-fidelity simulation. The annotation and quality control standards applied to that data determine whether the resulting policy generalizes to deployment environments.
- Episode-level annotation captures what a trajectory is trying to accomplish. Frame-level annotation captures the precise state of the robot and environment at each timestep. Both are required for manipulation policy training.
- Task success is not a binary label in manipulation. Policies trained on coarse success labels learn behaviors that pass the label criteria but fail on the physical variation that deployment introduces.
- Scene diversity and object diversity are as important as demonstration volume. A dataset of ten thousand demonstrations from five object types in one scene will produce a more brittle policy than a smaller dataset with genuine scene and object coverage.
- Human-in-the-loop quality control is not optional for manipulating data. Annotators who cannot recognize whether a grasp is stable or a placement is physically viable will pass failures into the training set that automated filters cannot catch.
What Demonstration Data for Manipulation Requires
The Collection Problem
Demonstration data is collected by recording a robot or a human operator performing a manipulation task: picking an object, placing it in a target location, assembling components, or executing a multi-step task sequence. Each recorded episode captures the sequence of observations and actions that produced a successful or unsuccessful task execution. The policy trained on this data learns to reproduce the demonstrated behavior in new environments.
The quality of the resulting policy is directly determined by the quality and diversity of the demonstrations. Demonstrations collected in a single lab environment with a fixed set of objects produce policies that generalize poorly to new environments, new objects, and new lighting conditions. Demonstrations collected at scale across varied environments, object sets, and task configurations produce policies with the coverage required for deployment.
Teleoperation vs. Autonomous Collection
Most high-quality manipulation demonstration data is collected through teleoperation: a human operator controls the robot in real time to execute tasks, and the operator’s control inputs and the resulting robot states are recorded as the training data. Teleoperation produces high-quality demonstrations because a skilled operator can navigate the physical contingencies that arise in real environments, recover from near-failures, and complete tasks in ways that purely autonomous collection cannot replicate.
The annotation burden for teleoperated demonstrations differs from autonomous collection. Teleoperated episodes need annotation that captures the operator’s intent at each stage of the task, the task completion status at the episode level, and the quality of the physical execution at the frame level. Data collection and curation services that include structured teleoperation protocols alongside the annotation pipeline produce demonstration datasets where the collection and annotation stages are designed together rather than sequenced independently.
Episode-Level and Frame-Level Annotation
What Episode-Level Annotation Captures
Episode-level annotation assigns labels to each recorded demonstration as a complete unit. The minimum required fields are task identity, task success or failure, and a brief description of what the episode was attempting to accomplish. For manipulation policies trained on language-conditioned or instruction-following architectures, natural language task descriptions are a required component of every episode annotation, not an optional enrichment.
Episode-level annotation also captures metadata about the collection context: the scene configuration, the object set used, the robot platform, and any environmental conditions that vary across the collection. This metadata is what allows the training pipeline to balance the dataset across scene types, object categories, and task types rather than training on whatever distribution the collection produced by default.
What Frame-Level Annotation Captures
Frame-level annotation assigns labels to individual timesteps within an episode. For manipulation tasks, the critical frame-level labels are object state, end-effector state, contact state, and task phase. Object state captures whether an object is grasped, in motion, or at rest, and in what configuration. End-effector state captures gripper aperture, contact forces where available, and the spatial relationship between the end-effector and the target object or surface.
A frame-level annotation record for a single timestep typically looks something like this in practice:
{“timestep”: 142, “object_state”: {“id”: “mug_03”, “status”: “grasped”, “pose”: [x, y, z, qx, qy, qz, qw]}, “end_effector_state”: {“gripper_aperture”: 0.018, “contact_force”: [0.4, 0.1, 2.3], “distance_to_target”: 0.002}, “contact_state”: “stable_contact”, “task_phase”: “transport”}
Each field maps to a specific training signal. The object_state.status field, which moves from approaching to grasped to released across an episode, is what lets a model learn the discrete state transitions a task moves through. The contact_force vector inside end_effector_state is what distinguishes a firm, centered grasp from one that is barely holding the object, a distinction that a binary success label cannot make. The task_phase field is the value that drives the phase-weighted loss described below: a training pipeline can assign higher loss weight to frames labeled grasp or place than to frames labeled transport, because errors during contact-rich phases are more consequential than errors during free-space movement.
Task phase annotation divides an episode into labeled stages: approach, grasp, transport, place, and release. Phase labels are what allow the training pipeline to apply different loss weighting to different stages of the task, which matters because the failure modes associated with the approach are different from those associated with grasp, and a model that weights all phases equally will underfit the phases where precision is highest.
Fine-grained frame-level annotation requires annotators who understand the physical mechanics of the manipulation task being annotated. An annotator who cannot distinguish a stable grasp from a marginal one will label marginal grasps as successful, introducing systematic failures into the training set at exactly the points where the policy needs the most reliable supervision signal. Video annotation services that include domain expertise in robotic manipulation mechanics produce frame-level annotations that reflect the physical realities of the task rather than surface-level pattern matching on the visual appearance of the episode.
Quality Control for Manipulation Datasets
Why Automated Filters Are Not Enough
Automated quality control for manipulating data can filter obvious failures: episodes with missing sensor modalities, episodes that end before the task window closes, and episodes with out-of-range sensor readings. What automated filters cannot catch is the physically marginal case: a grasp that looks successful in the video but is held with a contact configuration that would drop the object under any perturbation, a placement that appears to reach the target but is not mechanically stable, a task completion that passes the binary label criteria but executes in a way that will not generalize to slightly different object weights or surface textures.
These marginal cases are the ones that cause policies to fail at deployment. A policy trained on a dataset where ten percent of ‘successful’ demonstrations are physically marginal will learn a behavior that reproduces that margin. In deployment, where the physical environment is not a controlled collection setup, the marginal behavior fails consistently.
Human-in-the-Loop Review for Physical Validity
Human quality review for manipulating data requires annotators with physical intuition about the tasks being reviewed. The reviewer needs to be able to watch a manipulation episode and identify whether the grasp is stable, whether the object placement is physically viable, whether the robot’s approach trajectory would generalize to a slightly different object position, and whether the task completion would survive the perturbations the deployment environment will introduce.
This is a fundamentally different skill requirement from text annotation or image classification. It requires annotators who have either direct experience with robotic manipulation or strong physical intuition developed through adjacent domains. Review teams staffed with general-purpose annotators produce quality control that catches visual anomalies but passes physical failures.
Dataset Balance and Coverage Auditing
A manipulation dataset can pass individual episode quality checks while still being unbalanced in ways that produce brittle policies. If ninety percent of demonstrations use objects of similar weight, size, and texture, the policy learns a manipulation behavior calibrated to that distribution. Object diversity at the dataset level requires deliberate coverage auditing: checking that the final dataset includes adequate representation across object categories, size ranges, texture types, and scene configurations before training begins. Data collection and curation services that include dataset-level coverage auditing as a standard component of the curation process produce training datasets with the balance that generalizable manipulation policies require.
How Digital Divide Data Can Help
Digital Divide Data supports robotics teams building manipulation training datasets across the full data pipeline, from collection protocol design through annotation and quality control. For programs collecting teleoperation demonstration data, physical AI data services cover collection protocol design, scene and object diversity planning, and the structured teleoperation workflows that produce demonstrations with consistent annotation coverage across task phases.
For programs annotating collected manipulation episodes, video annotation services provide domain-aware annotation teams capable of applying frame-level labels for object state, end-effector state, contact state, and task phase, with quality control processes designed to catch physically marginal demonstrations rather than only visual failures. For programs evaluating whether collected datasets produce the policy generalization their deployment requires, model evaluation services design evaluation frameworks built around the deployment environment rather than the collection environment.
If your manipulation dataset collection program does not have annotation standards and quality control processes designed for physical validity, the policy failures at deployment will trace back to the dataset. Talk to an expert.
Conclusion
Manipulation of the dataset quality is determined at collection and annotation, not at training. The diversity decisions made during collection, the annotation standards applied to each episode and frame, and the quality control processes that distinguish physically valid demonstrations from marginal ones are what separate training datasets that produce generalizable policies from those that produce policies that work in the lab and fail in deployment.
The field has demonstrated that scale and diversity drive generalization in manipulation learning. Building datasets with the scale and diversity that generalization requires means treating collection, annotation, and quality control as an integrated program rather than three sequential steps.
Before your next collection run, it is worth checking your pipeline against four specific questions:
- Does every episode carry frame-level labels for object state, end-effector state, contact state, and task phase, or only episode-level success and failure?
- Has anyone audited your dataset’s scene and object distribution against your actual deployment targets, or only against what was convenient to collect?
- Are your human reviewers screening for physically marginal grasps and placements, or only for visually obvious failures?
- And if you are building toward cross-embodiment training, is your action schema aligned to a shared standard now, before conversion becomes the expensive afterthought it always becomes later?
A pipeline that cannot answer all four with a clear yes has a specific, fixable gap rather than a vague data quality problem.
References
Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., et al. (2024). DROID: A large-scale in-the-wild robot manipulation dataset. arXiv:2403.12945. https://arxiv.org/abs/2403.12945
O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. (2024). Open X-Embodiment: Robotic learning datasets and RT-X models. In the IEEE International Conference on Robotics and Automation. https://arxiv.org/abs/2310.08864
Belkhale, S., Cui, Y., & Sadigh, D. (2023). Data quality in imitation learning. In Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2306.02437
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024). π0: A vision-language-action flow model for general robot control. arXiv:2410.24164. https://arxiv.org/abs/2410.24164
Frequently Asked Questions
Q1. How many demonstration episodes does a manipulation policy need to generalize reliably?
There is no universal number because the required volume depends on task complexity, scene diversity, and object diversity. A policy trained on a narrow task with limited object variation can generalize adequately with hundreds of demonstrations. A policy intended to generalize across many object types, scene configurations, and task variations needs thousands of diverse demonstrations. Cross-embodiment datasets have shown that diversity drives generalization more reliably than raw volume. A smaller dataset with genuine coverage across scene types and object categories will typically produce a more capable policy than a larger dataset with narrow coverage.
Q2. What is the difference between episode-level and frame-level annotation, and when is each required?
Episode-level annotation labels the demonstration as a whole: task identity, success or failure, natural language task description, and collection metadata. Frame-level annotation labels individual timesteps: object state, end-effector state, contact state, and task phase. Episode-level annotation is required for all manipulation datasets. Frame-level annotation is required for policies trained with dense supervision signals, such as those using imitation learning with phase-weighted loss, or for policies trained on instruction-following architectures where the relationship between natural language commands and physical states needs to be captured at a fine-grained level.
Q3. How do you identify physically marginal demonstrations during quality control?
Physically marginal demonstrations require reviewers with physical intuition about the manipulation task. The indicators include grasps where the object is held at the edge of the gripper contact surface rather than centered, placements where the object is technically at the target but in a mechanically unstable configuration, approach trajectories that would require unusually precise alignment to replicate, and task completions that depend on specific surface friction properties. Automated filters that check for sensor completeness and episode duration will not catch these. Human review by annotators familiar with manipulation mechanics is the only reliable method.
Q4. How should object and scene diversity be planned before collection begins?
Define the object categories, size ranges, texture types, and weight classes that the policy needs to handle in deployment. Design the collection protocol to sample deliberately across those dimensions rather than collecting whatever is convenient. For scene diversity, identify the surface types, lighting conditions, and environmental configurations that the deployment context will include and ensure each is represented in the collection. Audit the dataset against these coverage specifications before training begins, not after. Discovering coverage gaps after training requires collecting more data, which is expensive. Discovering them before training allows the collection protocol to be adjusted.
Q5. What annotation format is required for cross-embodiment training?
Cross-embodiment training requires annotation that captures robot-agnostic task descriptions alongside robot-specific action and state data. The task description must be in natural language and describe what the episode accomplishes rather than how the specific robot executed it. The action and state data must be formatted in a standard schema that allows trajectories from different robot platforms to be combined in the same training batch.
Udit Khanna leads the delivery of scalable AI and data solutions at Digital Divide Data, with a deep specialization in Physical AI. With a background in presales, solutioning, and customer success, he brings a mix of technical depth and business fluency, helping global enterprises move their AI projects from prototype to real-world deployment without losing momentum.
