Author name: DDD

DDD Solutions Engineering Team

September 19, 2024

The autonomous driving industry is gaining momentum with big players like Tesla, Google, and Uber eyeing to achieve the ultimate goal: “full autonomy.” The global market for autonomous vehicles was about $42.3 billion in 2022 and is expected to grow at a CAGR of 21.9% from 2023 to 2030.

These cutting-edge vehicles have the ability to analyze the environment around them to navigate safely. For this innovative technology to replicate the human decision-making process, vast and diverse amounts of data are required.  To aid this process, a lot of development and funding has been pivoted towards data annotation services, which are critical for training autonomous vehicles to interpret and respond to their environments.

In this article, we cover the importance of data annotation in building autonomous vehicles and how it’s revolutionizing the industry.

Data Annotation in Autonomous Vehicles 

Most self-driving cars are taught to drive with trained data in the form of annotated images, bounding boxes, polygon annotation, semantic segmentation, and LiDAR annotation. This data is harnessed consistently to supplement any new or unique driving scenario. Annotated data for self-driving cars is vast in scope and is not confined to the common road scenario of traffic signal, pedestrian, and vehicle interaction.

Completing tasks such as ideation, categorizing, and annotating new objects constitute only 70%, after which detailed data annotations are required to build high-performing models as these models require both safety and regulation.

Techniques and Tools for Data Annotation in Autonomous Vehicles

A core component of developing autonomous vehicles involves ML data operation solutions to perfect their functionality and build safer and reliable autonomous vehicles.

In the context of autonomous vehicles, the components of data can be images, videos, sensor data, etc. Techniques and tools used to annotate these components are different and specialized. Data annotation primitives represent an extensive and complementary set of metadata that describe the important aspects of the content in which the labels exist. This allows easy sorting and filtering of the data.

Image Annotation

Many software programs like Amazon Mechanical Turk and Google’s Open Images dataset provide a ready-to-use schema for image annotation. These schemas help in classifying the objects present in the images according to their location, represented using conventions like bounding boxes or segmentation masks.

Video Annotation

Video annotation is even tougher than image annotation. As in the case of the sequence of frames, there is another dimension attached to it. Consequently, in addition to finding objects of interest, there is a need to label these same objects in consecutive frames and also label the inter-object relationships.

Sensor Data Annotation

In ADAS, many detectors are used like LiDAR, Radar, UV, or any additional advanced sensor. The data collected via these sensors need to be annotated precisely. For example, to generate 3D point clouds from LIDAR data of the vehicle’s surroundings, it is necessary to annotate the various elements present in the scene.

Synthetic dataset annotation

With synthetic data training, you can model any environment that is difficult to recreate physically. These programmatically created virtual simulations can add high-quality vehicles, pedestrian behavior, weather conditions, and obstacles to make AV performance more accurate and safe for human use.

Read more: Enhancing Safety Through Perception: The Role of Sensor Fusion in Autonomous Driving Training

Challenges in Data Annotation for Autonomous Vehicles

Autonomous vehicles’ performance is highly correlated with the amount and quality of data they are trained on. Training computer vision models for AV is challenging because of the amount of annotated data required. Training data is almost always one of the most critical factors in machine learning model performance.

The approximated number of annotated training images is in millions, and one scene can be produced at different times, seasons, and weather conditions. Additionally, annotating the pixel-wise image of 10-minute videos for only one scene can take up to 5 days. Annotated data gets captured at the time of prediction for facilitating the training and inference. The most popular applications rely heavily on real-time data annotations to attain the highest accuracy and degree of detail.

Data annotation in AV is under non-trivial challenges that are to be accomplished. Annotations should be performed in real-time, and highly diversified in terms of the background scene and weather conditions. Another obstacle in the driving scenario can consist of heavy dynamic regions of interest.

Accuracy has a vital role in dealing with the variation in obstacles, and authorization of different lanes at high speed. Controlled velocity is necessary to achieve better results on these heavy dynamic labels and time management. The drop in real-time labels reduces labor’s attention resulting in a downfall in the quality of the labels. Moreover, annotation should be provided in the sensor’s augmented aerial view obstacles so that the labels of stacked semantic categories can be easily differentiated from each other.

Apart from the period of these data capturing activities, these platforms heavily depend on GPS for car position and driver status annotation to bridge down the carter space position to real-world local demography. These systems are facilitated with the help of Unity, Radar, and Monocular stereo camera preprocessing.

Impact of High-Quality Data Annotation in Autonomous Vehicles

The availability of large volumes of expertly annotated data is the fundamental generator of the success of AI learning algorithms, the development of autonomous vehicles heavily depends on the data collection, curation, and organization that happens at the hands of data annotators.

This is critical for self-driving cars because the volume of their collected data is growing by the day, and the complexity of sensorial data at even a single time point is a lot for vehicle technology to manage without human guidance. Thus, data annotators are a necessary part of the autonomous vehicle industry and directly heavily impact vehicle performance and safety. For the same, the government budget allocations for research and development (GBARD) of the EU allocated $ 118.16 billion which represents 0.74% of the GDP of the EU for high-quality data and R&D into AVs. Data annotation is already paramount in ensuring the efficient application and robust development of self-driving technologies.

An AI that learns how to make decisions by studying driver inputs for lane changes, eye tracking for pedestrians, and brake pedal response to traffic can learn to make those same decisions without humans behind the wheel to correct any mistakes. The abstract inferences it can make about these patterns through data annotation have direct life-or-death impacts.

Final Thoughts

Data annotation has become an important industry in machine learning and AI in many applications, especially autonomous driving. It is poised for growth with AI and ML algorithms being increasingly used across various industries and expected to grow in many scientific domains, serving a broader array of fields.

In particular, data annotation for autonomous vehicles is likely to grow, presenting opportunities for development and innovation. Data annotation using AR or 3D techniques is used for automotive training data, annotating various scenarios/objects on images like stop signs, pedestrians, cars, etc.

One interesting direction for annotating data for autonomous vehicles may be a focus on 3D point clouds as a complementary technique to image-based annotation. With continued advancements in artificial intelligence and machine learning across computing, storage, networking, and technology platforms, data curation via annotation with this compute-intensive data is growing rapidly.

As one of the leading data annotation companies, we focus on providing comprehensive data annotation and labeling solutions for autonomous driving vehicles. You can book a free call with our experts to discuss your data annotation needs.

By Umang Dayal

August 21, 2024

Autonomous vehicles requires large quantities of sensory data, fueling the development of accurate and capable sensors. These sensors can be categorized depending on their sensing modalities, such as cameras, lidars, radars, ultrasonics, or microphones, and by their position in the car, such as in-car, on-vehicle, or external sensors.

Not all vehicles carry all sensor types, and the choice of what sensor to employ is influenced by operating conditions (e.g., inside cities, on highways) and technical and economic constraints.

Despite their differences, rendering sensor data intelligible to an autonomous driving agent, for example by annotating them with geometric shapes. These include bounding boxes, critical in the development of sensor-independent models for a multitude of tasks like object detection, semantic segmentation, optical flow, and pose estimation.

Common to all sensors is also the need to record the vehicle’s own state and position relative to the environment, whether for situational awareness, adaptive speed control, or navigation. Recording these signals often also requires a means to combine and synchronize the autonomous driving sensor streams.

Types of Autonomous Driving Sensors

When discussing annotation techniques for autonomous cars, it is crucial to mention the different types of sensors used in these vehicles. The data from these sensors is likely to prove useful in different ways and may have unique annotation techniques. Autonomous driving cars use various types of sensors: Lidar, Radar, and camera.

LiDAR

LiDAR sensors measure the distance to objects based on the travel time of laser signals, and the scan range of most LiDAR configurations includes 360° of horizontal field of view, 30° to 40° (and up to 45°) of vertical field of view, and can reach ranges of up to 300m.

LiDAR point clouds, consisting of coordinate data and the reflectance or intensity signal for every measured point, are typically used as a backbone for obstacle detection, feature extraction, and most mapping techniques. A notable disadvantage of LiDAR is the distribution of the point cloud over the sensor’s 3D range field, which follows a specific scan pattern.

One revolution in horizontal and vertical space produces a complete scan with a certain number of layers, but the limited measurement rate of each individual sensor leads to a low number of points per scan layer.

Three main challenges exist when working with LiDAR data.

• The reflectance measured with LiDARs is a function of the 3D object’s surface and the light intensity, an abrupt change in the 3D geometry (in areas such as vertical structures, car corners, and tackles) can cause low-intensity in-plane measurements, making these spots tougher to distinguish than off-plane objects in fog or dust.

• The high density of information is due to multiple measurements of planar and point structures that LiDAR devices can provide.

• 3D object boundaries in the point clouds are often more evident than within them, which is why center point offsets are used. Unfortunately, low-point density objects may have problems with the topological analysis, which will lead to ambiguity during the annotation processes.

Radar

Autonomous cars often utilize radar sensors to enable important key features, such as blind spot monitoring, cross-traffic alert, or adaptive cruise control. Its technical concept works fundamentally in real-world scenarios, no matter if it is dark or foggy, independent of other road users, and unaffected by environmental conditions. These features are currently in full industrial utilization.

The most important attribute of autonomous driving is a competent system that navigates complex, crowded urban settings. Annotating massive radar sensor data to document user preferences can amount to significant manpower efforts and serve to understand industrial development decisions.

Currently, only radar sensor rotation poses lead to the available and significantly increased degradation of sensor-specific low-level (box-long) result prediction. Available long-term radar annotations depend on the utilization of radar reflections caused by the environment’s nearby objects.

Camera

Cameras are vital sensors for ADAS applications, and many autonomous driving datasets consist of or contain camera data. Images are also a critical part of many annotation pipelines. Digital cameras capture color images that have a range of spatial resolutions and influence the speed at which the image data can be processed.

The majority of camera sensors in autonomous driving applications capture visible light. This creates the possibility of using a common sense and object recognition model that is already trained on visible light images. There is also significant literature on increasing the capability of cameras in different conditions and scenes. Additionally, there are reduced sensor requirements for LiDAR and radar, which makes the camera an attractive sensor choice in some applications.

There have been several advances in the automated annotation of camera images for autonomous vehicles. Perspective boxes are popular annotation types for camera images, and images are often taken prior to the greatest distance of a lidar or radar sensor. This is because they can then be used to help with other sensors’ depth estimation and data association.

The challenges of camera data for annotation include being closer to the ground (causing overlap), of the axis of the vehicle motion (causing relative motion articulation), trees and buildings that can make areas without visible labels, a lack of an explicit rotation signal, and a wide range of light conditions.

Challenges in Annotating Sensor Data

Annotating sensor data is an essential step in the development of intelligent systems. This step becomes particularly challenging in the context of autonomous driving for several reasons.

• The scale at which data is collected in autonomous driving leads to a large volume that humans alone cannot fully process.

• A wide variety of sensor inputs are involved and should be annotated. Their diversity covers inputs from cameras, Light Detection And Ranging (LiDAR) sensors, Global Navigation Satellite System (GNSS) modules, as well as environmental information such as semantic maps, road furniture, and events. Furthermore, while cameras are widely employed as peripheral sensors in autonomous cars, no limiting factor compels other sensor modalities to be used.

• The content, in the form of objects and events, must be accurately annotated as it is classifiable and used as input for decision-making. Finally, these elements, especially 2-D bounding boxes for object detection, characterize a 3-D world mapped onto 2-D space and encapsulate sensor noise, a characteristic of sensory data.

Problems mainly arise from variations in the collected data. Some categories in the provided MOD are inadequate, especially defects (errors and omissions) that incur quality costs such as missed obstacles, and crashes, making the data useless for supervised learning.

Sensor input data is often noisy or extremely time-consuming to overwrite during annotation review. Due to the inherent diversity of sensor data, some annotations are not even possible. Overall, missing, mislabeled, or low-quality annotations generate non-representative data that bias model training and degrade the model’s performance.

Thus, it is important to improve annotation quality and assess annotation consistency thoroughly before using a different annotation system to implement and test semi-automated annotation approaches.

Read More: Enhancing Safety Through Perception: The Role of Sensor Fusion in Autonomous Driving Training

Traditional Annotation Methods

There are a number of traditional road car sensor streams like structured data such as internal state variables, structured outputs from image processing pipelines, etc., with typically annotation-based approaches to generating them. The manual annotation is not usually feasible with respect to computational cost and/or the inconsistent agreement between different annotators.

Rule-based systems that can infer these structured outputs directly from raw sensor data are not available. Vision-based internal function estimation is still an open issue, despite significant attempts in computer graphics, computer vision, and machine learning.

The level of annotation can be segregated into manual, semi-automatic, automatic, and hyper-automatic. The last term refers to a version of the automatic annotation by which unsupervised learning methods are able to annotate the video according to the same criteria used by humans.

Many video perception systems require a clear description of the experimental conditions (targets, perspectives, brightness) and a clear instruction set for the annotation task. It is known that the performance of automatic methods strongly depends on the level of detail needed for the description, with a forward dependency from the confidence threshold required to successfully assess the answers to the annotation.

As an example, a simple annotation of car equipment is easily carried out by end-users by only considering the evidence of a boundary of the windshield in the captured images. After this step, more specific driving lanes can be clearly defined as identified homogeneous-color pixels, leading to a form of initial segmentation.

Advanced Annotation Techniques

Even with the latest annotation technology, many companies are either using or experimenting with various state-of-the-art annotation tools that can provide a more precise way of solving specified autonomy requirements. These tools employ clever machine learning or computer vision algorithms to achieve difficult annotations. It’s one of the primary pioneers in using this technology to support advanced ADAS and automated vehicle programs.

Today, these tools are built into a large-scale machine learning platform that enables an end-to-end ML training pipeline with advanced support for annotating vast datasets of many different sensor types and ADAS features.

Read More: How Image Segmentation and AI is Revolutionizing Traffic Management

Conclusion

The development of autonomous driving cars has proved to be highly complicated, partly due to the closed-loop system where perception and reasoning abilities rely on a high-level understanding of vehicle motion and complex surrounding scenes. More importantly, robust autonomous driving algorithms with different sensors actually rely on high-quality, large-scale, sensor-specific annotation datasets. In summary, the techniques for annotating different sensor data from autonomous vehicles would be an essential development for future self-driving vehicle use cases.

Object detection and instance segmentation are among the most popular computer vision tasks and form the basis for many others, representing a fundamental step of semantic scene understanding.

As one of the leading data annotation companies, we provide comprehensive data annotation solutions for diverse autonomous driving sensor streams.